{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/method/transformer-xl","slug":"transformer-xl","name":"Transformer-XL","full_name":"Transformer-XL","full_name_withheld":false,"description_markdown":"**Transformer-XL** (meaning extra long) is a [Transformer](https://paperswithcode.com/method/transformer) architecture that introduces the notion of recurrence to the deep self-attention network. Instead of computing the hidden states from scratch for each new segment, Transformer-XL reuses the hidden states obtained in previous segments. The reused hidden states serve as memory for the current segment, which builds up a recurrent connection between the segments. As a result, modeling very long-term dependency becomes possible because information can be propagated through the recurrent connections. As an additional contribution, the Transformer-XL uses a new relative positional encoding formulation that generalizes to attention lengths longer than the one observed during training.","description_state":"present","introduced_year":null,"introduced_by":{"title":"Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context","paper":"/paper/transformer-xl-attentive-language-models","first_author":"Zihang Dai","n_authors":6,"url_abs":null,"archive_paper_url":"https://paperswithcode.com/paper/transformer-xl-attentive-language-models"},"source":{"url":"https://arxiv.org/abs/1901.02860v3","title":"Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context","url_on_a_paper_host":true},"code_snippet_url":null,"code_snippet_url_on_a_code_host":false,"categories":[{"area":"Natural Language Processing","area_id":"natural-language-processing","collection":"Autoregressive Transformers","url":"/methods/category/autoregressive-transformers","pwc_aliases":[]},{"area":"Natural Language Processing","area_id":"natural-language-processing","collection":"Transformers","url":"/methods/category/transformers","pwc_aliases":[]}],"n_papers_tagged":64,"archive_num_papers":64,"papers_newest_first":[{"paper":"/paper/rlbenchnet-the-right-network-for-the-right","slug":"rlbenchnet-the-right-network-for-the-right","title":"RLBenchNet: The Right Network for the Right Reinforcement Learning Task","date":"2025-05-21","arxiv_id":"2505.15040","n_code_links":1,"syntology":null},{"paper":null,"slug":"a-combined-encoder-and-transformer-approach","title":"A Combined Encoder and Transformer Approach for Coherent and High-Quality Text Generation","date":"2024-11-19","arxiv_id":"2411.12157","n_code_links":0,"syntology":null},{"paper":null,"slug":"large-body-language-models","title":"Large Body Language Models","date":"2024-10-21","arxiv_id":"2410.16533","n_code_links":0,"syntology":null},{"paper":null,"slug":"transformers-for-supervised-online-continual","title":"Transformers for Supervised Online Continual Learning","date":"2024-03-03","arxiv_id":"2403.01554","n_code_links":0,"syntology":null},{"paper":"/paper/unimem-towards-a-unified-view-of-long-context","slug":"unimem-towards-a-unified-view-of-long-context","title":"UniMem: Towards a Unified View of Long-Context Large Language Models","date":"2024-02-05","arxiv_id":"2402.03009","n_code_links":1,"syntology":null},{"paper":"/paper/memory-efficient-stochastic-methods-for","slug":"memory-efficient-stochastic-methods-for","title":"Memory-efficient Stochastic methods for Memory-based Transformers","date":"2023-11-14","arxiv_id":"2311.08123","n_code_links":1,"syntology":null},{"paper":"/paper/trams-training-free-memory-selection-for-long","slug":"trams-training-free-memory-selection-for-long","title":"TRAMS: Training-free Memory Selection for Long-range Language Modeling","date":"2023-10-24","arxiv_id":"2310.15494","n_code_links":1,"syntology":{"ran":1,"of":1,"n_ran_checked":0,"n_instrument":1,"unverified":0,"pointer_only":0,"phrase":"1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified","official":{"repos":["lwaekfjlk/trams"],"state":"official (archive's flag): 1 ran","n_ran":1,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":0,"ran_from_kinds":["official"]}}},{"paper":"/paper/approximating-two-layer-feedforward-networks","slug":"approximating-two-layer-feedforward-networks","title":"Approximating Two-Layer Feedforward Networks for Efficient Transformers","date":"2023-10-16","arxiv_id":"2310.10837","n_code_links":2,"syntology":{"ran":3,"of":4,"n_ran_checked":1,"n_instrument":2,"unverified":1,"pointer_only":0,"phrase":"3 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 1 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 1 unverified","official":{"repos":["robertcsordas/moe","robertcsordas/moe_layer"],"state":"official (archive's flag): 3 ran","n_ran":3,"n_constructed":0,"n_ran_no_instrument_failure":1,"n_unverified":1,"ran_from_kinds":["official"]}}},{"paper":"/paper/memory-gym-partially-observable-challenges-to","slug":"memory-gym-partially-observable-challenges-to","title":"Memory Gym: Towards Endless Tasks to Benchmark Memory Capabilities of Agents","date":"2023-09-29","arxiv_id":"2309.17207","n_code_links":1,"syntology":{"ran":1,"of":1,"n_ran_checked":1,"n_instrument":0,"unverified":0,"pointer_only":0,"phrase":"1 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified","official":{"repos":["marcometer/endless-memory-gym"],"state":"official (archive's flag): 1 ran","n_ran":1,"n_constructed":0,"n_ran_no_instrument_failure":1,"n_unverified":0,"ran_from_kinds":["official"]}}},{"paper":"/paper/random-access-infinite-context-length-for","slug":"random-access-infinite-context-length-for","title":"Random-Access Infinite Context Length for Transformers","date":"2023-09-21","arxiv_id":null,"n_code_links":1,"syntology":null},{"paper":"/paper/rcmha-relative-convolutional-multi-head","slug":"rcmha-relative-convolutional-multi-head","title":"RCMHA: Relative Convolutional Multi-Head Attention for Natural Language Modelling","date":"2023-08-07","arxiv_id":"2308.03429","n_code_links":1,"syntology":null},{"paper":"/paper/landmark-attention-random-access-infinite","slug":"landmark-attention-random-access-infinite","title":"Landmark Attention: Random-Access Infinite Context Length for Transformers","date":"2023-05-25","arxiv_id":"2305.16300","n_code_links":2,"syntology":{"ran":11,"of":13,"n_ran_checked":5,"n_instrument":6,"unverified":2,"pointer_only":0,"phrase":"11 ran (of which 5 constructed an object rather than computing a result; 5 with no instrument failure: 0 honoured, 0 violated, 5 with no contract checked; 6 where Syntology's instrument failed) · 2 unverified","official":{"repos":["epfml/landmark-attention"],"state":"official (archive's flag): 1 ran","n_ran":1,"n_constructed":1,"n_ran_no_instrument_failure":1,"n_unverified":0,"ran_from_kinds":["community","listed","official"]}}},{"paper":"/paper/transformer-based-world-models-are-happy-with","slug":"transformer-based-world-models-are-happy-with","title":"Transformer-based World Models Are Happy With 100k Interactions","date":"2023-03-13","arxiv_id":"2303.07109","n_code_links":1,"syntology":{"ran":16,"of":25,"n_ran_checked":8,"n_instrument":8,"unverified":9,"pointer_only":0,"phrase":"16 ran (of which 6 constructed an object rather than computing a result; 8 with no instrument failure: 1 honoured, 1 violated, 6 with no contract checked; 8 where Syntology's instrument failed) · 9 unverified","official":{"repos":["jrobine/twm"],"state":"official (archive's flag): 16 ran","n_ran":16,"n_constructed":6,"n_ran_no_instrument_failure":8,"n_unverified":9,"ran_from_kinds":["official"]}}},{"paper":null,"slug":"gtr-ctrl-instrument-and-genre-conditioning","title":"GTR-CTRL: Instrument and Genre Conditioning for Guitar-Focused Music Generation with Transformers","date":"2023-02-10","arxiv_id":"2302.05393","n_code_links":0,"syntology":null},{"paper":null,"slug":"an-comparative-analysis-of-different-pitch","title":"An Comparative Analysis of Different Pitch and Metrical Grid Encoding Methods in the Task of Sequential Music Generation","date":"2023-01-31","arxiv_id":"2301.13383","n_code_links":0,"syntology":null},{"paper":null,"slug":"efficient-sparsely-activated-transformers","title":"Efficient Sparsely Activated Transformers","date":"2022-08-31","arxiv_id":"2208.14580","n_code_links":0,"syntology":null},{"paper":"/paper/adan-adaptive-nesterov-momentum-algorithm-for","slug":"adan-adaptive-nesterov-momentum-algorithm-for","title":"Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Models","date":"2022-08-13","arxiv_id":"2208.06677","n_code_links":9,"syntology":{"ran":1,"of":1,"n_ran_checked":1,"n_instrument":0,"unverified":0,"pointer_only":1,"phrase":"1 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 1 violated, 0 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified","official":{"repos":["sail-sg/adan"],"state":"community repositories only","n_ran":0,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":0,"ran_from_kinds":["listed"]}}},{"paper":"/paper/recurrent-memory-transformer","slug":"recurrent-memory-transformer","title":"Recurrent Memory Transformer","date":"2022-07-14","arxiv_id":"2207.06881","n_code_links":3,"syntology":{"ran":6,"of":13,"n_ran_checked":5,"n_instrument":1,"unverified":7,"pointer_only":2,"phrase":"6 ran (of which 3 constructed an object rather than computing a result; 5 with no instrument failure: 1 honoured, 1 violated, 3 with no contract checked; 1 where Syntology's instrument failed) · 7 unverified","official":{"repos":["booydar/lm-rmt","booydar/transformer-xl"],"state":"official (archive's flag): 3 ran","n_ran":3,"n_constructed":3,"n_ran_no_instrument_failure":3,"n_unverified":6,"ran_from_kinds":["community","official","unlocated"]}}},{"paper":"/paper/emotion-aware-transformer-encoder-for-1","slug":"emotion-aware-transformer-encoder-for-1","title":"Emotion-Aware Transformer Encoder for Empathetic Dialogue Generation","date":"2022-04-24","arxiv_id":"2204.11320","n_code_links":1,"syntology":null},{"paper":"/paper/sintra-learning-an-inspiration-model-from-a","slug":"sintra-learning-an-inspiration-model-from-a","title":"SinTra: Learning an inspiration model from a single multi-track music segment","date":"2022-04-21","arxiv_id":"2204.09917","n_code_links":1,"syntology":null},{"paper":"/paper/litetransformersearch-training-free-on-device","slug":"litetransformersearch-training-free-on-device","title":"LiteTransformerSearch: Training-free Neural Architecture Search for Efficient Language Models","date":"2022-03-04","arxiv_id":"2203.02094","n_code_links":1,"syntology":{"ran":5,"of":5,"n_ran_checked":4,"n_instrument":1,"unverified":0,"pointer_only":1,"phrase":"5 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 0 honoured, 0 violated, 4 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified","official":{"repos":["microsoft/archai"],"state":"official (archive's flag): 5 ran","n_ran":5,"n_constructed":0,"n_ran_no_instrument_failure":4,"n_unverified":0,"ran_from_kinds":["official"]}}},{"paper":null,"slug":"reconsidering-the-past-optimizing-hidden-1","title":"Reconsidering the Past: Optimizing Hidden States in Language Models","date":"2021-12-16","arxiv_id":"2112.08653","n_code_links":0,"syntology":null},{"paper":null,"slug":"a-comparative-study-of-transformers-on-word","title":"A Comparative Study of Transformers on Word Sense Disambiguation","date":"2021-11-30","arxiv_id":"2111.15417","n_code_links":0,"syntology":null},{"paper":null,"slug":"how-much-do-language-models-copy-from-their","title":"How much do language models copy from their training data? Evaluating linguistic novelty in text generation using RAVEN","date":"2021-11-18","arxiv_id":"2111.09509","n_code_links":0,"syntology":null},{"paper":"/paper/synthesizing-collective-communication","slug":"synthesizing-collective-communication","title":"TACCL: Guiding Collective Algorithm Synthesis using Communication Sketches","date":"2021-11-08","arxiv_id":"2111.04867","n_code_links":2,"syntology":null},{"paper":null,"slug":"language-modelling-via-learning-to-rank","title":"Language Modelling via Learning to Rank","date":"2021-10-13","arxiv_id":"2110.06961","n_code_links":0,"syntology":null},{"paper":"/paper/layer-wise-pruning-of-transformer-attention","slug":"layer-wise-pruning-of-transformer-attention","title":"Layer-wise Pruning of Transformer Attention Heads for Efficient Language Modeling","date":"2021-10-07","arxiv_id":"2110.03252","n_code_links":1,"syntology":null},{"paper":"/paper/fnetar-mixing-tokens-with-autoregressive","slug":"fnetar-mixing-tokens-with-autoregressive","title":"FNetAR: Mixing Tokens with Autoregressive Fourier Transforms","date":"2021-07-22","arxiv_id":"2107.10932","n_code_links":1,"syntology":null},{"paper":"/paper/transformers-with-multi-modal-features-and","slug":"transformers-with-multi-modal-features-and","title":"Transformers with multi-modal features and post-fusion context for e-commerce session-based recommendation","date":"2021-07-11","arxiv_id":"2107.05124","n_code_links":0,"syntology":null},{"paper":null,"slug":"asr-adaptation-for-e-commerce-chatbots-using","title":"ASR Adaptation for E-commerce Chatbots using Cross-Utterance Context and Multi-Task Language Modeling","date":"2021-06-15","arxiv_id":"2106.09532","n_code_links":0,"syntology":null},{"paper":null,"slug":"what-s-the-context-long-context-nlm","title":"Adapting Long Context NLM for ASR Rescoring in Conversational Agents","date":"2021-04-21","arxiv_id":"2104.11070","n_code_links":0,"syntology":null},{"paper":"/paper/fastmoe-a-fast-mixture-of-expert-training","slug":"fastmoe-a-fast-mixture-of-expert-training","title":"FastMoE: A Fast Mixture-of-Expert Training System","date":"2021-03-24","arxiv_id":"2103.13262","n_code_links":3,"syntology":{"ran":5,"of":6,"n_ran_checked":4,"n_instrument":1,"unverified":1,"pointer_only":1,"phrase":"5 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 0 honoured, 0 violated, 4 with no contract checked; 1 where Syntology's instrument failed) · 1 unverified","official":{"repos":["davidmrau/mixture-of-experts","laekov/fastmoe"],"state":"official (archive's flag): 5 ran","n_ran":5,"n_constructed":0,"n_ran_no_instrument_failure":4,"n_unverified":1,"ran_from_kinds":["official"]}}},{"paper":null,"slug":"adaptive-semiparametric-language-models","title":"Adaptive Semiparametric Language Models","date":"2021-02-04","arxiv_id":"2102.02557","n_code_links":0,"syntology":null},{"paper":"/paper/pitfalls-of-static-language-modelling","slug":"pitfalls-of-static-language-modelling","title":"Mind the Gap: Assessing Temporal Generalization in Neural Language Models","date":"2021-02-03","arxiv_id":"2102.01951","n_code_links":1,"syntology":null},{"paper":"/paper/subformer-a-parameter-reduced-transformer","slug":"subformer-a-parameter-reduced-transformer","title":"Subformer: A Parameter Reduced Transformer","date":"2021-01-01","arxiv_id":null,"n_code_links":0,"syntology":null},{"paper":null,"slug":"hitachi-at-semeval-2020-task-8-simple-but","title":"Hitachi at SemEval-2020 Task 8: Simple but Effective Modality Ensemble for Meme Emotion Recognition","date":"2020-12-01","arxiv_id":null,"n_code_links":0,"syntology":null},{"paper":null,"slug":"improving-rnn-transducer-with-normalized","title":"Improving RNN transducer with normalized jointer network","date":"2020-11-03","arxiv_id":"2011.01576","n_code_links":0,"syntology":null},{"paper":null,"slug":"developing-real-time-streaming-transformer","title":"Developing Real-time Streaming Transformer Transducer for Speech Recognition on Large-scale Dataset","date":"2020-10-22","arxiv_id":"2010.11395","n_code_links":0,"syntology":null},{"paper":null,"slug":"transferable-graph-optimizers-for-ml","title":"Transferable Graph Optimizers for ML Compilers","date":"2020-10-21","arxiv_id":"2010.12438","n_code_links":0,"syntology":null},{"paper":null,"slug":"memformer-the-memory-augmented-transformer-1","title":"Memformer: A Memory-Augmented Transformer for Sequence Modeling","date":"2020-10-14","arxiv_id":"2010.06891","n_code_links":0,"syntology":null},{"paper":"/paper/pay-attention-when-required","slug":"pay-attention-when-required","title":"Pay Attention when Required","date":"2020-09-09","arxiv_id":"2009.04534","n_code_links":2,"syntology":null},{"paper":"/paper/the-jazz-transformer-on-the-front-line","slug":"the-jazz-transformer-on-the-front-line","title":"The Jazz Transformer on the Front Line: Exploring the Shortcomings of AI-composed Music through Quantitative Measures","date":"2020-08-04","arxiv_id":"2008.01307","n_code_links":2,"syntology":{"ran":13,"of":14,"n_ran_checked":13,"n_instrument":0,"unverified":1,"pointer_only":0,"phrase":"13 ran (of which 0 constructed an object rather than computing a result; 13 with no instrument failure: 0 honoured, 0 violated, 13 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified","official":{"repos":["slSeanWU/MusDr","slSeanWU/jazz_transformer"],"state":"official (archive's flag): 13 ran","n_ran":13,"n_constructed":0,"n_ran_no_instrument_failure":13,"n_unverified":1,"ran_from_kinds":["official"]}}},{"paper":"/paper/language-modelling-for-source-code-with","slug":"language-modelling-for-source-code-with","title":"Language Modelling for Source Code with Transformer-XL","date":"2020-07-31","arxiv_id":"2007.15813","n_code_links":1,"syntology":null},{"paper":"/paper/do-transformers-need-deep-long-range-memory-1","slug":"do-transformers-need-deep-long-range-memory-1","title":"Do Transformers Need Deep Long-Range Memory","date":"2020-07-07","arxiv_id":"2007.03356","n_code_links":1,"syntology":null},{"paper":null,"slug":"do-transformers-need-deep-long-range-memory","title":"Do Transformers Need Deep Long-Range Memory?","date":"2020-07-01","arxiv_id":null,"n_code_links":0,"syntology":null},{"paper":null,"slug":"probing-for-referential-information-in","title":"Probing for Referential Information in Language Models","date":"2020-07-01","arxiv_id":null,"n_code_links":0,"syntology":null},{"paper":null,"slug":"mind-the-facts-knowledge-boosted-coherent","title":"Mind The Facts: Knowledge-Boosted Coherent Abstractive Text Summarization","date":"2020-06-27","arxiv_id":"2006.15435","n_code_links":0,"syntology":null},{"paper":null,"slug":"exploring-transformers-for-large-scale-speech","title":"Exploring Transformers for Large-Scale Speech Recognition","date":"2020-05-19","arxiv_id":"2005.09684","n_code_links":0,"syntology":null},{"paper":null,"slug":"improving-neural-language-generation-with","title":"Improving Neural Language Generation with Spectrum Control","date":"2020-05-01","arxiv_id":null,"n_code_links":0,"syntology":null},{"paper":"/paper/segabert-pre-training-of-segment-aware-bert","slug":"segabert-pre-training-of-segment-aware-bert","title":"Segatron: Segment-Aware Transformer for Language Modeling and Understanding","date":"2020-04-30","arxiv_id":"2004.14996","n_code_links":1,"syntology":null},{"paper":null,"slug":"finnish-language-modeling-with-deep","title":"Finnish Language Modeling with Deep Transformer Models","date":"2020-03-14","arxiv_id":"2003.11562","n_code_links":0,"syntology":null},{"paper":"/paper/neural-academic-paper-generation","slug":"neural-academic-paper-generation","title":"Neural Academic Paper Generation","date":"2019-12-02","arxiv_id":"1912.01982","n_code_links":1,"syntology":null},{"paper":"/paper/define-deep-factorized-input-word-embeddings-1","slug":"define-deep-factorized-input-word-embeddings-1","title":"DeFINE: DEep Factorized INput Token Embeddings for Neural Sequence Modeling","date":"2019-11-27","arxiv_id":"1911.12385","n_code_links":1,"syntology":null},{"paper":"/paper/compressive-transformers-for-long-range-1","slug":"compressive-transformers-for-long-range-1","title":"Compressive Transformers for Long-Range Sequence Modelling","date":"2019-11-13","arxiv_id":"1911.05507","n_code_links":6,"syntology":{"ran":6,"of":11,"n_ran_checked":5,"n_instrument":1,"unverified":5,"pointer_only":0,"phrase":"6 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 0 honoured, 2 violated, 3 with no contract checked; 1 where Syntology's instrument failed) · 5 unverified","official":null}},{"paper":null,"slug":"correction-of-automatic-speech-recognition","title":"Correction of Automatic Speech Recognition with Transformer Sequence-to-sequence Model","date":"2019-10-23","arxiv_id":"1910.10697","n_code_links":0,"syntology":null},{"paper":"/paper/stabilizing-transformers-for-reinforcement-1","slug":"stabilizing-transformers-for-reinforcement-1","title":"Stabilizing Transformers for Reinforcement Learning","date":"2019-10-13","arxiv_id":"1910.06764","n_code_links":5,"syntology":{"ran":3,"of":3,"n_ran_checked":2,"n_instrument":1,"unverified":0,"pointer_only":3,"phrase":"3 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 2 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified","official":null}},{"paper":null,"slug":"gdp-generalized-device-placement-for-dataflow","title":"GDP: Generalized Device Placement for Dataflow Graphs","date":"2019-09-28","arxiv_id":"1910.01578","n_code_links":0,"syntology":null},{"paper":null,"slug":"a-self-attentional-neural-architecture-for","title":"A Self-Attentional Neural Architecture for Code Completion with Multi-Task Learning","date":"2019-09-16","arxiv_id":"1909.06983","n_code_links":0,"syntology":null},{"paper":"/paper/ouroboros-on-accelerating-training-of","slug":"ouroboros-on-accelerating-training-of","title":"Ouroboros: On Accelerating Training of Transformer-Based Language Models","date":"2019-09-14","arxiv_id":"1909.06695","n_code_links":1,"syntology":null},{"paper":"/paper/a-tensorized-transformer-for-language","slug":"a-tensorized-transformer-for-language","title":"A Tensorized Transformer for Language Modeling","date":"2019-06-24","arxiv_id":"1906.09777","n_code_links":1,"syntology":{"ran":0,"of":1,"n_ran_checked":0,"n_instrument":0,"unverified":1,"pointer_only":1,"phrase":"0 ran · 1 unverified","official":{"repos":["szhangtju/The-compression-of-Transformer"],"state":"official: harvested, nothing ran","n_ran":0,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":1,"ran_from_kinds":[]}}},{"paper":"/paper/xlnet-generalized-autoregressive-pretraining","slug":"xlnet-generalized-autoregressive-pretraining","title":"XLNet: Generalized Autoregressive Pretraining for Language Understanding","date":"2019-06-19","arxiv_id":"1906.08237","n_code_links":27,"syntology":{"ran":15,"of":24,"n_ran_checked":10,"n_instrument":5,"unverified":9,"pointer_only":4,"phrase":"15 ran (of which 2 constructed an object rather than computing a result; 10 with no instrument failure: 2 honoured, 0 violated, 8 with no contract checked; 5 where Syntology's instrument failed) · 9 unverified","official":{"repos":["zihangdai/xlnet"],"state":"official (archive's flag): 1 ran","n_ran":1,"n_constructed":0,"n_ran_no_instrument_failure":1,"n_unverified":1,"ran_from_kinds":["listed","official"]}}},{"paper":"/paper/interpreting-and-improving-natural-language","slug":"interpreting-and-improving-natural-language","title":"Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain)","date":"2019-05-28","arxiv_id":"1905.11833","n_code_links":1,"syntology":{"ran":0,"of":1,"n_ran_checked":0,"n_instrument":0,"unverified":1,"pointer_only":0,"phrase":"0 ran · 1 unverified","official":{"repos":["mtoneva/brain_language_nlp"],"state":"official: harvested, nothing ran","n_ran":0,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":1,"ran_from_kinds":[]}}},{"paper":null,"slug":"transformer-xl-language-modeling-with-longer","title":"Transformer-XL: Language Modeling with Longer-Term Dependency","date":"2019-05-01","arxiv_id":null,"n_code_links":0,"syntology":null},{"paper":"/paper/transformer-xl-attentive-language-models","slug":"transformer-xl-attentive-language-models","title":"Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context","date":"2019-01-09","arxiv_id":"1901.02860","n_code_links":37,"syntology":{"ran":67,"of":143,"n_ran_checked":49,"n_instrument":18,"unverified":76,"pointer_only":43,"phrase":"67 ran (of which 37 constructed an object rather than computing a result; 49 with no instrument failure: 4 honoured, 1 violated, 44 with no contract checked; 18 where Syntology's instrument failed) · 76 unverified","official":{"repos":["kimiyoung/transformer-xl"],"state":"official (archive's flag): 8 ran","n_ran":8,"n_constructed":7,"n_ran_no_instrument_failure":7,"n_unverified":5,"ran_from_kinds":["listed","official","unlocated"]}}}],"papers_shown":64,"papers_order":"date (newest first), then slug","paper_counts":{"archive_papers_tagged":64,"with_a_code_link":34,"where_syntology_ran_a_sample":14,"not_listed_spam_title":0,"listed":64,"listed_where_code_ran":14,"where_syntology_ran_a_sample_split":{"with_a_run_with_no_instrument_failure":13,"every_run_a_failure_of_syntologys_instrument":1,"listed_with_a_run_with_no_instrument_failure":13,"listed_every_run_a_failure_of_syntologys_instrument":1,"filter":{"states":["a run with no instrument failure","any run, instrument failures included"],"default":"a run with no instrument failure","note":"on the 'only where code ran' pages the default hides, in the browser, the rows where every run was a failure of Syntology's instrument; the second state shows them again. Rows are hidden, never re-ordered; these twins list every row"}},"definition":"distinct papers the archive tags; 'where Syntology ran a sample' counts papers with at least one harvested sample that ran, which is not a correctness claim"},"papers_where_code_ran":14,"papers_where_code_ran_first_page":"/method/transformer-xl/papers/ran/1","papers_where_code_ran_caption":"We ran code from the paper's repository; we did not isolate this method inside it.","pages":[{"order":"ran","page":1,"of":1,"url":"/method/transformer-xl/papers/ran/1","json":"/method/transformer-xl/papers/ran/1.json","rows":14}],"tasks":[{"task":"/task/language-modelling","name":"Language Modelling","papers":38},{"task":"/task/language-modeling","name":"Language Modeling","papers":27},{"task":"/task/decoder","name":"Decoder","papers":7},{"task":"/task/machine-translation","name":"Machine Translation","papers":6},{"task":"/task/speech-recognition","name":"Speech Recognition","papers":5},{"task":"/task/translation","name":"Translation","papers":5},{"task":"/task/speech-recognition-1","name":"speech-recognition","papers":5},{"task":"/task/text-generation","name":"Text Generation","papers":4},{"task":"/task/automatic-speech-recognition-2","name":"Automatic Speech Recognition","papers":3},{"task":"/task/automatic-speech-recognition","name":"Automatic Speech Recognition (ASR)","papers":3},{"task":"/task/paraphrase-identification","name":"Paraphrase Identification","papers":3},{"task":"/task/reinforcement-learning-1","name":"Reinforcement Learning (RL)","papers":3},{"task":"/task/sentence","name":"Sentence","papers":3},{"task":"/task/word-embeddings","name":"Word Embeddings","papers":3},{"task":"/task/abstractive-text-summarization","name":"Abstractive Text Summarization","papers":2},{"task":"/task/deep-attention","name":"Deep Attention","papers":2},{"task":"/task/deep-reinforcement-learning","name":"Deep Reinforcement Learning","papers":2},{"task":null,"name":"GPU","papers":2},{"task":"/task/graph-neural-network","name":"Graph Neural Network","papers":2},{"task":"/task/music-generation","name":"Music Generation","papers":2}],"tasks_shown":20,"n_tasks":86,"usage_by_year":[{"year":"2019","papers":13},{"year":"2020","papers":16},{"year":"2021","papers":14},{"year":"2022","papers":6},{"year":"2023","papers":10},{"year":"2024","papers":4},{"year":"2025","papers":1}],"row_source":"methods_table","archive":{"source":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","archive_url":"https://paperswithcode.com/method/transformer-xl"},"syntology_read_at":"2026-10-05T00:36:11+00:00","syntology":{"source":"Syntology's have check for this name (the free coverage lookup), read from Syntology's graph","have_url":"https://syntology.ai/api/graph/have?x=Transformer-XL","read_at":"2026-10-05T01:03:10+00:00","query":"Transformer-XL","kind":"method","have":"declared_code","tier":"repo_known","verified_implementation":false,"level":null,"level_meaning":"the served verification level, per answer; never summed across methods","resolved_to":{"method":"Transformer-XL","origin_arxiv_id":"1901.02860"},"resolution":"exact","resolved_name_differs":false,"absence_asserted":false,"absence_unchecked":null,"no_code_is_about_this_method":false,"subject":"same_paper","archive_paper_arxiv_id":"1901.02860","repositories":{"listed":5,"in_graph":36,"list_truncated":true,"strongest_link":"official"},"retrieve_with":null,"name_competition":{"verdict":"contested","may_be_wrong_sense":false,"bearers_found":2,"served":{"arxiv_id":"1901.02860","citations":4476},"leading_other_bearer":{"arxiv_id":"openreview_HJePno0cYm","title":"Transformer-XL: Language Modeling with Longer-Term Dependency","citations":null,"named_by":["title_prefix"],"bearer_name":"Transformer-XL: Language Modeling with Longer-Term Dependency","ask_with":"have('openreview_HJePno0cYm')"},"n_other_bearers_listed":1,"note":"1 other thing(s) here are also called 'Transformer-XL', and none of them is cited more than the paper this answer is about. They are listed in `also_named` -- check `served.arxiv_id` is the paper you meant."},"name_competition_text":null,"near_matches":null,"code_nearby":[{"name":"Agentic Transformer","relation":"shares_name_tokens","relation_means":"shares name tokens with this name; a different method","level":3,"retrieve_with":"get_reference_implementation(name='Agentic Transformer')","origin_arxiv_id":"2305.16554"},{"name":"Waypoint Transformer","relation":"shares_name_tokens","relation_means":"shares name tokens with this name; a different method","level":2,"retrieve_with":"get_reference_implementation(name='Waypoint Transformer')","origin_arxiv_id":"2306.14069"},{"name":"Block-State Transformer","relation":"shares_name_tokens","relation_means":"shares name tokens with this name; a different method","level":3,"retrieve_with":"get_reference_implementation(name='Block-State Transformer')","origin_arxiv_id":"2306.09539"},{"name":"Bootstrapped Transformer","relation":"shares_name_tokens","relation_means":"shares name tokens with this name; a different method","level":3,"retrieve_with":"get_reference_implementation(name='Bootstrapped Transformer')","origin_arxiv_id":"2206.08569"},{"name":"Focused Transformer (FOT)","relation":"shares_name_tokens","relation_means":"shares name tokens with this name; a different method","level":3,"retrieve_with":"get_reference_implementation(name='Focused Transformer (FOT)')","origin_arxiv_id":"2307.03170"},{"name":"Elastic Decision Transformer","relation":"shares_name_tokens","relation_means":"shares name tokens with this name; a different method","level":2,"retrieve_with":"get_reference_implementation(name='Elastic Decision Transformer')","origin_arxiv_id":"2307.02484"}],"code_nearby_caption":"Code Syntology holds under other names, as the lookup lists them (code_nearby); not a complete list, and none of them is this method.","origin_paper":{"arxiv_id":"1901.02860","page":"/paper/transformer-xl-attentive-language-models","syntology_url":"https://syntology.ai/paper/1901.02860","in_graph_layer":true,"samples":143,"ran":67,"n_ran_checked":49,"n_instrument":18,"unverified":76,"phrase":"67 ran (of which 37 constructed an object rather than computing a result; 49 with no instrument failure: 4 honoured, 1 violated, 44 with no contract checked; 18 where Syntology's instrument failed) · 76 unverified","instrument_means":"failures of Syntology's instrument, not of the code"},"line":"repository known, no verified implementation (declared_code; repositories: 5 listed of 36 in the graph, strongest link: official). This answer is about arXiv:1901.02860, the paper the archive credits with this method.","level_legend":{"state":"quoted","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","card_sha256":"479785a84658dd682a11b10237b3620d651d7f87799f5dc7add970aaa83f6d39","note":"Syntology's served verification levels, quoted verbatim from its MCP server card; a row is split where the card separates clauses with a double hyphen, and the separator is not reproduced","levels":[{"level":"V0","quoted":["imports"]},{"level":"V1","quoted":["executes against a synthesized fixture and returns a value of the recorded type"]},{"level":"V2","quoted":["two implementations GENERATED from the paper's signature and spec agree numerically (1e-6) on fixture inputs","neither of the two ever read the authors' own code, so V2 measures how unambiguously the paper specifies the method, not fidelity to the authors' implementation (measured 2026-09-11, commit 75eaa77: cross_checked_by is a second LLM on 2,628 of 2,831 served samples)"]},{"level":"V3","quoted":["paper-derived property tests pass on both of those same two generations."]}],"no_rung":"No rung on this ladder compares the served code against the authors' implementation, so treat every level as evidence about runnability and internal agreement, not about matching the paper's results."}},"syntology_have_state":"loaded","record_sha256":"2c8905077b4e258bfbe0f02f019f29b8be2afd96df49249be38fa20f82cdb977","record_changed_at":"2026-10-05","record_changed_at_basis":"changed"}