Methods › Natural Language Processing › Input Embedding Factorization › Adaptive Input Representations
Adaptive Input Representations
Introduced by Alexei Baevski et al. in Adaptive Input Representations for Neural Language Modeling
archive 2025-07-28 Description, source and code snippet are the archive's method entry.
Adaptive Input Embeddings extend the adaptive softmax to input word representations. The factorization assigns more capacity to frequent words and reduces the capacity for less frequent words with the benefit of reducing overfitting to rare words.
In Syntology name not resolved (unknown_name). The lookup matched no method, paper or venue by this name; that asserts nothing about whether code exists.
Near matches the lookup lists, each a different name: same-input rerun; input-only method; Input Perturbation; Mixed Problem Input; Hyper-Representations; LoRA adapter-as-input; input marginalization; hybrid input embedding.
Syntology's have check for this name (free, it answers coverage only), read 2026-10-05; ask it again live. A level is the served verification level of that one answer; nothing here is summed across methods.
Papers archive 2025-07-28
archive papers tagged: 66 · with a code link: 35 · where Syntology ran a sample: 14 (13 with a run with no instrument failure, 1 where every run was a failure of Syntology's instrument) Syntology
Show: all tagged papersonly where code ran (14 of 66 tagged: 13 with a run with no instrument failure, 1 where every run was a failure of Syntology's instrument)
Page 1 of 1: papers 1 to 66 of 66, newest first by the archive's date (ties by slug), in archive order.
Repository counts are the archive's code-links table. A Syntology line states what Syntology ran from that paper's harvested code, as “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the instrument figure counts failures of Syntology's instrument, not of the code. It is per sample and not a correctness claim. When the archive marks a repository official for the paper, the line starts with that repository's state (the archive's flag, not a verdict on who wrote the code; “community repositories only” when every sample that ran came from a community repository, “official: no sample here; runs from other or unrecorded repositories” when some came from a repository the paper names or has in its text, or from none recorded); hover it for the repositories the samples that ran came from.
-
RLBenchNet: The Right Network for the Right Reinforcement Learning Task 21 May 2025 · 1 repository · arXiv:2505.15040
-
A Combined Encoder and Transformer Approach for Coherent and High-Quality Text Generation 19 Nov 2024 · 0 repositories · arXiv:2411.12157
-
Large Body Language Models 21 Oct 2024 · 0 repositories · arXiv:2410.16533
-
Transformers for Supervised Online Continual Learning 3 Mar 2024 · 0 repositories · arXiv:2403.01554
-
UniMem: Towards a Unified View of Long-Context Large Language Models 5 Feb 2024 · 1 repository · arXiv:2402.03009
-
Memory-efficient Stochastic methods for Memory-based Transformers 14 Nov 2023 · 1 repository · arXiv:2311.08123
-
TRAMS: Training-free Memory Selection for Long-range Language Modeling 24 Oct 2023 · 1 repository · arXiv:2310.15494Syntology official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 1 harvested sample)
-
Approximating Two-Layer Feedforward Networks for Efficient Transformers 16 Oct 2023 · 2 repositories · arXiv:2310.10837Syntology official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 1 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 1 unverified (of 4 harvested samples)
-
Memory Gym: Towards Endless Tasks to Benchmark Memory Capabilities of Agents 29 Sep 2023 · 1 repository · arXiv:2309.17207Syntology official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified (of 1 harvested sample)
-
Random-Access Infinite Context Length for Transformers 21 Sep 2023 · 1 repository
-
RCMHA: Relative Convolutional Multi-Head Attention for Natural Language Modelling 7 Aug 2023 · 1 repository · arXiv:2308.03429
-
Landmark Attention: Random-Access Infinite Context Length for Transformers 25 May 2023 · 2 repositories · arXiv:2305.16300Syntology official (archive's flag): 1 ran · 11 ran (of which 5 constructed an object rather than computing a result; 5 with no instrument failure: 0 honoured, 0 violated, 5 with no contract checked; 6 where Syntology's instrument failed) · 2 unverified (of 13 harvested samples)
-
Transformer-based World Models Are Happy With 100k Interactions 13 Mar 2023 · 1 repository · arXiv:2303.07109Syntology official (archive's flag): 16 ran · 16 ran (of which 6 constructed an object rather than computing a result; 8 with no instrument failure: 1 honoured, 1 violated, 6 with no contract checked; 8 where Syntology's instrument failed) · 9 unverified (of 25 harvested samples)
-
GTR-CTRL: Instrument and Genre Conditioning for Guitar-Focused Music Generation with Transformers 10 Feb 2023 · 0 repositories · arXiv:2302.05393
-
An Comparative Analysis of Different Pitch and Metrical Grid Encoding Methods in the Task of Sequential Music Generation 31 Jan 2023 · 0 repositories · arXiv:2301.13383
-
Efficient Sparsely Activated Transformers 31 Aug 2022 · 0 repositories · arXiv:2208.14580
-
Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Models 13 Aug 2022 · 9 repositories · arXiv:2208.06677Syntology community repositories only · 1 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 1 violated, 0 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified (of 1 harvested sample) · 1 pointer-only (licence)
-
Recurrent Memory Transformer 14 Jul 2022 · 3 repositories · arXiv:2207.06881Syntology official (archive's flag): 3 ran · 6 ran (of which 3 constructed an object rather than computing a result; 5 with no instrument failure: 1 honoured, 1 violated, 3 with no contract checked; 1 where Syntology's instrument failed) · 7 unverified (of 13 harvested samples) · 2 pointer-only (licence)
-
Emotion-Aware Transformer Encoder for Empathetic Dialogue Generation 24 Apr 2022 · 1 repository · arXiv:2204.11320
-
SinTra: Learning an inspiration model from a single multi-track music segment 21 Apr 2022 · 1 repository · arXiv:2204.09917
-
LiteTransformerSearch: Training-free Neural Architecture Search for Efficient Language Models 4 Mar 2022 · 1 repository · arXiv:2203.02094Syntology official (archive's flag): 5 ran · 5 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 0 honoured, 0 violated, 4 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 5 harvested samples) · 1 pointer-only (licence)
-
Reconsidering the Past: Optimizing Hidden States in Language Models 16 Dec 2021 · 0 repositories · arXiv:2112.08653
-
A Comparative Study of Transformers on Word Sense Disambiguation 30 Nov 2021 · 0 repositories · arXiv:2111.15417
-
How much do language models copy from their training data? Evaluating linguistic novelty in text generation using RAVEN 18 Nov 2021 · 0 repositories · arXiv:2111.09509
-
TACCL: Guiding Collective Algorithm Synthesis using Communication Sketches 8 Nov 2021 · 2 repositories · arXiv:2111.04867
-
Language Modelling via Learning to Rank 13 Oct 2021 · 0 repositories · arXiv:2110.06961
-
DCT: Dynamic Compressive Transformer for Modeling Unbounded Sequence 10 Oct 2021 · 0 repositories · arXiv:2110.04821
-
Layer-wise Pruning of Transformer Attention Heads for Efficient Language Modeling 7 Oct 2021 · 1 repository · arXiv:2110.03252
-
FNetAR: Mixing Tokens with Autoregressive Fourier Transforms 22 Jul 2021 · 1 repository · arXiv:2107.10932
-
Transformers with multi-modal features and post-fusion context for e-commerce session-based recommendation 11 Jul 2021 · 0 repositories · arXiv:2107.05124
-
ASR Adaptation for E-commerce Chatbots using Cross-Utterance Context and Multi-Task Language Modeling 15 Jun 2021 · 0 repositories · arXiv:2106.09532
-
Adapting Long Context NLM for ASR Rescoring in Conversational Agents 21 Apr 2021 · 0 repositories · arXiv:2104.11070
-
FastMoE: A Fast Mixture-of-Expert Training System 24 Mar 2021 · 3 repositories · arXiv:2103.13262Syntology official (archive's flag): 5 ran · 5 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 0 honoured, 0 violated, 4 with no contract checked; 1 where Syntology's instrument failed) · 1 unverified (of 6 harvested samples) · 1 pointer-only (licence)
-
Adaptive Semiparametric Language Models 4 Feb 2021 · 0 repositories · arXiv:2102.02557
-
Mind the Gap: Assessing Temporal Generalization in Neural Language Models 3 Feb 2021 · 1 repository · arXiv:2102.01951
-
Subformer: A Parameter Reduced Transformer 1 Jan 2021 · 0 repositories
-
Hitachi at SemEval-2020 Task 8: Simple but Effective Modality Ensemble for Meme Emotion Recognition 1 Dec 2020 · 0 repositories
-
Improving RNN transducer with normalized jointer network 3 Nov 2020 · 0 repositories · arXiv:2011.01576
-
Developing Real-time Streaming Transformer Transducer for Speech Recognition on Large-scale Dataset 22 Oct 2020 · 0 repositories · arXiv:2010.11395
-
Transferable Graph Optimizers for ML Compilers 21 Oct 2020 · 0 repositories · arXiv:2010.12438
-
Memformer: A Memory-Augmented Transformer for Sequence Modeling 14 Oct 2020 · 0 repositories · arXiv:2010.06891
-
Pay Attention when Required 9 Sep 2020 · 2 repositories · arXiv:2009.04534
-
The Jazz Transformer on the Front Line: Exploring the Shortcomings of AI-composed Music through Quantitative Measures 4 Aug 2020 · 2 repositories · arXiv:2008.01307Syntology official (archive's flag): 13 ran · 13 ran (of which 0 constructed an object rather than computing a result; 13 with no instrument failure: 0 honoured, 0 violated, 13 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified (of 14 harvested samples)
-
Language Modelling for Source Code with Transformer-XL 31 Jul 2020 · 1 repository · arXiv:2007.15813
-
Do Transformers Need Deep Long-Range Memory 7 Jul 2020 · 1 repository · arXiv:2007.03356
-
Do Transformers Need Deep Long-Range Memory? 1 Jul 2020 · 0 repositories
-
Probing for Referential Information in Language Models 1 Jul 2020 · 0 repositories
-
Mind The Facts: Knowledge-Boosted Coherent Abstractive Text Summarization 27 Jun 2020 · 0 repositories · arXiv:2006.15435
-
Exploring Transformers for Large-Scale Speech Recognition 19 May 2020 · 0 repositories · arXiv:2005.09684
-
Improving Neural Language Generation with Spectrum Control 1 May 2020 · 0 repositories
-
Segatron: Segment-Aware Transformer for Language Modeling and Understanding 30 Apr 2020 · 1 repository · arXiv:2004.14996
-
Finnish Language Modeling with Deep Transformer Models 14 Mar 2020 · 0 repositories · arXiv:2003.11562
-
Neural Academic Paper Generation 2 Dec 2019 · 1 repository · arXiv:1912.01982
-
DeFINE: DEep Factorized INput Token Embeddings for Neural Sequence Modeling 27 Nov 2019 · 1 repository · arXiv:1911.12385
-
Compressive Transformers for Long-Range Sequence Modelling 13 Nov 2019 · 6 repositories · arXiv:1911.05507Syntology 6 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 0 honoured, 2 violated, 3 with no contract checked; 1 where Syntology's instrument failed) · 5 unverified (of 11 harvested samples)
-
Correction of Automatic Speech Recognition with Transformer Sequence-to-sequence Model 23 Oct 2019 · 0 repositories · arXiv:1910.10697
-
Stabilizing Transformers for Reinforcement Learning 13 Oct 2019 · 5 repositories · arXiv:1910.06764Syntology 3 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 2 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 3 harvested samples) · 3 pointer-only (licence)
-
GDP: Generalized Device Placement for Dataflow Graphs 28 Sep 2019 · 0 repositories · arXiv:1910.01578
-
A Self-Attentional Neural Architecture for Code Completion with Multi-Task Learning 16 Sep 2019 · 0 repositories · arXiv:1909.06983
-
Ouroboros: On Accelerating Training of Transformer-Based Language Models 14 Sep 2019 · 1 repository · arXiv:1909.06695
-
A Tensorized Transformer for Language Modeling 24 Jun 2019 · 1 repository · arXiv:1906.09777Syntology official: harvested, nothing ran · 0 ran · 1 unverified (of 1 harvested sample) · 1 pointer-only (licence)
-
XLNet: Generalized Autoregressive Pretraining for Language Understanding 19 Jun 2019 · 27 repositories · arXiv:1906.08237Syntology official (archive's flag): 1 ran · 15 ran (of which 2 constructed an object rather than computing a result; 10 with no instrument failure: 2 honoured, 0 violated, 8 with no contract checked; 5 where Syntology's instrument failed) · 9 unverified (of 24 harvested samples) · 4 pointer-only (licence)
-
Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain) 28 May 2019 · 1 repository · arXiv:1905.11833Syntology official: harvested, nothing ran · 0 ran · 1 unverified (of 1 harvested sample)
-
Transformer-XL: Language Modeling with Longer-Term Dependency 1 May 2019 · 0 repositories
-
Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context 9 Jan 2019 · 37 repositories · arXiv:1901.02860Syntology official (archive's flag): 8 ran · 67 ran (of which 37 constructed an object rather than computing a result; 49 with no instrument failure: 4 honoured, 1 violated, 44 with no contract checked; 18 where Syntology's instrument failed) · 76 unverified (of 143 harvested samples) · 43 pointer-only (licence)
-
Adaptive Input Representations for Neural Language Modeling 28 Sep 2018 · 3 repositories · arXiv:1809.10853
Tasks archive 2025-07-28
20 shown of 86 tasks the archive attaches to papers tagged with this method, by distinct papers. A task without a page in the catalog is plain text.
Usage over time archive 2025-07-28
Components: the archive holds no method-to-method composition, so PwC's Components table cannot be rebuilt; the Papers list carries no Results column for the same reason (the archive does not join its leaderboard rows to method tags).
Categories archive 2025-07-28
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections