Methods › General › Parameter Sharing › Weight Tying
Weight Tying
Introduced by Ofir Press et al. in Using the Output Embedding to Improve Language Models
archive 2025-07-28 Description, source and code snippet are the archive's method entry.
Weight Tying improves the performance of language models by tying (sharing) the weights of the embedding and softmax layers. This method also massively reduces the total number of parameters in the language models that it is applied to.
Language models are typically comprised of an embedding layer, followed by a number of Transformer or LSTM layers, which are finally followed by a softmax layer. Embedding layers learn word representations, such that similar words (in meaning) are represented by vectors that are near each other (in cosine distance). [Press & Wolf, 2016] showed that the softmax matrix, in which every word also has a vector representation, also exhibits this property. This leads them to propose to share the softmax and embedding matrices, which is done today in nearly all language models.
This method was independently introduced by Press & Wolf, 2016 and Inan et al, 2016.
Additionally, the Press & Wolf paper proposes Three-way Weight Tying, a method for NMT models in which the embedding matrix for the source language, the embedding matrix for the target language, and the softmax matrix for the target language are all tied. That method has been adopted by the Attention Is All You Need model and many other neural machine translation models.
In Syntology name not resolved (unknown_name). The lookup matched no method, paper or venue by this name; that asserts nothing about whether code exists.
Near matches the lookup lists, each a different name: Expert Tying; Weight Mixup; Weight Weaving; weight scaling; Weight Recycling; Weight Balancing; weight operators; Filter-then-Weight.
Syntology's have check for this name (free, it answers coverage only), read 2026-10-05; ask it again live. A level is the served verification level of that one answer; nothing here is summed across methods.
Papers archive 2025-07-28
archive papers tagged: 63 · with a code link: 30 · where Syntology ran a sample: 5 (3 with a run with no instrument failure, 2 where every run was a failure of Syntology's instrument) Syntology
Show: all tagged papersonly where code ran (5 of 63 tagged: 3 with a run with no instrument failure, 2 where every run was a failure of Syntology's instrument)
Page 1 of 1: papers 1 to 63 of 63, newest first by the archive's date (ties by slug), in archive order.
Repository counts are the archive's code-links table. A Syntology line states what Syntology ran from that paper's harvested code, as “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the instrument figure counts failures of Syntology's instrument, not of the code. It is per sample and not a correctness claim. When the archive marks a repository official for the paper, the line starts with that repository's state (the archive's flag, not a verdict on who wrote the code; “community repositories only” when every sample that ran came from a community repository, “official: no sample here; runs from other or unrecorded repositories” when some came from a repository the paper names or has in its text, or from none recorded); hover it for the repositories the samples that ran came from.
-
Advanced Deep Learning Techniques for Analyzing Earnings Call Transcripts: Methodologies and Applications 27 Feb 2025 · 0 repositories · arXiv:2503.01886
-
No Argument Left Behind: Overlapping Chunks for Faster Processing of Arbitrarily Long Legal Texts 24 Oct 2024 · 0 repositories · arXiv:2410.19184
-
RICo: Reddit ideological communities 5 Jun 2024 · 1 repository
-
Exploring Multi-Level Threats in Telegram Data with AI-Human Annotation: A Preliminary Study 15 Dec 2023 · 0 repositories
-
Illicit Darkweb Classification via Natural-language Processing: Classifying Illicit Content of Webpages based on Textual Information 8 Dec 2023 · 0 repositories · arXiv:2312.04944
-
Tied-Lora: Enhancing parameter efficiency of LoRA with weight tying 16 Nov 2023 · 0 repositories · arXiv:2311.09578
-
Headless Language Models: Learning without Predicting with Contrastive Weight Tying 15 Sep 2023 · 0 repositories · arXiv:2309.08351
-
Act3D: 3D Feature Field Transformers for Multi-Task Robotic Manipulation 30 Jun 2023 · 2 repositories · arXiv:2306.17817
-
Explainable and High-Performance Hate and Offensive Speech Detection 26 Jun 2022 · 0 repositories · arXiv:2206.12983
-
Approximately Equivariant Networks for Imperfectly Symmetric Dynamics 28 Jan 2022 · 1 repository · arXiv:2201.11969Syntology official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified (of 2 harvested samples) · 2 pointer-only (licence)
-
IIITT@Dravidian-CodeMix-FIRE2021: Transliterate or translate? Sentiment analysis of code-mixed text in Dravidian languages 15 Nov 2021 · 1 repository · arXiv:2111.07906
-
Offensive Language Identification in Low-resourced Code-mixed Dravidian languages using Pseudo-labeling 27 Aug 2021 · 1 repository · arXiv:2108.12177
-
Towards Offensive Language Identification for Tamil Code-Mixed YouTube Comments and Posts 24 Aug 2021 · 1 repository · arXiv:2108.10939
-
Learning ULMFiT and Self-Distillation with Calibration for Medical Dialogue System 20 Jul 2021 · 0 repositories · arXiv:2107.09625
-
WHOSe Heritage: Classification of UNESCO World Heritage "Outstanding Universal Value" Documents with Soft Labels 12 Apr 2021 · 1 repository · arXiv:2104.05547
-
L3CubeMahaSent: A Marathi Tweet-based Sentiment Analysis Dataset 21 Mar 2021 · 1 repository · arXiv:2103.11408
-
On the Theory of Implicit Deep Learning: Global Convergence with Implicit Layers 15 Feb 2021 · 0 repositories · arXiv:2102.07346
-
indicnlp@kgp at DravidianLangTech-EACL2021: Offensive Language Identification in Dravidian Languages 14 Feb 2021 · 1 repository · arXiv:2102.07150
-
indicnlp@ kgp at DravidianLangTech-EACL2021: Offensive Language Identification in Dravidian Languages 14 Feb 2021 · 1 repository
-
Train your classifier first: Cascade Neural Networks Training from upper layers to lower layers 9 Feb 2021 · 0 repositories · arXiv:2102.04697
-
Experimental Evaluation of Deep Learning models for Marathi Text Classification 13 Jan 2021 · 0 repositories · arXiv:2101.04899
-
LaDiff ULMFiT: A Layer Differentiated training approach for ULMFiT 13 Jan 2021 · 1 repository · arXiv:2101.04965
-
Post-Training Weighted Quantization of Neural Networks for Language Models 1 Jan 2021 · 0 repositories
-
HinglishNLP at SemEval-2020 Task 9: Fine-tuned Language Models for Hinglish Sentiment Detection 1 Dec 2020 · 2 repositories
-
Smash at SemEval-2020 Task 7: Optimizing the Hyperparameters of ERNIE 2.0 for Humor Ranking and Rating 1 Dec 2020 · 0 repositories
-
Palomino-Ochoa at SemEval-2020 Task 9: Robust System based on Transformer for Code-Mixed Sentiment Classification 18 Nov 2020 · 0 repositories · arXiv:2011.09448
-
Pagsusuri ng RNN-based Transfer Learning Technique sa Low-Resource Language 13 Oct 2020 · 2 repositories · arXiv:2010.06447
-
Gauravarora@HASOC-Dravidian-CodeMix-FIRE2020: Pre-training ULMFiT on Synthetically Generated Code-Mixed Data for Hate Speech Detection 5 Oct 2020 · 0 repositories · arXiv:2010.02094
-
Fine-tuning Pre-trained Contextual Embeddings for Citation Content Analysis in Scholarly Publication 12 Sep 2020 · 0 repositories · arXiv:2009.05836
-
HinglishNLP: Fine-tuned Language Models for Hinglish Sentiment Detection 22 Aug 2020 · 2 repositories · arXiv:2008.09820
-
Composer Style Classification of Piano Sheet Music Images Using Language Model Pretraining 29 Jul 2020 · 1 repository · arXiv:2007.14587
-
Probing for Referential Information in Language Models 1 Jul 2020 · 0 repositories
-
Evaluation Metrics for Headline Generation Using Deep Pre-Trained Embeddings 1 May 2020 · 0 repositories
-
Offensive language detection in Arabic using ULMFiT 1 May 2020 · 1 repository
-
Text Categorization for Conflict Event Annotation 1 May 2020 · 0 repositories
-
Inferring the source of official texts: can SVM beat ULMFiT? 2 Mar 2020 · 1 repository
-
MaxUp: A Simple Way to Improve Generalization of Neural Network Training 20 Feb 2020 · 1 repository · arXiv:2002.09024Syntology 0 ran · 1 unverified (of 1 harvested sample)
-
Localized Flood DetectionWith Minimal Labeled Social Media Data Using Transfer Learning 10 Feb 2020 · 0 repositories · arXiv:2003.04973
-
Natural language processing of MIMIC-III clinical notes for identifying diagnosis and procedures with neural networks 28 Dec 2019 · 0 repositories · arXiv:1912.12397
-
A Comparative Study of Pretrained Language Models on Thai Social Text Categorization 3 Dec 2019 · 0 repositories · arXiv:1912.01580
-
DeFINE: DEep Factorized INput Token Embeddings for Neural Sequence Modeling 27 Nov 2019 · 1 repository · arXiv:1911.12385
-
A Subword Level Language Model for Bangla Language 15 Nov 2019 · 0 repositories · arXiv:1911.07613
-
Evolution of transfer learning in natural language processing 16 Oct 2019 · 0 repositories · arXiv:1910.07370
-
The merits of Universal Language Model Fine-tuning for Small Datasets -- a case with Dutch book reviews 2 Oct 2019 · 2 repositories · arXiv:1910.00896
-
Analyzing Customer Feedback for Product Fit Prediction 28 Aug 2019 · 0 repositories · arXiv:1908.10896
-
Efficient Language Modeling with Automatic Relevance Determination in Recurrent Neural Networks 1 Aug 2019 · 0 repositories
-
Representation Degeneration Problem in Training Natural Language Generation Models 28 Jul 2019 · 1 repository · arXiv:1907.12009
-
Low-Shot Classification: A Comparison of Classical and Deep Transfer Machine Learning Approaches 17 Jul 2019 · 0 repositories · arXiv:1907.07543
-
Evaluating Language Model Finetuning Techniques for Low-resource Languages 30 Jun 2019 · 2 repositories · arXiv:1907.00409
-
Exploiting Unsupervised Pre-training and Automated Feature Engineering for Low-resource Hate Speech Detection in Polish 17 Jun 2019 · 0 repositories · arXiv:1906.09325
-
Figure Eight at SemEval-2019 Task 3: Ensemble of Transfer Learning Methods for Contextual Emotion Detection 1 Jun 2019 · 0 repositories
-
Speak up, Fight Back! Detection of Social Media Disclosures of Sexual Harassment 1 Jun 2019 · 0 repositories
-
An Empirical Evaluation of Text Representation Schemes on Multilingual Social Web to Filter the Textual Aggression 16 Apr 2019 · 0 repositories · arXiv:1904.08770
-
Low Resource Text Classification with ULMFit and Backtranslation 21 Mar 2019 · 1 repository · arXiv:1903.09244
-
Trellis Networks for Sequence Modeling 15 Oct 2018 · 1 repository · arXiv:1810.06682Syntology official (archive's flag): 7 ran · 7 ran (of which 0 constructed an object rather than computing a result; 6 with no instrument failure: 2 honoured, 0 violated, 4 with no contract checked; 1 where Syntology's instrument failed) · 1 unverified (of 8 harvested samples) · 2 pointer-only (licence)
-
Online Cyber-Attack Detection in Smart Grid: A Reinforcement Learning Approach 14 Sep 2018 · 1 repository · arXiv:1809.05258
-
Language Informed Modeling of Code-Switched Text 1 Jul 2018 · 0 repositories
-
Universal Language Model Fine-tuning for Text Classification 18 Jan 2018 · 66 repositories · arXiv:1801.06146Syntology community repositories only · 2 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 3 unverified (of 5 harvested samples) · 3 pointer-only (licence)
-
Breaking the Softmax Bottleneck: A High-Rank RNN Language Model 10 Nov 2017 · 9 repositories · arXiv:1711.03953Syntology official (archive's flag): 2 ran · 12 ran (of which 0 constructed an object rather than computing a result; 11 with no instrument failure: 0 honoured, 0 violated, 11 with no contract checked; 1 where Syntology's instrument failed) · 11 unverified (of 23 harvested samples) · 1 pointer-only (licence)
-
Regularizing and Optimizing LSTM Language Models 7 Aug 2017 · 45 repositories · arXiv:1708.02182Syntology community repositories only · 6 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 2 honoured, 0 violated, 0 with no contract checked; 4 where Syntology's instrument failed) · 1 unverified (of 7 harvested samples) · 7 pointer-only (licence)
-
The University of Edinburgh's Neural MT Systems for WMT17 2 Aug 2017 · 0 repositories · arXiv:1708.00726
-
Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modeling 4 Nov 2016 · 5 repositories · arXiv:1611.01462
-
Using the Output Embedding to Improve Language Models 20 Aug 2016 · 10 repositories · arXiv:1608.05859
Tasks archive 2025-07-28
20 shown of 76 tasks the archive attaches to papers tagged with this method, by distinct papers. A task without a page in the catalog is plain text.
Usage over time archive 2025-07-28
Components: the archive holds no method-to-method composition, so PwC's Components table cannot be rebuilt; the Papers list carries no Results column for the same reason (the archive does not join its leaderboard rows to method tags).
Categories archive 2025-07-28
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections