Papers › Pay Less Attention with Lightweight and Dynamic Convolutions

Pay Less Attention with Lightweight and Dynamic Convolutions

29 Jan 2019ICLR 2019 5arXiv:1901.10430archive 2025-07-28

Felix Wu, Angela Fan, Alexei Baevski, Yann N. Dauphin, Michael Auli

Self-attention is a useful mechanism to build generative models for language and images. It determines the importance of context elements by comparing each element to the current time step. In this paper, we show that a very lightweight convolution can perform competitively to the best reported self-attention results. Next, we introduce dynamic convolutions which are simpler and more efficient than self-attention. We predict separate convolution kernels based solely on the current time-step in order to determine the importance of context elements. The number of operations required by this approach scales linearly in the input length, whereas self-attention is quadratic. Experiments on large-scale machine translation, language modeling and abstractive summarization show that dynamic convolutions improve over strong self-attention models. On the WMT'14 English-German test set dynamic convolutions achieve a new state of the art of 29.7 BLEU.

PaperPDFConference PDFCode

In Syntology View this paper on Syntology, its page in Syntology's graph. That page lists the repositories linked to the paper, the abstract and the calls for agents.

Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.

For agents, on Syntology's MCP service (how to connect):

Code

pytorch/fairseq officialmentioned in paperpytorchSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to LEWIS: Levenshtein Editing for Unsupervised Text Style Transfer (arXiv:2105.08206; that paper's own run record: 1 ran (of which 1 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified; the one sample that ran constructed an object rather than computing a result), Cascaded Head-colliding Attention (arXiv:2105.14850; that paper's own run record: 9 ran (of which 4 constructed an object rather than computing a result; 4 with no instrument failure: 1 honoured, 1 violated, 2 with no contract checked; 5 where Syntology's instrument failed) · 1 unverified) and Modeling Dual Read/Write Paths for Simultaneous Machine Translation (arXiv:2203.09163; that paper's own run record: 4 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 2 where Syntology's instrument failed) · 5 unverified); nothing checks that they implement this paper's method. report
bytedance/neurst mentioned on GitHubtfSyntology: not harvested report
dqqcasia/st mentioned on GitHubtfSyntology: not harvested report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

A paper named beside a repository with no sample linked to this paper is shown with that paper's own run record, not this paper's: “ran” means executed on a synthesized input, not that the code is correct, and “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. Papers are listed in arXiv-id order, at most three per repository.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Abstractive Text SummarizationLanguage ModelingLanguage ModellingMachine TranslationTranslation

Results from the paper archive 2025-07-28

TaskDatasetModelMetricValueRank at snapshotLeaderboardReport
Abstractive Text Summarization CNN / Daily Mail Dynamic Conv ROUGE-1 39.84 #45 of 53 Archive leaderboard report
Abstractive Text Summarization CNN / Daily Mail Dynamic Conv ROUGE-2 16.25 #45 of 53 Archive leaderboard report
Abstractive Text Summarization CNN / Daily Mail Dynamic Conv ROUGE-L 36.73 #45 of 53 Archive leaderboard report
Document Summarization CNN / Daily Mail DynamicConv ROUGE-1 39.84 #21 of 26 Archive leaderboard report
Document Summarization CNN / Daily Mail DynamicConv ROUGE-2 16.25 #21 of 26 Archive leaderboard report
Document Summarization CNN / Daily Mail DynamicConv ROUGE-L 36.73 #21 of 26 Archive leaderboard report
Document Summarization CNN / Daily Mail LightConv ROUGE-1 39.52 #22 of 26 Archive leaderboard report
Document Summarization CNN / Daily Mail LightConv ROUGE-2 15.97 #22 of 26 Archive leaderboard report
Document Summarization CNN / Daily Mail LightConv ROUGE-L 36.51 #22 of 26 Archive leaderboard report
Language Modelling One Billion Word DynamicConv Number of params 0.34B #15 of 27 Archive leaderboard report
Language Modelling One Billion Word DynamicConv PPL 26.67 #15 of 27 Archive leaderboard report
Machine Translation IWSLT2014 German-English DynamicConv BLEU score 35.2 #24 of 34 Archive leaderboard report
Machine Translation IWSLT2014 German-English LightConv BLEU score 34.8 #26 of 34 Archive leaderboard report
Machine Translation WMT 2017 English-Chinese DynamicConv BLEU score 24.4 #1 of 3 Archive leaderboard report
Machine Translation WMT 2017 English-Chinese LightConv BLEU score 24.3 #2 of 3 Archive leaderboard report
Machine Translation WMT2014 English-French DynamicConv BLEU score 43.2 #13 of 57 Archive leaderboard report
Machine Translation WMT2014 English-French LightConv BLEU score 43.1 #15 of 57 Archive leaderboard report
Machine Translation WMT2014 English-German DynamicConv BLEU score 29.7 #18 of 91 Archive leaderboard report
Machine Translation WMT2014 English-German DynamicConv Number of Params 213M #18 of 91 Archive leaderboard report
Machine Translation WMT2014 English-German LightConv BLEU score 28.9 #35 of 91 Archive leaderboard report
Machine Translation WMT2014 English-German LightConv Number of Params 202M #35 of 91 Archive leaderboard report

Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.

Methods

Introduced by this paper: DynamicConv

ConvolutionDepthwise ConvolutionDropConnectDynamicConvGated Linear UnitLightConvLinear LayerSoftmax

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections