Methods › Natural Language Processing › Autoencoding Transformers › Longformer
Longformer
Introduced by Iz Beltagy et al. in Longformer: The Long-Document Transformer
archive 2025-07-28 Description, source and code snippet are the archive's method entry.
Longformer is a modified Transformer architecture. Traditional Transformer-based models are unable to process long sequences due to their self-attention operation, which scales quadratically with the sequence length. To address this, Longformer uses an attention pattern that scales linearly with sequence length, making it easy to process documents of thousands of tokens or longer. The attention mechanism is a drop-in replacement for the standard self-attention and combines a local windowed attention with a task motivated global attention.
The attention patterns utilised include: sliding window attention, dilated sliding window attention and global + sliding window. These can be viewed in the components section of this page.
In Syntology repository known, no verified implementation (declared_code; repositories: 5 listed of 22 in the graph, strongest link: official). This answer is about arXiv:2004.05150, the paper the archive credits with this method.
Origin paper of this answer, arXiv:2004.05150 (its page on this site · its page on Syntology): Syntology's run record for its harvested code, 22 ran (of which 4 constructed an object rather than computing a result; 14 with no instrument failure: 1 honoured, 1 violated, 12 with no contract checked; 8 where Syntology's instrument failed) · 13 unverified (of 35 harvested samples) · 5 pointer-only (licence). The instrument figure counts failures of Syntology's instrument, not of the code. Ran means executed on a synthesized input, not that the code is correct.
Syntology's have check for this name (free, it answers coverage only), read 2026-10-05; ask it again live. A level is the served verification level of that one answer; nothing here is summed across methods.
Papers archive 2025-07-28
archive papers tagged: 87 · with a code link: 42 · where Syntology ran a sample: 10 (7 with a run with no instrument failure, 3 where every run was a failure of Syntology's instrument) Syntology
Show: all tagged papersonly where code ran (10 of 87 tagged: 7 with a run with no instrument failure, 3 where every run was a failure of Syntology's instrument)
Page 1 of 1: papers 1 to 87 of 87, newest first by the archive's date (ties by slug), in archive order.
Repository counts are the archive's code-links table. A Syntology line states what Syntology ran from that paper's harvested code, as “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the instrument figure counts failures of Syntology's instrument, not of the code. It is per sample and not a correctness claim. When the archive marks a repository official for the paper, the line starts with that repository's state (the archive's flag, not a verdict on who wrote the code; “community repositories only” when every sample that ran came from a community repository, “official: no sample here; runs from other or unrecorded repositories” when some came from a repository the paper names or has in its text, or from none recorded); hover it for the repositories the samples that ran came from.
-
I Know Which LLM Wrote Your Code Last Summer: LLM generated Code Stylometry for Authorship Attribution 18 Jun 2025 · 0 repositories · arXiv:2506.17323
-
Enhancing Abstractive Summarization of Scientific Papers Using Structure Information 20 May 2025 · 1 repository · arXiv:2505.14179
-
CacheFormer: High Attention-Based Segment Caching 18 Apr 2025 · 0 repositories · arXiv:2504.13981
-
ARLED: Leveraging LED-based ARMAN Model for Abstractive Summarization of Persian Long Documents 13 Mar 2025 · 0 repositories · arXiv:2503.10233
-
Understanding Players as if They Are Talking to the Game in a Customized Language: A Pilot Study 24 Oct 2024 · 0 repositories · arXiv:2410.18605
-
Extra Global Attention Designation Using Keyword Detection in Sparse Transformer Architectures 11 Oct 2024 · 0 repositories · arXiv:2410.08971
-
The Accuracy Paradox in RLHF: When Better Reward Models Don't Yield Better Language Models 9 Oct 2024 · 1 repository · arXiv:2410.06554Syntology official (archive's flag): 5 ran · 5 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 5 where Syntology's instrument failed) · 4 unverified (of 9 harvested samples)
-
RICo: Reddit ideological communities 5 Jun 2024 · 1 repository
-
Transfer Learning in Pre-Trained Large Language Models for Malware Detection Based on System Calls 15 May 2024 · 0 repositories · arXiv:2405.09318
-
Advancing AI with Integrity: Ethical Challenges and Solutions in Neural Machine Translation 1 Apr 2024 · 0 repositories · arXiv:2404.01070
-
A multi-cohort study on prediction of acute brain dysfunction states using selective state space models 11 Mar 2024 · 0 repositories · arXiv:2403.07201
-
Adaptation of Biomedical and Clinical Pretrained Models to French Long Documents: A Comparative Study 26 Feb 2024 · 1 repository · arXiv:2402.16689
-
Accurate and Well-Calibrated ICD Code Assignment Through Attention Over Diverse Label Embeddings 5 Feb 2024 · 1 repository · arXiv:2402.03172Syntology official (archive's flag): 7 ran · 7 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 1 honoured, 0 violated, 2 with no contract checked; 4 where Syntology's instrument failed) · 0 unverified (of 7 harvested samples)
-
UniMem: Towards a Unified View of Long-Context Large Language Models 5 Feb 2024 · 1 repository · arXiv:2402.03009
-
Enhanced Labeling Technique for Reddit Text and Fine-Tuned Longformer Models for Classifying Depression Severity in English and Luganda 25 Jan 2024 · 0 repositories · arXiv:2401.14240
-
Exploring Automatic Text Simplification of German Narrative Documents 15 Dec 2023 · 1 repository · arXiv:2312.09907
-
LLVMs4Protest: Harnessing the Power of Large Language and Vision Models for Deciphering Protests in the News 30 Nov 2023 · 1 repository · arXiv:2311.18241
-
Towards Harmful Erotic Content Detection through Coreference-Driven Contextual Analysis 22 Oct 2023 · 0 repositories · arXiv:2310.14325
-
Multi-level Contrastive Learning for Script-based Character Understanding 20 Oct 2023 · 1 repository · arXiv:2310.13231
-
Improving Long Document Topic Segmentation Models With Enhanced Coherence Modeling 18 Oct 2023 · 1 repository · arXiv:2310.11772
-
Hallucination Reduction in Long Input Text Summarization 28 Sep 2023 · 1 repository · arXiv:2309.16781
-
An NLP Benchmark Dataset for Assessing Corporate Climate Policy Engagement 26 Sep 2023 · 0 repositories
-
Language Models for Novelty Detection in System Call Traces 5 Sep 2023 · 0 repositories · arXiv:2309.02206
-
Local Large Language Models for Complex Structured Medical Tasks 3 Aug 2023 · 1 repository · arXiv:2308.01727
-
Can Model Fusing Help Transformers in Long Document Classification? An Empirical Study 18 Jul 2023 · 1 repository · arXiv:2307.09532
-
NOWJ at COLIEE 2023 -- Multi-Task and Ensemble Approaches in Legal Information Processing 8 Jun 2023 · 0 repositories · arXiv:2306.04903
-
MultiLegalPile: A 689GB Multilingual Legal Corpus 3 Jun 2023 · 0 repositories · arXiv:2306.02069
-
DEPLAIN: A German Parallel Corpus with Intralingual Translations into Plain Language for Sentence and Document Simplification 30 May 2023 · 1 repository · arXiv:2305.18939
-
Incorporating Distributions of Discourse Structure for Long Document Abstractive Summarization 26 May 2023 · 0 repositories · arXiv:2305.16784
-
Neural Summarization of Electronic Health Records 24 May 2023 · 0 repositories · arXiv:2305.15222
-
Text Is All You Need: Learning Language Representations for Sequential Recommendation 23 May 2023 · 1 repository · arXiv:2305.13731Syntology 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 1 harvested sample) · 1 pointer-only (licence)
-
A General-Purpose Multilingual Document Encoder 11 May 2023 · 1 repository · arXiv:2305.07016
-
Predicting COVID-19 and pneumonia complications from admission texts 5 May 2023 · 0 repositories · arXiv:2305.03661
-
Leveraging BERT Language Model for Arabic Long Document Classification 4 May 2023 · 0 repositories · arXiv:2305.03519
-
Unlimiformer: Long-Range Transformers with Unlimited Length Input 2 May 2023 · 2 repositories · arXiv:2305.01625Syntology official (archive's flag): 4 ran · 4 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 3 where Syntology's instrument failed) · 1 unverified (of 5 harvested samples)
-
Domain-specific Continued Pretraining of Language Models for Capturing Long Context in Mental Health 20 Apr 2023 · 0 repositories · arXiv:2304.10447
-
Multi-class Categorization of Reasons behind Mental Disturbance in Long Texts 8 Apr 2023 · 0 repositories · arXiv:2304.04118
-
Finding the Needle in a Haystack: Unsupervised Rationale Extraction from Long Text Classifiers 14 Mar 2023 · 0 repositories · arXiv:2303.07991
-
Is In-hospital Meta-information Useful for Abstractive Discharge Summary Generation? 10 Mar 2023 · 0 repositories · arXiv:2303.06002
-
VuLASTE: Long Sequence Model with Abstract Syntax Tree Embedding for vulnerability Detection 5 Feb 2023 · 0 repositories · arXiv:2302.02345
-
Longformer: Longitudinal Transformer for Alzheimer's Disease Classification with Structural MRIs 2 Feb 2023 · 1 repository · arXiv:2302.00901
-
A Comparative Study of Pretrained Language Models for Long Clinical Text 27 Jan 2023 · 1 repository · arXiv:2301.11847
-
Understanding Political Polarisation using Language Models: A dataset and method 2 Jan 2023 · 0 repositories · arXiv:2301.00891
-
Exploiting Rich Textual User-Product Context for Improving Sentiment Analysis 17 Dec 2022 · 0 repositories · arXiv:2212.08888
-
BudgetLongformer: Can we Cheaply Pretrain a SotA Legal Language Model From Scratch? 30 Nov 2022 · 0 repositories · arXiv:2211.17135
-
Processing Long Legal Documents with Pre-trained Transformers: Modding LegalBERT and Longformer 2 Nov 2022 · 0 repositories · arXiv:2211.00974
-
ClassActionPrediction: A Challenging Benchmark for Legal Judgment Prediction of Class Action Cases in the US 1 Nov 2022 · 1 repository · arXiv:2211.00582
-
How Long Is Enough? Exploring the Optimal Intervals of Long-Range Clinical Note Language Modeling 25 Oct 2022 · 1 repository · arXiv:2211.07713
-
LittleBird: Efficient Faster & Longer Transformer for Question Answering 21 Oct 2022 · 0 repositories · arXiv:2210.11870
-
Multi-granularity Argument Mining in Legal Texts 17 Oct 2022 · 0 repositories · arXiv:2210.09472
-
An Exploration of Hierarchical Attention Transformers for Efficient Long Document Classification 11 Oct 2022 · 0 repositories · arXiv:2210.05529
-
Knowledge Injected Prompt Based Fine-tuning for Multi-label Few-shot ICD Coding 7 Oct 2022 · 1 repository · arXiv:2210.03304Syntology official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 1 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 2 harvested samples)
-
Code Comment Inconsistency Detection with BERT and Longformer 29 Jul 2022 · 1 repository · arXiv:2207.14444
-
SALO: An Efficient Spatial Accelerator Enabling Hybrid Sparse Attention Mechanisms for Long Sequences 29 Jun 2022 · 0 repositories · arXiv:2206.14550
-
Abstractive summarization of hospitalisation histories with transformer networks 5 Apr 2022 · 0 repositories · arXiv:2204.02208
-
The NLP Task Effectiveness of Long-Range Transformers 16 Feb 2022 · 0 repositories · arXiv:2202.07856
-
Clinical-Longformer and Clinical-BigBird: Transformers for long clinical sequences 27 Jan 2022 · 1 repository · arXiv:2201.11838
-
Hierarchical Neural Network Approaches for Long Document Classification 18 Jan 2022 · 0 repositories · arXiv:2201.06774
-
Simple Local Attentions Remain Competitive for Long-Context Tasks 16 Jan 2022 · 0 repositories
-
SCROLLS: Standardized CompaRison Over Long Language Sequences 10 Jan 2022 · 2 repositories · arXiv:2201.03533Syntology official (archive's flag): 6 ran · 6 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 0 honoured, 0 violated, 4 with no contract checked; 2 where Syntology's instrument failed) · 2 unverified (of 8 harvested samples)
-
Keyphrase Generation Beyond the Boundaries of Title and Abstract 13 Dec 2021 · 1 repository · arXiv:2112.06776
-
PLSUM: Generating PT-BR Wikipedia by Summarizing Multiple Websites 2 Dec 2021 · 1 repository · arXiv:2112.01591
-
New Approaches to Long Document Summarization: Fourier Transform Based Attention in a Transformer Model 25 Nov 2021 · 0 repositories · arXiv:2111.15473
-
MarCQAp: Effective Context Modeling for Conversational Question Answering 16 Nov 2021 · 0 repositories
-
PRIMERA: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization 16 Oct 2021 · 3 repositories · arXiv:2110.08499Syntology official (archive's flag): 4 ran · 7 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 1 honoured, 0 violated, 3 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified (of 7 harvested samples)
-
An Exploratory Study on Long Dialogue Summarization: What Works and What's Next 10 Sep 2021 · 1 repository · arXiv:2109.04609
-
BumbleBee: A Transformer for Music 7 Jul 2021 · 0 repositories · arXiv:2107.03443
-
Efficient Transformer for Direct Speech Translation 7 Jul 2021 · 0 repositories · arXiv:2107.03069
-
What Helps Transformers Recognize Conversational Structure? Importance of Context, Punctuation, and Labels in Dialog Act Recognition 5 Jul 2021 · 1 repository · arXiv:2107.02294
-
Self-Supervised Document Similarity Ranking via Contextualized Language Models and Hierarchical Inference 2 Jun 2021 · 1 repository · arXiv:2106.01186
-
CiteWorth: Cite-Worthiness Detection for Improved Scientific Document Understanding 23 May 2021 · 1 repository · arXiv:2105.10912
-
NLP-IIS@UT at SemEval-2021 Task 4: Machine Reading Comprehension using the Long Document Transformer 8 May 2021 · 0 repositories · arXiv:2105.03775
-
Text Guide: Improving the quality of long text classification by a text selection method based on feature importance 15 Apr 2021 · 1 repository · arXiv:2104.07225
-
NLQuAD: A Non-Factoid Long Question Answering Data Set 1 Apr 2021 · 1 repository
-
Multi-Scale Vision Longformer: A New Vision Transformer for High-Resolution Image Encoding 29 Mar 2021 · 3 repositories · arXiv:2103.15358Syntology official (archive's flag): 3 ran · 8 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 1 honoured, 2 violated, 2 with no contract checked; 3 where Syntology's instrument failed) · 2 unverified (of 10 harvested samples) · 2 pointer-only (licence)
-
A Practical Survey on Faster and Lighter Transformers 26 Mar 2021 · 0 repositories · arXiv:2103.14636
-
Identifying Machine-Paraphrased Plagiarism 22 Mar 2021 · 2 repositories · arXiv:2103.11909
-
Introduction to Neural Transfer Learning with Transformers for Social Science Text Analysis 3 Feb 2021 · 0 repositories · arXiv:2102.02111
-
Grid Search Hyperparameter Benchmarking of BERT, ALBERT, and LongFormer on DuoRC 15 Jan 2021 · 0 repositories · arXiv:2101.06326
-
CDLM: Cross-Document Language Modeling 2 Jan 2021 · 2 repositories · arXiv:2101.00406
-
Pre-training Protein Language Models with Label-Agnostic Binding Pairs Enhances Performance in Downstream Tasks 5 Dec 2020 · 1 repository · arXiv:2012.03084
-
DebateSum: A large-scale argument mining and summarization dataset 14 Nov 2020 · 3 repositories · arXiv:2011.07251Syntology official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 1 harvested sample) · 1 pointer-only (licence)
-
Longformer for MS MARCO Document Re-ranking Task 20 Sep 2020 · 1 repository · arXiv:2009.09392
-
Efficient Transformers: A Survey 14 Sep 2020 · 0 repositories · arXiv:2009.06732
-
Fine-Tune Longformer for Jointly Predicting Rumor Stance and Veracity 15 Jul 2020 · 0 repositories · arXiv:2007.07803
-
Document Classification for COVID-19 Literature 15 Jun 2020 · 1 repository · arXiv:2006.13816
-
Longformer: The Long-Document Transformer 10 Apr 2020 · 22 repositories · arXiv:2004.05150Syntology official (archive's flag): 9 ran · 22 ran (of which 4 constructed an object rather than computing a result; 14 with no instrument failure: 1 honoured, 1 violated, 12 with no contract checked; 8 where Syntology's instrument failed) · 13 unverified (of 35 harvested samples) · 5 pointer-only (licence)
Tasks archive 2025-07-28
20 shown of 111 tasks the archive attaches to papers tagged with this method, by distinct papers. A task without a page in the catalog is plain text.
Usage over time archive 2025-07-28
Components: the archive holds no method-to-method composition, so PwC's Components table cannot be rebuilt; the Papers list carries no Results column for the same reason (the archive does not join its leaderboard rows to method tags).
Categories archive 2025-07-28
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections