Methods › Natural Language Processing › Autoencoding Transformers › DeBERTa
DeBERTa
Introduced by Pengcheng He et al. in DeBERTa: Decoding-enhanced BERT with Disentangled Attention
archive 2025-07-28 Description, source and code snippet are the archive's method entry.
DeBERTa is a Transformer-based neural language model that aims to improve the BERT and RoBERTa models with two techniques: a disentangled attention mechanism and an enhanced mask decoder. The disentangled attention mechanism is where each word is represented unchanged using two vectors that encode its content and position, respectively, and the attention weights among words are computed using disentangle matrices on their contents and relative positions. The enhanced mask decoder is used to replace the output softmax layer to predict the masked tokens for model pre-training. In addition, a new virtual adversarial training method is used for fine-tuning to improve model’s generalization on downstream tasks.
In Syntology repository known, no verified implementation (declared_code; repositories: 5 listed of 11 in the graph, strongest link: official). This answer is about arXiv:2006.03654, the paper the archive credits with this method.
Origin paper of this answer, arXiv:2006.03654 (its page on this site · its page on Syntology): Syntology's run record for its harvested code, 4 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 2 honoured, 1 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 9 unverified (of 13 harvested samples) · 3 pointer-only (licence). The instrument figure counts failures of Syntology's instrument, not of the code. Ran means executed on a synthesized input, not that the code is correct.
Syntology's have check for this name (free, it answers coverage only), read 2026-10-05; ask it again live. A level is the served verification level of that one answer; nothing here is summed across methods.
Papers archive 2025-07-28
archive papers tagged: 90 · with a code link: 39 · where Syntology ran a sample: 8 (8 with a run with no instrument failure, 0 where every run was a failure of Syntology's instrument) Syntology
Show: all tagged papersonly where code ran (8 of 90 tagged: 8 with a run with no instrument failure, 0 where every run was a failure of Syntology's instrument)
Page 1 of 1: papers 1 to 90 of 90, newest first by the archive's date (ties by slug), in archive order.
Repository counts are the archive's code-links table. A Syntology line states what Syntology ran from that paper's harvested code, as “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the instrument figure counts failures of Syntology's instrument, not of the code. It is per sample and not a correctness claim. When the archive marks a repository official for the paper, the line starts with that repository's state (the archive's flag, not a verdict on who wrote the code; “community repositories only” when every sample that ran came from a community repository, “official: no sample here; runs from other or unrecorded repositories” when some came from a repository the paper names or has in its text, or from none recorded); hover it for the repositories the samples that ran came from.
-
AI Wizards at CheckThat! 2025: Enhancing Transformer-Based Embeddings with Sentiment for Subjectivity Detection in News Articles 15 Jul 2025 · 1 repository · arXiv:2507.11764
-
PlantBert: An Open Source Language Model for Plant Science 10 Jun 2025 · 0 repositories · arXiv:2506.08897
-
WeightLoRA: Keep Only Necessary Adapters 3 Jun 2025 · 0 repositories · arXiv:2506.02724
-
Decom-Renorm-Merge: Model Merging on the Right Space Improves Multitasking 29 May 2025 · 0 repositories · arXiv:2505.23117
-
Transformer-Based Named Entity Recognition for Automated Server Provisioning 1 Apr 2025 · 1 repository
-
Sarang at DEFACTIFY 4.0: Detecting AI-Generated Text Using Noised Data and an Ensemble of DeBERTa Models 24 Feb 2025 · 0 repositories · arXiv:2502.16857
-
Code-Mixed Telugu-English Hate Speech Detection 15 Feb 2025 · 0 repositories · arXiv:2502.10632
-
Zero-Shot Belief: A Hard Problem for LLMs 12 Feb 2025 · 0 repositories · arXiv:2502.08777
-
Feature Alignment-Based Knowledge Distillation for Efficient Compression of Large Language Models 27 Dec 2024 · 0 repositories · arXiv:2412.19449
-
Lightweight Safety Classification Using Pruned Language Models 18 Dec 2024 · 0 repositories · arXiv:2412.13435
-
SEKE: Specialised Experts for Keyword Extraction 18 Dec 2024 · 1 repository · arXiv:2412.14087
-
RAGulator: Lightweight Out-of-Context Detectors for Grounded Text Generation 6 Nov 2024 · 0 repositories · arXiv:2411.03920
-
Bonafide at LegalLens 2024 Shared Task: Using Lightweight DeBERTa Based Encoder For Legal Violation Detection and Resolution 30 Oct 2024 · 1 repository · arXiv:2410.22977
-
Evaluating Transformer Models for Suicide Risk Detection on Social Media 10 Oct 2024 · 0 repositories · arXiv:2410.08375
-
Large Language Model Inference Acceleration: A Comprehensive Hardware Perspective 6 Oct 2024 · 1 repository · arXiv:2410.04466
-
Multimodal Coherent Explanation Generation of Robot Failures 1 Oct 2024 · 1 repository · arXiv:2410.00659
-
Improving Academic Skills Assessment with NLP and Ensemble Learning 23 Sep 2024 · 0 repositories · arXiv:2409.19013
-
Instruct-DeBERTa: A Hybrid Approach for Aspect-based Sentiment Analysis on Textual Reviews 23 Aug 2024 · 0 repositories · arXiv:2408.13202
-
Scientific QA System with Verifiable Answers 16 Jul 2024 · 1 repository · arXiv:2407.11485
-
Turn-Level Empathy Prediction Using Psychological Indicators 11 Jul 2024 · 0 repositories · arXiv:2407.08607
-
SecureNet: A Comparative Study of DeBERTa and Large Language Models for Phishing Detection 10 Jun 2024 · 0 repositories · arXiv:2406.06663
-
BERTs are Generative In-Context Learners 7 Jun 2024 · 1 repository · arXiv:2406.04823Syntology official (archive's flag): 13 ran · 13 ran (of which 0 constructed an object rather than computing a result; 11 with no instrument failure: 1 honoured, 1 violated, 9 with no contract checked; 2 where Syntology's instrument failed) · 13 unverified (of 26 harvested samples)
-
Modeling Emotional Trajectories in Written Stories Utilizing Transformers and Weakly-Supervised Learning 4 Jun 2024 · 1 repository · arXiv:2406.02251
-
Explainable Automatic Grading with Neural Additive Models 1 May 2024 · 0 repositories · arXiv:2405.00489
-
Comparative Analysis of Deep Natural Networks and Large Language Models for Aspect-Based Sentiment Analysis 17 Apr 2024 · 1 repository
-
DKE-Research at SemEval-2024 Task 2: Incorporating Data Augmentation with Generative Models and Biomedical Knowledge to Enhance Inference Robustness 14 Apr 2024 · 0 repositories · arXiv:2404.09206
-
TLDR at SemEval-2024 Task 2: T5-generated clinical-Language summaries for DeBERTa Report Analysis 14 Apr 2024 · 1 repository · arXiv:2404.09136
-
BPDec: Unveiling the Potential of Masked Language Modeling Decoder in BERT pretraining 29 Jan 2024 · 0 repositories · arXiv:2401.15861
-
Enhancing Essay Scoring with Adversarial Weights Perturbation and Metric-specific AttentionPooling 6 Jan 2024 · 0 repositories · arXiv:2401.05433
-
More than Correlation: Do Large Language Models Learn Causal Representations of Space? 26 Dec 2023 · 0 repositories · arXiv:2312.16257
-
PROPRES: Investigating the Projectivity of Presupposition with Various Triggers and Environments 14 Dec 2023 · 1 repository · arXiv:2312.08755
-
Enhancing Sentiment Analysis Results through Outlier Detection Optimization 25 Nov 2023 · 0 repositories · arXiv:2311.16185
-
Deep Learning Brasil at ABSAPT 2022: Portuguese Transformer Ensemble Approaches 8 Nov 2023 · 1 repository · arXiv:2311.05051
-
DeepLearningBrasil@LT-EDI-2023: Exploring Deep Learning Techniques for Detecting Depression in Social Media Text 8 Nov 2023 · 1 repository · arXiv:2311.05047
-
Interpretable-by-Design Text Understanding with Iteratively Generated Concept Bottleneck 30 Oct 2023 · 1 repository · arXiv:2310.19660
-
An Ensemble Method Based on the Combination of Transformers with Convolutional Neural Networks to Detect Artificially Generated Text 26 Oct 2023 · 0 repositories · arXiv:2310.17312
-
A Comparative Study of Transformer-based Neural Text Representation Techniques on Bug Triaging 10 Oct 2023 · 0 repositories · arXiv:2310.06913
-
Structural Self-Supervised Objectives for Transformers 15 Sep 2023 · 1 repository · arXiv:2309.08272
-
ARC-NLP at PAN 2023: Transition-Focused Natural Language Inference for Writing Style Detection 27 Jul 2023 · 0 repositories · arXiv:2307.14913
-
Findings of Factify 2: Multimodal Fake News Detection 19 Jul 2023 · 0 repositories · arXiv:2307.10475
-
Math Word Problem Solving by Generating Linguistic Variants of Problem Statements 24 Jun 2023 · 1 repository · arXiv:2306.13899
-
Enhancing social network hate detection using back translation and GPT-3 augmentations during training and test-time 17 Jun 2023 · 1 repository
-
Neural models for Factual Inconsistency Classification with Explanations 15 Jun 2023 · 1 repository · arXiv:2306.08872
-
Enhancing Social Network Hate Detection Using Back Translation and GPT-3 Augmentations During Training and Test-Time 13 Jun 2023 · 1 repository
-
EaSyGuide : ESG Issue Identification Framework leveraging Abilities of Generative Large Language Models 11 Jun 2023 · 1 repository · arXiv:2306.06662
-
Data-Efficient French Language Modeling with CamemBERTa 2 Jun 2023 · 0 repositories · arXiv:2306.01497
-
ScoNe: Benchmarking Negation Reasoning in Language Models With Fine-Tuning and In-Context Learning 30 May 2023 · 1 repository · arXiv:2305.19426
-
Ahead-of-Time P-Tuning 18 May 2023 · 0 repositories · arXiv:2305.10835
-
Knowledge Rumination for Pre-trained Language Models 15 May 2023 · 1 repository · arXiv:2305.08732
-
Advancing Neural Encoding of Portuguese with Transformer Albertina PT-* 11 May 2023 · 0 repositories · arXiv:2305.06721
-
Stanford MLab at SemEval-2023 Task 10: Exploring GloVe- and Transformer-Based Methods for the Explainable Detection of Online Sexism 7 May 2023 · 0 repositories · arXiv:2305.04356
-
Evaluation of Social Biases in Recent Large Pre-Trained Models 13 Apr 2023 · 0 repositories · arXiv:2304.06861
-
Extracting Thyroid Nodules Characteristics from Ultrasound Reports Using Transformer-based Natural Language Processing Methods 31 Mar 2023 · 0 repositories · arXiv:2304.00115
-
Adapting Pretrained Language Models for Solving Tabular Prediction Problems in the Electronic Health Record 27 Mar 2023 · 1 repository · arXiv:2303.14920
-
Large Language Models in the Workplace: A Case Study on Prompt Engineering for Job Type Classification 13 Mar 2023 · 0 repositories · arXiv:2303.07142
-
Few-shot learning approaches for classifying low resource domain specific software requirements 14 Feb 2023 · 0 repositories · arXiv:2302.06951
-
ViDeBERTa: A powerful pre-trained language model for Vietnamese 25 Jan 2023 · 1 repository · arXiv:2301.10439Syntology official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 0 where Syntology's instrument failed) · 2 unverified (of 4 harvested samples)
-
InPars-Light: Cost-Effective Unsupervised Training of Efficient Rankers 8 Jan 2023 · 0 repositories · arXiv:2301.02998
-
Artificial Text Detection with Multiple Training Strategies 10 Dec 2022 · 0 repositories · arXiv:2212.05194
-
X-PuDu at SemEval-2022 Task 6: Multilingual Learning for English and Arabic Sarcasm Detection 30 Nov 2022 · 0 repositories · arXiv:2211.16883
-
MolE: a molecular foundation model for drug discovery 3 Nov 2022 · 0 repositories · arXiv:2211.02657
-
Two is Better than Many? Binary Classification as an Effective Approach to Multi-Choice Question Answering 29 Oct 2022 · 1 repository · arXiv:2210.16495Syntology official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 0 where Syntology's instrument failed) · 5 unverified (of 7 harvested samples)
-
The Better Your Syntax, the Better Your Semantics? Probing Pretrained Language Models for the English Comparative Correlative 24 Oct 2022 · 0 repositories · arXiv:2210.13181
-
Zero-Shot Ranking Socio-Political Texts with Transformer Language Models to Reduce Close Reading Time 17 Oct 2022 · 0 repositories · arXiv:2210.09179
-
A Simple and Strong Baseline for End-to-End Neural RST-style Discourse Parsing 15 Oct 2022 · 1 repository · arXiv:2210.08355
-
Vote'n'Rank: Revision of Benchmarking with Social Choice Theory 11 Oct 2022 · 1 repository · arXiv:2210.05769Syntology official (archive's flag): 6 ran · 6 ran (of which 0 constructed an object rather than computing a result; 6 with no instrument failure: 0 honoured, 0 violated, 6 with no contract checked; 0 where Syntology's instrument failed) · 11 unverified (of 17 harvested samples)
-
ByteTransformer: A High-Performance Transformer Boosted for Variable-Length Inputs 6 Oct 2022 · 1 repository · arXiv:2210.03052
-
Detecting Generated Scientific Papers using an Ensemble of Transformer Models 17 Sep 2022 · 1 repository · arXiv:2209.08283
-
Probing for Understanding of English Verb Classes and Alternations in Large Pre-trained Language Models 11 Sep 2022 · 0 repositories · arXiv:2209.04811
-
Is Your Model Sensitive? SPeDaC: A New Benchmark for Detecting and Classifying Sensitive Personal Data 12 Aug 2022 · 0 repositories · arXiv:2208.06216
-
BEIKE NLP at SemEval-2022 Task 4: Prompt-Based Paragraph Classification for Patronizing and Condescending Language Detection 2 Aug 2022 · 0 repositories · arXiv:2208.01312
-
giMLPs: Gate with Inhibition Mechanism in MLPs 1 Aug 2022 · 1 repository · arXiv:2208.00929
-
A Cognitive Study on Semantic Similarity Analysis of Large Corpora: A Transformer-based Approach 24 Jul 2022 · 0 repositories · arXiv:2207.11716
-
A Systematic Review and Replicability Study of BERT4Rec for Sequential Recommendation 15 Jul 2022 · 1 repository · arXiv:2207.07483Syntology official (archive's flag): 2 ran · 4 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 1 honoured, 1 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified (of 4 harvested samples) · 4 pointer-only (licence)
-
A Novel DeBERTa-based Model for Financial Question Answering Task 12 Jul 2022 · 0 repositories · arXiv:2207.05875
-
Aspect-Based Sentiment Analysis using Local Context Focus Mechanism with DeBERTa 6 Jul 2022 · 0 repositories · arXiv:2207.02424
-
KnowDA: All-in-One Knowledge Mixture Model for Data Augmentation in Low-Resource NLP 21 Jun 2022 · 0 repositories · arXiv:2206.10265
-
Fewer Errors, but More Stereotypes? The Effect of Model Size on Gender Bias 20 Jun 2022 · 1 repository · arXiv:2206.09860
-
SciDeBERTa: Learning DeBERTa for Science Technology Documents and Fine-Tuning Information Extraction Tasks 8 Jun 2022 · 1 repository
-
RigoBERTa: A State-of-the-Art Language Model For Spanish 27 Apr 2022 · 0 repositories · arXiv:2205.10233
-
Autoencoding Language Model Based Ensemble Learning for Commonsense Validation and Explanation 7 Apr 2022 · 0 repositories · arXiv:2204.03324
-
DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing 18 Nov 2021 · 3 repositories · arXiv:2111.09543Syntology official (archive's flag): 6 ran · 6 ran (of which 0 constructed an object rather than computing a result; 6 with no instrument failure: 0 honoured, 0 violated, 6 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified (of 7 harvested samples)
-
abcbpc at SemEval-2021 Task 7: ERNIE-based Multi-task Model for Detecting and Rating Humor and Offense 1 Aug 2021 · 0 repositories
-
Duluth at SemEval-2021 Task 11: Applying DeBERTa to Contributing Sentence Selection and Dependency Parsing for Entity Extraction 1 Aug 2021 · 0 repositories
-
HumorHunter at SemEval-2021 Task 7: Humor and Offense Recognition with Disentangled Attention 1 Aug 2021 · 0 repositories
-
LeCun at SemEval-2021 Task 6: Detecting Persuasion Techniques in Text Using Ensembled Pretrained Transformers and Data Augmentation 1 Aug 2021 · 0 repositories
-
Aspect-based Sentiment Analysis using BERT with Disentangled Attention 18 Jul 2021 · 1 repository
-
LoRA: Low-Rank Adaptation of Large Language Models 17 Jun 2021 · 74 repositories · arXiv:2106.09685Syntology community repositories only · 51 ran (of which 19 constructed an object rather than computing a result; 43 with no instrument failure: 1 honoured, 0 violated, 42 with no contract checked; 8 where Syntology's instrument failed) · 33 unverified (of 84 harvested samples) · 30 pointer-only (licence)
-
SAS: Self-Augmentation Strategy for Language Model Pre-training 14 Jun 2021 · 1 repository · arXiv:2106.07176
-
DeBERTa: Decoding-enhanced BERT with Disentangled Attention 5 Jun 2020 · 14 repositories · arXiv:2006.03654Syntology official: no sample here; runs from other or unrecorded repositories · 4 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 2 honoured, 1 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 9 unverified (of 13 harvested samples) · 3 pointer-only (licence)
Tasks archive 2025-07-28
20 shown of 133 tasks the archive attaches to papers tagged with this method, by distinct papers. A task without a page in the catalog is plain text.
Usage over time archive 2025-07-28
Components: the archive holds no method-to-method composition, so PwC's Components table cannot be rebuilt; the Papers list carries no Results column for the same reason (the archive does not join its leaderboard rows to method tags).
Categories archive 2025-07-28
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections