Methods › Natural Language Processing › Transformers › RoBERTa
RoBERTa
archive 2025-07-28 Description, source and code snippet are the archive's method entry.
RoBERTa is an extension of BERT with changes to the pretraining procedure. The modifications include:
- training the model longer, with bigger batches, over more data
- removing the next sentence prediction objective
- training on longer sequences
- dynamically changing the masking pattern applied to the training data. The authors also collect a large new dataset (CC-News) of comparable size to other privately used datasets, to better control for training set size effects
In Syntology repository known, no verified implementation (declared_code; repositories: 5 listed of 67 in the graph, strongest link: official). The archive credits no paper with an arXiv id for this method, so whether this answer (about arXiv:1907.11692) is about this method is not checked.
Origin paper of this answer, arXiv:1907.11692 (its page on this site · its page on Syntology): Syntology's run record for its harvested code, 37 ran (of which 11 constructed an object rather than computing a result; 36 with no instrument failure: 2 honoured, 0 violated, 34 with no contract checked; 1 where Syntology's instrument failed) · 11 unverified (of 48 harvested samples) · 24 pointer-only (licence). The instrument figure counts failures of Syntology's instrument, not of the code. Ran means executed on a synthesized input, not that the code is correct.
Syntology's have check for this name (free, it answers coverage only), read 2026-10-05; ask it again live. A level is the served verification level of that one answer; nothing here is summed across methods.
Papers archive 2025-07-28
archive papers tagged: 913 · with a code link: 399 · where Syntology ran a sample: 87 (62 with a run with no instrument failure, 25 where every run was a failure of Syntology's instrument) Syntology
Show: all tagged papersonly where code ran (87 of 913 tagged: 62 with a run with no instrument failure, 25 where every run was a failure of Syntology's instrument)
Page 1 of 10: papers 1 to 100 of 913, newest first by the archive's date (ties by slug), in archive order.
Repository counts are the archive's code-links table. A Syntology line states what Syntology ran from that paper's harvested code, as “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the instrument figure counts failures of Syntology's instrument, not of the code. It is per sample and not a correctness claim. When the archive marks a repository official for the paper, the line starts with that repository's state (the archive's flag, not a verdict on who wrote the code; “community repositories only” when every sample that ran came from a community repository, “official: no sample here; runs from other or unrecorded repositories” when some came from a repository the paper names or has in its text, or from none recorded); hover it for the repositories the samples that ran came from.
-
Rethinking the effects of data contamination in Code Intelligence 3 Jun 2025 · 0 repositories · arXiv:2506.02791
-
Evaluating AI capabilities in detecting conspiracy theories on YouTube 29 May 2025 · 1 repository · arXiv:2505.23570
-
Detection of Suicidal Risk on Social Media: A Hybrid Model 26 May 2025 · 0 repositories · arXiv:2505.23797
-
Hermes@DravidianLangTech 2025: Sentiment Analysis of Dravidian Languages using XLM-RoBERTa 25 May 2025 · 1 repository
-
Optimized Text Embedding Models and Benchmarks for Amharic Passage Retrieval 25 May 2025 · 1 repository · arXiv:2505.19356
-
Multi-Scale Probabilistic Generation Theory: A Hierarchical Framework for Interpreting Large Language Models 23 May 2025 · 0 repositories · arXiv:2505.18244
-
Climate Research Domain BERTs: Pretraining, Adaptation, and Evaluation 19 May 2025 · 0 repositories
-
Suicide Risk Assessment Using Multimodal Speech Features: A Study on the SW1 Challenge Dataset 19 May 2025 · 0 repositories · arXiv:2505.13069
-
KGAlign: Joint Semantic-Structural Knowledge Encoding for Multimodal Fake News Detection 18 May 2025 · 1 repository · arXiv:2505.14714
-
Comparative sentiment analysis of public perception: Monkeypox vs. COVID-19 behavioral insights 12 May 2025 · 0 repositories · arXiv:2505.07430
-
The Sound of Populism: Distinct Linguistic Features Across Populist Variants 10 May 2025 · 0 repositories · arXiv:2505.07874
-
Leveraging Language Models for Automated Patient Record Linkage 21 Apr 2025 · 0 repositories · arXiv:2504.15261
-
LLMs as Data Annotators: How Close Are We to Human Performance 21 Apr 2025 · 0 repositories · arXiv:2504.15022
-
The Synthetic Imputation Approach: Generating Optimal Synthetic Texts For Underrepresented Categories In Supervised Classification Tasks 21 Apr 2025 · 0 repositories · arXiv:2504.15160
-
ModernBERT or DeBERTaV3? Examining Architecture and Data Influence on Transformer Encoder Models Performance 11 Apr 2025 · 0 repositories · arXiv:2504.08716
-
Beyond LLMs: A Linguistic Approach to Causal Graph Generation from Narrative Texts 10 Apr 2025 · 0 repositories · arXiv:2504.07459
-
Exploring Generative AI Techniques in Government: A Case Study 6 Apr 2025 · 0 repositories · arXiv:2504.10497
-
A thorough benchmark of automatic text classification: From traditional approaches to large language models 2 Apr 2025 · 1 repository · arXiv:2504.01930
-
Context-Aware Toxicity Detection in Multiplayer Games: Integrating Domain-Adaptive Pretraining and Match Metadata 2 Apr 2025 · 1 repository · arXiv:2504.01534
-
Transformer-Based Named Entity Recognition for Automated Server Provisioning 1 Apr 2025 · 1 repository
-
Measuring Online Hate on 4chan using Pre-trained Deep Learning Models 30 Mar 2025 · 0 repositories · arXiv:2504.00045
-
Enhancing Recommender Systems Using Textual Embeddings from Pre-trained Language Models 24 Mar 2025 · 0 repositories · arXiv:2504.08746
-
LakotaBERT: A Transformer-based Model for Low Resource Lakota Language 23 Mar 2025 · 0 repositories · arXiv:2503.18212
-
Predicting Human Choice Between Textually Described Lotteries 18 Mar 2025 · 0 repositories · arXiv:2503.14004
-
An Evaluation of LLMs for Detecting Harmful Computing Terms 12 Mar 2025 · 0 repositories · arXiv:2503.09341
-
Multimodal Emotion Recognition and Sentiment Analysis in Multi-Party Conversation Contexts 9 Mar 2025 · 0 repositories · arXiv:2503.06805
-
Seeing Delta Parameters as JPEG Images: Data-Free Delta Compression with Discrete Cosine Transform 9 Mar 2025 · 0 repositories · arXiv:2503.06676
-
Constructions are Revealed in Word Distributions 8 Mar 2025 · 1 repository · arXiv:2503.06048
-
Evaluating open-source Large Language Models for automated fact-checking 7 Mar 2025 · 0 repositories · arXiv:2503.05565
-
Intermediate-Task Transfer Learning: Leveraging Sarcasm Detection for Stance Detection 5 Mar 2025 · 0 repositories · arXiv:2503.03172
-
Large language models in finance : what is financial sentiment? 5 Mar 2025 · 0 repositories · arXiv:2503.03612
-
Sarcasm Detection as a Catalyst: Improving Stance Detection with Cross-Target Capabilities 5 Mar 2025 · 0 repositories · arXiv:2503.03787
-
NeoBERT: A Next-Generation BERT 26 Feb 2025 · 1 repository · arXiv:2502.19587
-
Broadening Discovery through Structural Models: Multimodal Combination of Local and Structural Properties for Predicting Chemical Features 25 Feb 2025 · 0 repositories · arXiv:2502.17986
-
"Actionable Help" in Crises: A Novel Dataset and Resource-Efficient Models for Identifying Request and Offer Social Media Posts 24 Feb 2025 · 0 repositories · arXiv:2502.16839
-
Adversarial Training for Defense Against Label Poisoning Attacks 24 Feb 2025 · 1 repository · arXiv:2502.17121Syntology official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 1 harvested sample)
-
Iterative Auto-Annotation for Scientific Named Entity Recognition Using BERT-Based Models 22 Feb 2025 · 0 repositories · arXiv:2502.16312
-
Extraction multi-étiquettes de relations en utilisant des couches de Transformer 21 Feb 2025 · 0 repositories · arXiv:2502.15619
-
AdaSplash: Adaptive Sparse Flash Attention 17 Feb 2025 · 1 repository · arXiv:2502.12082Syntology official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 1 unverified (of 3 harvested samples)
-
Predicting Cognitive Decline: A Multimodal AI Approach to Dementia Screening from Speech 13 Feb 2025 · 0 repositories · arXiv:2502.08862
-
Making Language Models Robust Against Negation 11 Feb 2025 · 0 repositories · arXiv:2502.07717
-
AlphaAdam:Asynchronous Masked Optimization with Dynamic Alpha for Selective Updates 30 Jan 2025 · 0 repositories · arXiv:2501.18094
-
General Embedding vs. Task-Specific Embedding: A Comparative Approach to Enhancing NLP Performance 30 Jan 2025 · 0 repositories
-
A Comprehensive Study on Fine-Tuning Large Language Models for Medical Question Answering Using Classification Models and Comparative Analysis 27 Jan 2025 · 0 repositories · arXiv:2501.17190
-
TrustDataFilter:Leveraging Trusted Knowledge Base Data for More Effective Filtering of Unknown Information 25 Jan 2025 · 0 repositories · arXiv:2502.15714
-
A Transformer-based Autoregressive Decoder Architecture for Hierarchical Text Classification 23 Jan 2025 · 1 repository · arXiv:2501.13598
-
Expanding Vietnamese SentiWordNet to Improve Performance of Vietnamese Sentiment Analysis Models 15 Jan 2025 · 0 repositories · arXiv:2501.08758
-
Aligning Brain Activity with Advanced Transformer Models: Exploring the Role of Punctuation in Semantic Processing 10 Jan 2025 · 1 repository · arXiv:2501.06278
-
Text to Band Gap: Pre-trained Language Models as Encoders for Semiconductor Band Gap Prediction 7 Jan 2025 · 1 repository · arXiv:2501.03456
-
LLM Content Moderation and User Satisfaction: Evidence from Response Refusals in Chatbot Arena 4 Jan 2025 · 0 repositories · arXiv:2501.03266
-
Column Property Annotation using Large Language Models 1 Jan 2025 · 1 repository
-
Labels Generated by Large Language Model Helps Measuring People's Empathy in Vitro 1 Jan 2025 · 1 repository · arXiv:2501.00691
-
Dementia Detection using Multi-modal Methods on Audio Data 31 Dec 2024 · 0 repositories · arXiv:2501.00465
-
On Adversarial Robustness of Language Models in Transfer Learning 29 Dec 2024 · 0 repositories · arXiv:2501.00066
-
Assessing Text Classification Methods for Cyberbullying Detection on Social Media Platforms 27 Dec 2024 · 0 repositories · arXiv:2412.19928
-
Gradient Weight-normalized Low-rank Projection for Efficient LLM Training 27 Dec 2024 · 1 repository · arXiv:2412.19616
-
Injecting Bias into Text Classification Models using Backdoor Attacks 25 Dec 2024 · 0 repositories · arXiv:2412.18975
-
Unlocking the Potential of Multiple BERT Models for Bangla Question Answering in NCTB Textbooks 24 Dec 2024 · 0 repositories · arXiv:2412.18440
-
Identifying Cyberbullying Roles in Social Media 21 Dec 2024 · 0 repositories · arXiv:2412.16417
-
Adversarial Robustness through Dynamic Ensemble Learning 20 Dec 2024 · 0 repositories · arXiv:2412.16254
-
Decade of Natural Language Processing in Chronic Pain: A Systematic Review 19 Dec 2024 · 0 repositories · arXiv:2412.15360
-
Till the Layers Collapse: Compressing a Deep Neural Network through the Lenses of Batch Normalization Layers 19 Dec 2024 · 1 repository · arXiv:2412.15077
-
FarExStance: Explainable Stance Detection for Farsi 18 Dec 2024 · 2 repositories · arXiv:2412.14008
-
Optimized Quran Passage Retrieval Using an Expanded QA Dataset and Fine-Tuned Language Models 16 Dec 2024 · 0 repositories · arXiv:2412.11431
-
Tokens, the oft-overlooked appetizer: Large language models, the distributional hypothesis, and meaning 14 Dec 2024 · 0 repositories · arXiv:2412.10924
-
Advancing Single and Multi-task Text Classification through Large Language Model Fine-tuning 11 Dec 2024 · 0 repositories · arXiv:2412.08587
-
Su-RoBERTa: A Semi-supervised Approach to Predicting Suicide Risk through Social Media using Base Language Models 2 Dec 2024 · 0 repositories · arXiv:2412.01353
-
Development of Pre-Trained Transformer-based Models for the Nepali Language 24 Nov 2024 · 0 repositories · arXiv:2411.15734
-
A Comparative Analysis of Transformer and LSTM Models for Detecting Suicidal Ideation on Reddit 23 Nov 2024 · 1 repository · arXiv:2411.15404
-
Inducing Human-like Biases in Moral Reasoning Language Models 23 Nov 2024 · 0 repositories · arXiv:2411.15386
-
BERT-Based Approach for Automating Course Articulation Matrix Construction with Explainable AI 21 Nov 2024 · 1 repository · arXiv:2411.14254
-
Suicide Risk Assessment on Social Media with Semi-Supervised Learning 18 Nov 2024 · 0 repositories · arXiv:2411.12767
-
CamemBERT 2.0: A Smarter French Language Model Aged to Perfection 13 Nov 2024 · 0 repositories · arXiv:2411.08868
-
TempCharBERT: Keystroke Dynamics for Continuous Access Control Based on Pre-trained Language Models 11 Nov 2024 · 0 repositories · arXiv:2411.07224
-
HeartBERT: A Self-Supervised ECG Embedding Model for Efficient and Effective Medical Signal Analysis 8 Nov 2024 · 1 repository · arXiv:2411.11896
-
Words that Move Markets- Quantifying the Impact of RBI's Monetary Policy Communications on Indian Financial Market 7 Nov 2024 · 0 repositories · arXiv:2411.04808
-
Understanding the Effects of Human-written Paraphrases in LLM-generated Text Detection 6 Nov 2024 · 1 repository · arXiv:2411.03806
-
Enriching Tabular Data with Contextual LLM Embeddings: A Comprehensive Ablation Study for Ensemble Classifiers 3 Nov 2024 · 0 repositories · arXiv:2411.01645
-
ProTransformer: Robustify Transformers via Plug-and-Play Paradigm 30 Oct 2024 · 1 repository · arXiv:2410.23182
-
Meta-Learning Adaptable Foundation Models 29 Oct 2024 · 0 repositories · arXiv:2410.22264
-
KD-LoRA: A Hybrid Approach to Efficient Fine-Tuning with LoRA and Knowledge Distillation 28 Oct 2024 · 1 repository · arXiv:2410.20777
-
uOttawa at LegalLens-2024: Transformer-based Classification Experiments 28 Oct 2024 · 1 repository · arXiv:2410.21139
-
Natural GaLore: Accelerating GaLore for memory-efficient LLM Training and Fine-tuning 21 Oct 2024 · 1 repository · arXiv:2410.16029
-
Medical-GAT: Cancer Document Classification Leveraging Graph-Based Residual Network for Scenarios with Limited Data 19 Oct 2024 · 0 repositories · arXiv:2410.15198
-
Implicit Regularization of Sharpness-Aware Minimization for Scale-Invariant Problems 18 Oct 2024 · 0 repositories · arXiv:2410.14802
-
Sentiment Analysis Based on RoBERTa for Amazon Review: An Empirical Study on Decision Making 18 Oct 2024 · 0 repositories · arXiv:2411.00796
-
Detecting AI-Generated Texts in Cross-Domains 17 Oct 2024 · 1 repository · arXiv:2410.13966
-
Linguistically Grounded Analysis of Language Models using Shapley Head Values 17 Oct 2024 · 0 repositories · arXiv:2410.13396
-
LoLDU: Low-Rank Adaptation via Lower-Diag-Upper Decomposition for Parameter-Efficient Fine-Tuning 17 Oct 2024 · 1 repository · arXiv:2410.13618Syntology official (archive's flag): 4 ran · 4 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 2 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified (of 4 harvested samples) · 4 pointer-only (licence)
-
RoCoFT: Efficient Finetuning of Large Language Models with Row-Column Updates 14 Oct 2024 · 1 repository · arXiv:2410.10075
-
Will LLMs Replace the Encoder-Only Models in Temporal Relation Classification? 14 Oct 2024 · 1 repository · arXiv:2410.10476Syntology official (archive's flag): 5 ran · 5 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 0 honoured, 1 violated, 4 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified (of 6 harvested samples)
-
DICE: Discrete Inversion Enabling Controllable Editing for Multinomial Diffusion and Masked Generative Models 10 Oct 2024 · 0 repositories · arXiv:2410.08207
-
Fine-Tuning Language Models for Ethical Ambiguity: A Comparative Study of Alignment with Human Responses 10 Oct 2024 · 0 repositories · arXiv:2410.07826
-
Robust AI-Generated Text Detection by Restricted Embeddings 10 Oct 2024 · 1 repository · arXiv:2410.08113Syntology official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 1 where Syntology's instrument failed) · 3 unverified (of 6 harvested samples) · 6 pointer-only (licence)
-
SparseGrad: A Selective Method for Efficient Fine-tuning of MLP Layers 9 Oct 2024 · 1 repository · arXiv:2410.07383
-
A Comparative Study of Hybrid Models in Health Misinformation Text Classification 8 Oct 2024 · 0 repositories · arXiv:2410.06311
-
Learning Semantic Structure through First-Order-Logic Translation 4 Oct 2024 · 0 repositories · arXiv:2410.03203
-
Emotion-Aware Embedding Fusion in LLMs (Flan-T5, LLAMA 2, DeepSeek-R1, and ChatGPT 4) for Intelligent Response Generation 2 Oct 2024 · 0 repositories · arXiv:2410.01306
-
INSIGHTBUDDY-AI: Medication Extraction and Entity Linking using Large Language Models and Ensemble Learning 28 Sep 2024 · 2 repositories · arXiv:2409.19467
-
Suicide Phenotyping from Clinical Notes in Safety-Net Psychiatric Hospital Using Multi-Label Classification with Pre-Trained Language Models 27 Sep 2024 · 0 repositories · arXiv:2409.18878
Tasks archive 2025-07-28
20 shown of 477 tasks the archive attaches to papers tagged with this method, by distinct papers. A task without a page in the catalog is plain text.
Usage over time archive 2025-07-28
Components: the archive holds no method-to-method composition, so PwC's Components table cannot be rebuilt; the Papers list carries no Results column for the same reason (the archive does not join its leaderboard rows to method tags).
Categories archive 2025-07-28
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections