Methods › Natural Language Processing › Transformers › CodeBERT
CodeBERT
archive 2025-07-28 Description, source and code snippet are the archive's method entry.
CodeBERT is a bimodal pre-trained model for programming language (PL) and natural language (NL). CodeBERT learns general-purpose representations that support downstream NL-PL applications such as natural language code search, code documentation generation, etc. CodeBERT is developed with a Transformer-based neural architecture, and is trained with a hybrid objective function that incorporates the pre-training task of replaced token detection, which is to detect plausible alternatives sampled from generators. This enables the utilization of both bimodal data of NL-PL pairs and unimodal data, where the former provides input tokens for model training while the latter helps to learn better generators.
In Syntology repository known, no verified implementation (declared_code; repositories: 5 listed of 10 in the graph, strongest link: official). The archive credits no paper with an arXiv id for this method, so whether this answer (about arXiv:2002.08155) is about this method is not checked.
Origin paper of this answer, arXiv:2002.08155 (its page on this site · its page on Syntology): Syntology's run record for its harvested code, 14 ran (of which 0 constructed an object rather than computing a result; 14 with no instrument failure: 0 honoured, 0 violated, 14 with no contract checked; 0 where Syntology's instrument failed) · 6 unverified (of 20 harvested samples) · 4 pointer-only (licence). The instrument figure counts failures of Syntology's instrument, not of the code. Ran means executed on a synthesized input, not that the code is correct.
Syntology's have check for this name (free, it answers coverage only), read 2026-10-05; ask it again live. A level is the served verification level of that one answer; nothing here is summed across methods.
Papers archive 2025-07-28
archive papers tagged: 66 · with a code link: 27 · where Syntology ran a sample: 3 (3 with a run with no instrument failure, 0 where every run was a failure of Syntology's instrument) Syntology
Show: all tagged papersonly where code ran (3 of 66 tagged: 3 with a run with no instrument failure, 0 where every run was a failure of Syntology's instrument)
Page 1 of 1: papers 1 to 66 of 66, newest first by the archive's date (ties by slug), in archive order.
Repository counts are the archive's code-links table. A Syntology line states what Syntology ran from that paper's harvested code, as “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the instrument figure counts failures of Syntology's instrument, not of the code. It is per sample and not a correctness claim. When the archive marks a repository official for the paper, the line starts with that repository's state (the archive's flag, not a verdict on who wrote the code; “community repositories only” when every sample that ran came from a community repository, “official: no sample here; runs from other or unrecorded repositories” when some came from a repository the paper names or has in its text, or from none recorded); hover it for the repositories the samples that ran came from.
-
A Multi-Dataset Evaluation of Models for Automated Vulnerability Repair 5 Jun 2025 · 0 repositories · arXiv:2506.04987
-
LoRACode: LoRA Adapters for Code Embeddings 7 Mar 2025 · 0 repositories · arXiv:2503.05315
-
Poster: Long PHP webshell files detection based on sliding window attention 26 Feb 2025 · 1 repository · arXiv:2502.19257
-
Less is More: On the Importance of Data Quality for Unit Test Generation 20 Feb 2025 · 0 repositories · arXiv:2502.14212
-
LLMs in Software Security: A Survey of Vulnerability Detection Techniques and Insights 10 Feb 2025 · 0 repositories · arXiv:2502.07049
-
Repository-level Code Search with Neural Retrieval Methods 10 Feb 2025 · 1 repository · arXiv:2502.07067
-
How to Select Pre-Trained Code Models for Reuse? A Learning Perspective 7 Jan 2025 · 1 repository · arXiv:2501.03783
-
EnStack: An Ensemble Stacking Framework of Large Language Models for Enhanced Vulnerability Detection in Source Code 25 Nov 2024 · 0 repositories · arXiv:2411.16561
-
CodeSAM: Source Code Representation Learning by Infusing Self-Attention with Multi-Code-View Graphs 21 Nov 2024 · 1 repository · arXiv:2411.14611
-
Automated Vulnerability Detection Using Deep Learning Technique 29 Oct 2024 · 0 repositories · arXiv:2410.21968
-
LecPrompt: A Prompt-based Approach for Logical Error Correction with CodeBERT 10 Oct 2024 · 0 repositories · arXiv:2410.08241
-
CLNX: Bridging Code and Natural Language for C/C++ Vulnerability-Contributing Commits Identification 11 Sep 2024 · 0 repositories · arXiv:2409.07407
-
Unlearning Trojans in Large Language Models: A Comparison Between Natural Language and Source Code 22 Aug 2024 · 0 repositories · arXiv:2408.12416
-
CodeMirage: Hallucinations in Code Generated by Large Language Models 14 Aug 2024 · 0 repositories · arXiv:2408.08333
-
Enhancing Code Translation in Language Models with Few-Shot Learning via Retrieval-Augmented Generation 29 Jul 2024 · 0 repositories · arXiv:2407.19619
-
Advanced Detection of Source Code Clones via an Ensemble of Unsupervised Similarity Measures 3 May 2024 · 1 repository · arXiv:2405.02095
-
Software Vulnerability Prediction in Low-Resource Languages: An Empirical Study of CodeBERT and ChatGPT 26 Apr 2024 · 1 repository · arXiv:2404.17110
-
Enhancing Source Code Classification Effectiveness via Prompt Learning Incorporating Knowledge Features 10 Jan 2024 · 1 repository · arXiv:2401.05544
-
Source Code is a Graph, Not a Sequence: A Cross-Lingual Perspective on Code Clone Detection 27 Dec 2023 · 0 repositories · arXiv:2312.16488
-
Naturalness of Attention: Revisiting Attention in Code Language Models 22 Nov 2023 · 0 repositories · arXiv:2311.13508
-
Learning Defect Prediction from Unrealistic Data 2 Nov 2023 · 0 repositories · arXiv:2311.00931
-
Gem5Pred: Predictive Approaches For Gem5 Simulation Time 10 Oct 2023 · 0 repositories · arXiv:2310.06290
-
A Comparative Study of Filters and Deep Learning Models to predict Diabetic Retinopathy 26 Sep 2023 · 0 repositories · arXiv:2309.15216
-
XGV-BERT: Leveraging Contextualized Language Model and Graph Neural Network for Efficient Software Vulnerability Detection 26 Sep 2023 · 0 repositories · arXiv:2309.14677
-
Code quality assessment using transformers 17 Sep 2023 · 0 repositories · arXiv:2309.09264
-
Pop Quiz! Do Pre-trained Code Models Possess Knowledge of Correct API Names? 14 Sep 2023 · 0 repositories · arXiv:2309.07804
-
How Does Naming Affect LLMs on Code Analysis Tasks? 24 Jul 2023 · 0 repositories · arXiv:2307.12488
-
SecureFalcon: Are We There Yet in Automated Software Vulnerability Detection with LLMs? 13 Jul 2023 · 0 repositories · arXiv:2307.06616
-
FlakyFix: Using Large Language Models for Predicting Flaky Test Fix Categories and Test Code Repair 21 Jun 2023 · 0 repositories · arXiv:2307.00012
-
Automatic Code Summarization via ChatGPT: How Far Are We? 22 May 2023 · 0 repositories · arXiv:2305.12865
-
A Black-Box Attack on Code Models via Representation Nearest Neighbor Search 10 May 2023 · 0 repositories · arXiv:2305.05896
-
The EarlyBIRD Catches the Bug: On Exploiting Early Layers of Encoder Models for More Efficient Code Classification 8 May 2023 · 1 repository · arXiv:2305.04940
-
Towards Efficient Fine-tuning of Pre-trained Code Models: An Experimental Study and Beyond 11 Apr 2023 · 1 repository · arXiv:2304.05216
-
Better Language Models of Code through Self-Improvement 2 Apr 2023 · 1 repository · arXiv:2304.01228
-
CodeBERTScore: Evaluating Code Generation with Pretrained Models of Code 10 Feb 2023 · 1 repository · arXiv:2302.05527
-
FLAME: A small language model for spreadsheet formulas 31 Jan 2023 · 0 repositories · arXiv:2301.13779
-
Unveiling Code Pre-Trained Models: Investigating Syntax and Semantics Capacities 20 Dec 2022 · 0 repositories · arXiv:2212.10017
-
MIXCODE: Enhancing Code Classification by Mixup-Based Data Augmentation 6 Oct 2022 · 1 repository · arXiv:2210.03003
-
T5QL: Taming language models for SQL generation 21 Sep 2022 · 0 repositories · arXiv:2209.10254
-
Code Compliance Assessment as a Learning Problem 10 Sep 2022 · 0 repositories · arXiv:2209.04602
-
Generalizability of Code Clone Detection on CodeBERT 26 Aug 2022 · 0 repositories · arXiv:2208.12588
-
No More Fine-Tuning? An Experimental Evaluation of Prompt Tuning in Code Intelligence 24 Jul 2022 · 1 repository · arXiv:2207.11680Syntology official (archive's flag): 7 ran · 7 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 0 violated, 3 with no contract checked; 4 where Syntology's instrument failed) · 7 unverified (of 14 harvested samples) · 2 pointer-only (licence)
-
Probing Semantic Grounding in Language Models of Code with Representational Similarity Analysis 15 Jul 2022 · 0 repositories · arXiv:2207.07706
-
Learning code summarization from a small and local dataset 2 Jun 2022 · 0 repositories · arXiv:2206.00804
-
CodeAttack: Code-Based Adversarial Attacks for Pre-trained Programming Language Models 31 May 2022 · 2 repositories · arXiv:2206.00052
-
Understanding Long Programming Languages with Structure-Aware Sparse Attention 27 May 2022 · 1 repository · arXiv:2205.13730
-
On The Cross-Modal Transfer from Natural Language to Code through Adapter Modules 19 Apr 2022 · 1 repository · arXiv:2204.08653
-
CoCoSoDa: Effective Contrastive Learning for Code Search 7 Apr 2022 · 0 repositories · arXiv:2204.03293
-
An Exploratory Study on Code Attention in BERT 5 Apr 2022 · 0 repositories · arXiv:2204.10200
-
Probing Pretrained Models of Source Code 16 Feb 2022 · 1 repository · arXiv:2202.08975
-
What Do They Capture? -- A Structural Analysis of Pre-Trained Language Models for Source Code 14 Feb 2022 · 1 repository · arXiv:2202.06840
-
Analyzing CodeBERT's Performance on Natural Language Code Search 16 Jan 2022 · 0 repositories
-
Assemble Foundation Models for Automatic Code Summarization 13 Jan 2022 · 1 repository · arXiv:2201.05222
-
Bridging Pre-trained Models and Downstream Tasks for Source Code Understanding 4 Dec 2021 · 1 repository · arXiv:2112.02268Syntology official (archive's flag): 12 ran · 12 ran (of which 0 constructed an object rather than computing a result; 12 with no instrument failure: 0 honoured, 0 violated, 12 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified (of 13 harvested samples)
-
A New Search Paradigm for Natural Language Code Search 16 Nov 2021 · 0 repositories
-
Repo4QA: Answering Complex Coding Questions via Dense Retrieval on GitHub Repositories 16 Nov 2021 · 0 repositories
-
FACOS: Finding API Relevant Contents on Stack Overflow with Semantic and Syntactic Analysis 14 Nov 2021 · 0 repositories · arXiv:2111.07238
-
AugmentedCode: Examining the Effects of Natural Language Resources in Code Retrieval Models 16 Oct 2021 · 1 repository · arXiv:2110.08512
-
ReGVD: Revisiting Graph Neural Networks for Vulnerability Detection 14 Oct 2021 · 1 repository · arXiv:2110.07317
-
GRAPHIX: A Pre-trained Graph Edit Model for Automated Program Repair 29 Sep 2021 · 0 repositories
-
Improving Stack Overflow question title generation with copying enhanced CodeBERT model and bi-modal information 27 Sep 2021 · 1 repository · arXiv:2109.13073
-
What do pre-trained code models know about code? 25 Aug 2021 · 1 repository · arXiv:2108.11308
-
Memorization and Generalization in Neural Code Intelligence Models 16 Jun 2021 · 2 repositories · arXiv:2106.08704
-
CoSQA: 20,000+ Web Queries for Code Search and Question Answering 27 May 2021 · 1 repository · arXiv:2105.13239Syntology official: harvested, nothing ran · 0 ran · 1 unverified (of 1 harvested sample)
-
Literature review on vulnerability detection using NLP technology 23 Apr 2021 · 0 repositories · arXiv:2104.11230
-
CodeBERT: A Pre-Trained Model for Programming and Natural Languages 19 Feb 2020 · 9 repositories · arXiv:2002.08155Syntology official: harvested, nothing ran · 14 ran (of which 0 constructed an object rather than computing a result; 14 with no instrument failure: 0 honoured, 0 violated, 14 with no contract checked; 0 where Syntology's instrument failed) · 6 unverified (of 20 harvested samples) · 4 pointer-only (licence)
Tasks archive 2025-07-28
20 shown of 80 tasks the archive attaches to papers tagged with this method, by distinct papers. A task without a page in the catalog is plain text.
Usage over time archive 2025-07-28
Components: the archive holds no method-to-method composition, so PwC's Components table cannot be rebuilt; the Papers list carries no Results column for the same reason (the archive does not join its leaderboard rows to method tags).
Categories archive 2025-07-28
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections