{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/graphcodebert-pre-training-code","title":"GraphCodeBERT: Pre-training Code Representations with Data Flow","arxiv_id":"2009.08366","date":"2020-09-17","proceeding":"ICLR 2021 1","authors":["Daya Guo","Shuo Ren","Shuai Lu","Zhangyin Feng","Duyu Tang","Shujie Liu","Long Zhou","Nan Duan","Alexey Svyatkovskiy","Shengyu Fu","Michele Tufano","Shao Kun Deng","Colin Clement","Dawn Drain","Neel Sundaresan","Jian Yin","Daxin Jiang","Ming Zhou"],"abstract":"Pre-trained models for programming language have achieved dramatic empirical improvements on a variety of code-related tasks such as code search, code completion, code summarization, etc. However, existing pre-trained models regard a code snippet as a sequence of tokens, while ignoring the inherent structure of code, which provides crucial code semantics and would enhance the code understanding process. We present GraphCodeBERT, a pre-trained model for programming language that considers the inherent structure of code. Instead of taking syntactic-level structure of code like abstract syntax tree (AST), we use data flow in the pre-training stage, which is a semantic-level structure of code that encodes the relation of \"where-the-value-comes-from\" between variables. Such a semantic-level structure is neat and does not bring an unnecessarily deep hierarchy of AST, the property of which makes the model more efficient. We develop GraphCodeBERT based on Transformer. In addition to using the task of masked language modeling, we introduce two structure-aware pre-training tasks. One is to predict code structure edges, and the other is to align representations between source code and code structure. We implement the model in an efficient way with a graph-guided masked attention function to incorporate the code structure. We evaluate our model on four tasks, including code search, clone detection, code translation, and code refinement. Results show that code structure and newly introduced pre-training tasks can improve GraphCodeBERT and achieves state-of-the-art performance on the four downstream tasks. We further show that the model prefers structure-level attentions over token-level attentions in the task of code search.","url_abs":"https://arxiv.org/abs/2009.08366v4","url_pdf":"https://arxiv.org/pdf/2009.08366v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"graphcodebert-pre-training-code","repo_url":"https://github.com/microsoft/CodeBERT","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"clone-detection","task_name":"Clone Detection"},{"task_slug":"code-completion","task_name":"Code Completion"},{"task_slug":"code-search","task_name":"Code Search"},{"task_slug":"code-summarization-1","task_name":"Code Summarization"},{"task_slug":"code-translation","task_name":"Code Translation"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"masked-language-modeling","task_name":"Masked Language Modeling"},{"task_slug":"code-summarization","task_name":"Source Code Summarization"},{"task_slug":"type-prediction","task_name":"Type prediction"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/code-search-on-codesearchnet","task":"Code Search","dataset":"CodeSearchNet","model":"GraphCodeBERT","rank_in_archive_order":4,"of":6,"metrics":{"Go":"84.1","JS":"71.1","Java":"75.7","Overall":"77.4","PHP":"72.5 ","Python":"87.9","Ruby":"73.2"},"uses_additional_data":false},{"leaderboard":"/sota/type-prediction-on-manytypes4typescript","task":"Type prediction","dataset":"ManyTypes4TypeScript","model":"GraphCodeBERT","rank_in_archive_order":3,"of":9,"metrics":{"Average Accuracy":"62.51","Average F1":"60.57","Average Precision":"60.06","Average Recall":"61.08"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2009.08366","atlas_url":"https://app.syntology.ai/?focus=2009.08366","mcp":null,"developers":"https://syntology.ai/developers","held":true,"reason":"arxiv_id_in_atlas","checked_against":{"atlas_ids":549449,"atlas_ids_sha256":"afd72399dafd9906e9062927f70655952385a3d75488558e1f0484e950ab09b7","read_at":"2026-10-05T01:46:21Z"},"agent_calls":[{"tool":"get_citation_path","arguments":{"paper_1":"2009.08366"},"arguments_in":{"paper_2":"another paper's arXiv id or title"},"call":"get_citation_path(paper_1=\"2009.08366\", paper_2=\"…\")"},{"tool":"get_concepts_for_paper","arguments":{"arxiv_id":"2009.08366"},"call":"get_concepts_for_paper(arxiv_id=\"2009.08366\")"}],"by_repo":[{"repo":"microsoft/CodeBERT","kind":"official","samples":0,"ran":0,"instrument":0,"unverified":0,"harvested_for_other_papers":4,"harvested_for":["2002.08155","2203.03850","2203.09095","2306.14893"],"harvested_for_calls":[{"arxiv_id":"2002.08155","tool":"get_harvested_code_for_paper","arguments":{"arxiv_id":"2002.08155"},"call":"get_harvested_code_for_paper(arxiv_id=\"2002.08155\")"},{"arxiv_id":"2203.03850","tool":"get_harvested_code_for_paper","arguments":{"arxiv_id":"2203.03850"},"call":"get_harvested_code_for_paper(arxiv_id=\"2203.03850\")"},{"arxiv_id":"2203.09095","tool":"get_harvested_code_for_paper","arguments":{"arxiv_id":"2203.09095"},"call":"get_harvested_code_for_paper(arxiv_id=\"2203.09095\")"},{"arxiv_id":"2306.14893","tool":"get_harvested_code_for_paper","arguments":{"arxiv_id":"2306.14893"},"call":"get_harvested_code_for_paper(arxiv_id=\"2306.14893\")"}]}],"by_repo_harvested_for_is":"harvested_for: the arXiv ids (as Syntology's graph holds them) of every paper whose samples Syntology's graph links to this repository, in arXiv-id order, complete; the page names the first 3 and says how many more. A link only: it does not say the papers are the same work, that one copied the other, or that the code implements this paper's method; harvested_for_calls gives the get_harvested_code_for_paper call for each arXiv-shaped id among them","by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":null,"arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"4d5079fb9cb408c585f4be2440148421a55c778b1624a7df3e22abd3c19eaf4a","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}