{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/leveraging-pre-trained-language-models-for-3","title":"Leveraging pre-trained language models for code generation","arxiv_id":null,"date":"2024-02-29","proceeding":"Complex & Intelligent Systems 2024 2","authors":["Ahmed Soliman","Samir Shaheen","Mayada Hadhoud"],"abstract":"Code assistance refers to the utilization of various tools, techniques, and models to help developers in the process of software development. As coding tasks become increasingly complex, code assistant plays a pivotal role in enhancing developer productivity, reducing errors, and facilitating a more efficient coding workflow. This assistance can manifest in various forms, including code autocompletion, error detection and correction, code generation, documentation support, and context-aware suggestions. Language models have emerged as integral components of code assistance, offering developers the capability to receive intelligent suggestions, generate code snippets, and enhance overall coding proficiency. In this paper, we propose new hybrid models for code generation by leveraging pre-trained language models BERT, RoBERTa, ELECTRA, and LUKE with the Marian Causal Language Model. Selecting these models based on their strong performance in various natural language processing tasks. We evaluate the performance of these models on two datasets CoNaLa and DJANGO and compare them to existing state-of-the-art models. We aim to investigate the potential of pre-trained transformer language models to revolutionize code generation, offering improved precision and efficiency in navigating complex coding scenarios. Additionally, conducting error analysis and refining the generated code. Our results show that these models, when combined with the Marian Decoder, significantly improve code generation accuracy and efficiency. Notably, the RoBERTaMarian model achieved a maximum BLEU score of 35.74 and an exact match accuracy of 13.8% on CoNaLa, while LUKE-Marian attained a BLEU score of 89.34 and an exact match accuracy of 78.50% on DJANGO. Implementation of this work is available at https://github.com/AhmedSSoliman/Leveraging-Pretrained-Language-Models-for-Code-Generation.","url_abs":"https://doi.org/10.1007/s40747-024-01373-8","url_pdf":"https://link.springer.com/content/pdf/10.1007/s40747-024-01373-8.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"leveraging-pre-trained-language-models-for-3","repo_url":"https://github.com/AhmedSSoliman/Leveraging-Pretrained-Language-Models-for-Code-Generation","is_official":0,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":null}],"tasks":[{"task_slug":"code-generation","task_name":"Code Generation"},{"task_slug":"language-modelling","task_name":"Language Modelling"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bert","method_name":"BERT"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"electra","method_name":"ELECTRA"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-linear-decay","method_name":"Linear Warmup With Linear Decay"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"roberta","method_name":"RoBERTa"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"weight-decay","method_name":"Weight Decay"},{"method_slug":"wordpiece","method_name":"WordPiece"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/code-generation-on-conala","task":"Code Generation","dataset":"CoNaLa","model":"RoBERTaMarian","rank_in_archive_order":2,"of":14,"metrics":{"BLEU":"35.74","Exact Match Accuracy":"13.8"},"uses_additional_data":false},{"leaderboard":"/sota/code-generation-on-conala","task":"Code Generation","dataset":"CoNaLa","model":"BERTMarian","rank_in_archive_order":6,"of":14,"metrics":{"BLEU":"32.46","Exact Match Accuracy":"12.40"},"uses_additional_data":false},{"leaderboard":"/sota/code-generation-on-conala","task":"Code Generation","dataset":"CoNaLa","model":"ELECTRAMarian","rank_in_archive_order":10,"of":14,"metrics":{"BLEU":"30.18","Exact Match Accuracy":"10.0"},"uses_additional_data":false},{"leaderboard":"/sota/code-generation-on-conala","task":"Code Generation","dataset":"CoNaLa","model":"LUKEMarian","rank_in_archive_order":12,"of":14,"metrics":{"BLEU":"29.83","Exact Match Accuracy":"7.6"},"uses_additional_data":false},{"leaderboard":"/sota/code-generation-on-django","task":"Code Generation","dataset":"Django","model":"LUKEMarian","rank_in_archive_order":5,"of":11,"metrics":{"Accuracy":"78.50","BLEU Score":"89.34"},"uses_additional_data":false},{"leaderboard":"/sota/code-generation-on-django","task":"Code Generation","dataset":"Django","model":"RoBERTaMarian","rank_in_archive_order":6,"of":11,"metrics":{"Accuracy":"77.95","BLEU Score":"88.91"},"uses_additional_data":false},{"leaderboard":"/sota/code-generation-on-django","task":"Code Generation","dataset":"Django","model":"BERTMarian","rank_in_archive_order":7,"of":11,"metrics":{"Accuracy":"76.68","BLEU Score":"56.55"},"uses_additional_data":false},{"leaderboard":"/sota/code-generation-on-django","task":"Code Generation","dataset":"Django","model":"ELECTRAMarian","rank_in_archive_order":9,"of":11,"metrics":{"Accuracy":"65.32","BLEU Score":"53.02"},"uses_additional_data":false}],"syntology":{"syntology_url":null,"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers","held":null,"reason":"no_arxiv_id","checked_against":null,"by_repo":[{"repo":"AhmedSSoliman/Leveraging-Pretrained-Language-Models-for-Code-Generation","kind":"named_in_paper","samples":0,"ran":0,"instrument":0,"unverified":0}],"by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":null,"arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"f188a47c622f60bf7b0fdb62cdad4f191ba21ead99da489b89ade3a62e57756d","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}