{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/optimizing-large-language-models-for-openapi","title":"Optimizing Large Language Models for OpenAPI Code Completion","arxiv_id":"2405.15729","date":"2024-05-24","proceeding":null,"authors":["Bohdan Petryshyn","Mantas Lukoševičius"],"abstract":"Recent advancements in Large Language Models (LLMs) and their utilization in code generation tasks have significantly reshaped the field of software development. Despite the remarkable efficacy of code completion solutions in mainstream programming languages, their performance lags when applied to less ubiquitous formats such as OpenAPI definitions. This study evaluates the OpenAPI completion performance of GitHub Copilot, a prevalent commercial code completion tool, and proposes a set of task-specific optimizations leveraging Meta's open-source model Code Llama. A semantics-aware OpenAPI completion benchmark proposed in this research is used to perform a series of experiments through which the impact of various prompt-engineering and fine-tuning techniques on the Code Llama model's performance is analyzed. The fine-tuned Code Llama model reaches a peak correctness improvement of 55.2% over GitHub Copilot despite utilizing 25 times fewer parameters than the commercial solution's underlying Codex model. Additionally, this research proposes an enhancement to a widely used code infilling training technique, addressing the issue of underperformance when the model is prompted with context sizes smaller than those used during training. The dataset, the benchmark, and the model fine-tuning code are made publicly available.","url_abs":"https://arxiv.org/abs/2405.15729v2","url_pdf":"https://arxiv.org/pdf/2405.15729v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"optimizing-large-language-models-for-openapi","repo_url":"https://github.com/BohdanPetryshyn/code-llama-fim-fine-tuning","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"optimizing-large-language-models-for-openapi","repo_url":"https://github.com/BohdanPetryshyn/openapi-completion-benchmark","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":null}],"tasks":[{"task_slug":"code-completion","task_name":"Code Completion"},{"task_slug":"code-generation","task_name":"Code Generation"},{"task_slug":"openapi-code-completion","task_name":"OpenAPI code completion"},{"task_slug":"prompt-engineering","task_name":"Prompt Engineering"}],"methods":[{"method_slug":"llama","method_name":"LLaMA"},{"method_slug":"set","method_name":"SET"}],"datasets_introduced":[{"slug":"openapi-code-completion","name":"OpenAPI completion refined","full_name":""}],"methods_introduced":[],"results":[{"leaderboard":"/sota/openapi-code-completion-on-openapi-code","task":"OpenAPI code completion","dataset":"OpenAPI completion refined","model":"Code Llama 7B, fine-tuned with document splitting","rank_in_archive_order":1,"of":4,"metrics":{"Correctness, avg., %":"34","Correctness, max., %":"42","Validness, avg., %":"69.1","Validness, max., %":"76"},"uses_additional_data":false},{"leaderboard":"/sota/openapi-code-completion-on-openapi-code","task":"OpenAPI code completion","dataset":"OpenAPI completion refined","model":"Code Llama 7B, fine-tuned at 4096 tokens","rank_in_archive_order":2,"of":4,"metrics":{"Correctness, avg., %":"32","Correctness, max., %":"45","Validness, avg., %":"63.1","Validness, max., %":"84"},"uses_additional_data":false},{"leaderboard":"/sota/openapi-code-completion-on-openapi-code","task":"OpenAPI code completion","dataset":"OpenAPI completion refined","model":"Code Llama 7B","rank_in_archive_order":3,"of":4,"metrics":{"Correctness, avg., %":"31.1","Correctness, max., %":"36","Validness, avg., %":"60.7","Validness, max., %":"64"},"uses_additional_data":false},{"leaderboard":"/sota/openapi-code-completion-on-openapi-code","task":"OpenAPI code completion","dataset":"OpenAPI completion refined","model":"GitHub Copilot","rank_in_archive_order":4,"of":4,"metrics":{"Correctness, avg., %":"29","Correctness, max., %":"29","Validness, avg., %":"68","Validness, max., %":"68"},"uses_additional_data":false}],"syntology":{"syntology_url":null,"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers","held":null,"reason":"arxiv_id_not_in_atlas","checked_against":{"atlas_ids":549449,"atlas_ids_sha256":"afd72399dafd9906e9062927f70655952385a3d75488558e1f0484e950ab09b7","read_at":"2026-10-05T01:46:21Z"},"by_repo":[{"repo":"BohdanPetryshyn/code-llama-fim-fine-tuning","kind":"official","samples":0,"ran":0,"instrument":0,"unverified":0,"harvested_for_other_papers":1,"harvested_for":["2308.12950"],"harvested_for_calls":[{"arxiv_id":"2308.12950","tool":"get_harvested_code_for_paper","arguments":{"arxiv_id":"2308.12950"},"call":"get_harvested_code_for_paper(arxiv_id=\"2308.12950\")"}]},{"repo":"BohdanPetryshyn/openapi-completion-benchmark","kind":"official","samples":0,"ran":0,"instrument":0,"unverified":0}],"by_repo_harvested_for_is":"harvested_for: the arXiv ids (as Syntology's graph holds them) of every paper whose samples Syntology's graph links to this repository, in arXiv-id order, complete; the page names the first 3 and says how many more. A link only: it does not say the papers are the same work, that one copied the other, or that the code implements this paper's method; harvested_for_calls gives the get_harvested_code_for_paper call for each arXiv-shaped id among them","by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":null,"arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"17ef47c26de253d2a4485dfcf0989ba9a9150d35ea2b1a3e7062a7266f7a53a5","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}