Browse State-of-the-Art › Text-to-Code Generation
Text-to-Code Generation
12 papers with code · 1 benchmark · 10 datasets archive 2025-07-28
Text-to-Code Generation is a task where we can generate code based on the natural language description.
Description from the archive archive 2025-07-28.
Benchmarks archive 2025-07-28
1 leaderboard table shown for this task, 1 with rows (a “benchmark” on this site is a table with at least one row, as on /sota), ordered by row count. “Best model” is the first row in the archive's own order at snapshot; nothing is re-ranked here and metric direction is not recorded in the archive. PwC's Trend sparklines are not in the archive, so that column is omitted.
| Dataset | Best model (first row in archive order) | Paper | Code | Syntology | Compare |
|---|---|---|---|---|---|
| CodeXGLUE - CONCODE (2 rows) | CodeT5 | CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder... | code | Syntology official: harvested, nothing ran · 6 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 0 honoured, 0 violated, 5 with no contract checked; 1 where Syntology's instrument failed) · 5 unverified | Compare |
Syntology column: samples harvested from the paper's repositories and executed on synthesized fixtures; “ran” is not a correctness claim and does not order the table. Each line reads “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the instrument figure counts failures of Syntology's instrument, not of the code. When the archive marks a repository official for the row's paper, the line starts with that repository's state (“official” is the archive's flag, not a verdict on who wrote the code; “community repositories only” when every sample that ran came from a community repository, “official: no sample here; runs from other or unrecorded repositories” when some came from a repository the paper names or has in its text, or from none recorded); hover it for the repositories the samples that ran came from. A dash means no Syntology record for that paper, not a recorded non-run. Each “ran” line links to that paper's sample list. Syntology's record for this page last changed ; when this build read Syntology's graph is in the build record. For agents: get_harvested_code_for_paper(arxiv_id) lists each paper's samples; how to connect.
Libraries
Not in the archive: the export carries no per-task library table, so there is nothing to show at snapshot 2025-07-28.
Datasets archive 2025-07-28
10 datasets whose archive record lists this task, ordered by the archive's paper count.
Subtasks archive 2025-07-28
No subtask under this task in the archive's task tree.
Papers archive 2025-07-28
archive papers tagged: 20 · with a code link: 12 · where Syntology ran a sample: 8 (5 with a run with no instrument failure, 3 where every run was a failure of Syntology's instrument) Syntology
Show: all tagged papersonly where code ran (8 of 20 tagged: 5 with a run with no instrument failure, 3 where every run was a failure of Syntology's instrument)
Page 1 of 1: papers 1 to 20 of 20, in archive order: by repositories listed in the archive (most first), then newest first, not by stars (the archive holds no stars, so PwC's “Social” and “Latest” sorts cannot be reproduced). Papers that list no repository come after every paper that lists one.
Papers without a page here are shown as plain text. A Syntology line reads “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the figure “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. When the archive marks a repository official for the paper, the line starts with that repository's state (the archive's flag, not a verdict on who wrote the code); hover it for the repositories the samples that ran came from.
-
9 Feb 2021 7 repositories listed Syntology official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 1 harvested sample)Benchmark datasets have a significant impact on accelerating research in programming language tasks.
-
2 Sep 2021 5 repositories listed Syntology official: harvested, nothing ran · 6 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 0 honoured, 0 violated, 5 with no contract checked; 1 where Syntology's instrument failed) · 5 unverified (of 11 harvested samples)We present CodeT5, a unified pre-trained encoder-decoder Transformer model that better leverages the code semantics conveyed from the developer-assigned identifiers.
-
4 Dec 2023 3 repositories listed Syntology official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 1 unverified (of 2 harvested samples)Magicoder models are trained on 75K synthetic instruction data using OSS-Instruct, a novel approach to enlightening LLMs with open-source code snippets to generate diverse instruction data for code.
-
11 Jun 2025 1 repository listedThis paper presents a system developed for SemEval 2025 Task 8: Question Answering (QA) over tabular data.
-
21 Nov 2024 1 repository listed Syntology official (archive's flag): 9 ran · 9 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 0 honoured, 0 violated, 4 with no contract checked; 5 where Syntology's instrument failed) · 0 unverified (of 9 harvested samples)LPW also sets new state-of-the-art Pass@1 accuracy, achieving 98.
-
8 Jul 2024 1 repository listed Syntology official (archive's flag): 11 ran · 11 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 9 where Syntology's instrument failed) · 0 unverified (of 11 harvested samples) · 11 pointer-only (licence)Recent advancements in open-source code large language models (LLMs) have been driven by fine-tuning on the data generated from powerful closed-source LLMs, which are expensive to obtain.
-
16 Mar 2024 1 repository listedWe systematically investigate the performance of LLMs in robot routing by constructing a dataset with 80 unique robot routing problems across 8 variants in both single and multi-robot settings.
-
19 Jun 2023 1 repository listed Syntology official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 1 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 2 harvested samples)We construct a repository-level dataset PragmaticCode for method-completion in Java and evaluate MGD on it.
-
8 May 2023 1 repository listedCode execution is a fundamental aspect of programming language semantics that reflects the exact behavior of the code.
-
20 Nov 2022 1 repository listedIn the Copy Phase, a binary classifier is employed to determine and mask the pseudocode tokens that can be directly copied into the code.
-
22 Jul 2022 1 repository listedWe present PanGu-Coder, a pretrained decoder-only language model adopting the PanGu-Alpha architecture for text-to-code generation, i. e. the synthesis of programming language solutions given a natural language problem…
-
10 Jun 2022 1 repository listed Syntology official (archive's flag): 6 ran · 6 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 0 honoured, 0 violated, 5 with no contract checked; 1 where Syntology's instrument failed) · 2 unverified (of 8 harvested samples) · 1 pointer-only (licence)This paper addresses the problem of code generation, where the goal is to generate target code given source code in a different language or a natural language description.
-
Aligning Crowd-sourced Human Feedback for Reinforcement Learning on Code Generation by Large Language Models19 Mar 2025 0 repositories listedThis paper studies how AI-assisted programming and large language models (LLM) improve software developers' ability via AI tools (LLM agents) like Github Copilot and Amazon CodeWhisperer, while integrating human…
-
SparsePO: Controlling Preference Alignment of LLMs via Sparse Token Masks7 Oct 2024 0 repositories listedPreference Optimization (PO) has proven an effective step for aligning language models to human-desired behaviors.
-
11 Sep 2024 0 repositories listed Syntology 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 1 harvested sample) · 1 pointer-only (licence)To ensure large language models (LLMs) are used safely, one must reduce their propensity to hallucinate or to generate unacceptable answers.
-
Can OpenSource beat ChatGPT? -- A Comparative Study of Large Language Models for Text-to-Code Generation6 Sep 2024 0 repositories listedIn recent years, large language models (LLMs) have emerged as powerful tools with potential applications in various fields, including software engineering.
-
Generating Unseen Code Tests In Infinitum29 Jul 2024 0 repositories listedLarge Language Models (LLMs) are used for many tasks, including those related to coding.
-
Text-to-Code Generation with Modality-relative Pre-training8 Feb 2024 0 repositories listedLarge pre-trained language models have recently been expanded and applied to programming language tasks with great success, often through further pre-training of a strictly-natural language model--where training…
-
26 Jun 2023 0 repositories listedWe study the ability of pretrained large language models (LLM) to answer questions from online question answering fora such as Stack Overflow.
-
Compilable Neural Code Generation with Compiler Feedback10 Mar 2022 0 repositories listedAutomatically generating compilable programs with (or without) natural language descriptions has always been a touchstone problem for computational linguistics and automated software engineering.
Syntology lines on 8 of the papers shown; no Syntology record for the others (a paper without an arXiv id cannot be joined to the graph, and absence from the graph layer is not a recorded non-run). “Ran” means the sample executed on a synthesized fixture, not that the paper's result was reproduced; each line links to that paper's sample list. Syntology's record for this page last changed ; when this build read Syntology's graph is in the build record. For agents: get_harvested_code_for_paper(arxiv_id) lists each paper's samples; how to connect.
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections