Home › Code › estimate_pass_at_k

estimate_pass_at_k

Syntologyentry name in harvested code

Syntology's record for this page last changed ; when this build read Syntology's graph is in the build record.

estimate_pass_at_k appears in the code Syntology harvested for 28 papers, as 14 distinct code bodies found in 29 places (a place is one code body under one paper). Of the 28 papers, 28 with a sample that ran: 8 with a run with no instrument failure, 20 where every run was a failure of Syntology's instrument. 5 of the code bodies carry a behaviour fingerprint.

What this page is not. Routines are grouped here by the exact string of their function or class name. Nothing asserts that two samples named estimate_pass_at_k do the same thing, share code, or are comparable; the name is a string, not an identity. Behaviour outputs (what a fingerprinted sample returned on the shared battery) are not in this export and are not shown here; get_code(code_sha256="…") returns this sample's fingerprint on the shared battery, with the battery's shuffled null and lift; not shown on this site (how to connect). "Ran" means executed on a synthesized fixture, not that the code is correct or reproduces a paper.

Samples Syntology

The 14 distinct code bodies named estimate_pass_at_k: 14 ran (of which 0 constructed an object rather than computing a result; 6 with no instrument failure: 2 honoured, 0 violated, 4 with no contract checked; 8 where Syntology's instrument failed) · 0 unverified. “Where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. One tile per status, in the site's fixed vocabulary, each code body counted once:

2ran · honoured contract
0ran · violated contract
8ran · our draft was wrong
0ran; the contract check could not run
4ran
0unverified
5fingerprinted

Licence is a property of each copy, so it is counted per place: 15 of the 29 places are pointer only (Syntology does not serve that copy's text); at 9 of those this paper's copy is under a permissive licence, but the stored copy of the same code that Syntology serves is not cleared for its text (hover the licence cell for why). This site shows no code text for any sample; every row below links to the file in its repository where the record names one.

“Ran” means the sample executed on a synthesized input; it does not mean the output is correct. “Honoured” and “violated” refer to a contract Syntology drafted from the code itself; “our draft was wrong” and “the contract check could not run” (RAISES, DEP_MISSING or TIMEOUT; hover a row's label for which) are failures of Syntology's instrument, not of the code, and those samples did run. The ran count above is every status except unverified, the same rule as each paper page; “with no instrument failure” is honoured, violated and ran with no contract checked. A metamorphic tier, where shown, is evidence of the output's shape, not that the sample implements the method.

Papers

28 papers shown of 28, newest first; 29 places in the table. A paper with no recorded date is placed by the month its arXiv id encodes, shown in the Date column as YYYY-MM (from id). One row per place: a paper whose repository defines the name more than once appears more than once, and the same code body held for several papers appears once under each, with the same status. Titles and dates are the archive's archive 2025-07-28 for papers in the archive, and the graph's for 7 papers added by Syntology. Status and fingerprint are Syntology's record of each code body; licence is recorded for each place. The File cell ends with the code body's code_sha256. A sample's code_sha256 is Syntology's identity for that exact code. It is printed in full (16 hexadecimal characters for every sample in this build), and get_code(code_sha256="…") takes it exactly as printed. An agent calls it on Syntology's MCP service (how to connect); click an identity to copy that call.

PaperDateFileStatus SyntologyLicence
Beyond "What to Retrieve": Uncertainty in Retrieval-Augmented Code Generation added by Syntology 2026-07 (from id) Rocky5502/OpenCoder_V1/opencoder/evaluation/metrics.py 42c81d3f43739308 ran · honoured contract fingerprinted MIT (permissive)
From Trajectories to Prefixes: Reusing Teacher Trajectories via Replayed Prefixes and Online Continuation added by Syntology 2026-07 (from id) HappynessI/Prefix_GRPO/legacy_eval/eval/eval_webshop_vllm_server.py bfc71d0afa64f3f5 ran · honoured contract fingerprinted Apache-2.0 (permissive)
TARPO: Token-Wise Latent-Explicit Reasoning via Action-Routing Policy Optimization added by Syntology 2026-06 (from id) NKU-LITI/TARPO-master/humanevaleval.py e080db2eaa81fdb3 ran · our draft was wrong fingerprinted no licence file found · pointer only
REAM: Merging Improves Pruning of Experts in LLMs added by Syntology 2026-04 (from id) zai-org/glm-simple-evals/evals/livecodebench_eval.py 9df10eb02582b09f ran · our draft was wrong no licence file found · pointer only
On Representation Redundancy in Large-Scale Instruction Tuning Data Selection added by Syntology 2026-02 (from id) tdano1/CRDS/main/eval/codex_humaneval/evaluation.py ff29b26b4a0b7f51 ran · our draft was wrong Apache-2.0 (permissive)
PACE: Prefix-Protected and Difficulty-Aware Compression for Efficient Reasoning added by Syntology 2026-02 (from id) LiveCodeBench/LiveCodeBench/lcb_runner/evaluation/pass_k_utils.py 9df10eb02582b09f ran · our draft was wrong MIT (permissive) here; the stored copy is pointer only
CPMöbius: Iterative Coach-Player Reasoning for Data-Free Reinforcement Learning added by Syntology 2026-02 (from id) thunlp/CPMobius/evaluation/utils/evaluation.py e080db2eaa81fdb3 ran · our draft was wrong fingerprinted Apache-2.0 (permissive) here; the stored copy is pointer only
Process Reinforcement through Implicit Rewards 3 Feb 2025 prime-rl/prime/eval/utils/evaluation.py e080db2eaa81fdb3 ran · our draft was wrong fingerprinted Apache-2.0 (permissive) here; the stored copy is pointer only
Inference Scaling fLaws: The Limits of LLM Resampling with Imperfect Verifiers 26 Nov 2024 benediktstroebl/inference-scaling-limits/race/codeeval/evaluation_utils_evalplus.py 5a283b50e4b14e37 ran · our draft was wrong MIT (permissive)
Beyond Correctness: Benchmarking Multi-dimensional Code Generation for Large Language Models 16 Jul 2024 jszheng21/race/race/codeeval/evaluation_utils_evalplus.py 5a283b50e4b14e37 ran · our draft was wrong Apache-2.0 (permissive)
MG-Verilog: Multi-grained Dataset Towards Enhanced LLM-assisted Verilog Generation 2 Jul 2024 luke-avionics/mg-verilog/verilog_eval/verilog_eval/evaluation.py 0fe5b6fed06ae660 ran · our draft was wrong MIT (permissive)
Symbolic Learning Enables Self-Evolving Agents 26 Jun 2024 aiwaves-cn/agents/src/agents/datasets/humaneval.py 9c2911e17309fc8f ran · our draft was wrong Apache-2.0 (permissive)
Benchmarking Educational Program Repair 8 May 2024 koutchemecharles/gaied_nips23/src/utils/evaluation.py df629018634c1265 ran · our draft was wrong no licence file found · pointer only
Advancing LLM Reasoning Generalists with Preference Trees 2 Apr 2024 openbmb/eurus/eval/utils/evaluation.py e080db2eaa81fdb3 ran · our draft was wrong fingerprinted Apache-2.0 (permissive) here; the stored copy is pointer only
HumanEval on Latest GPT Models -- 2024 20 Feb 2024 daniel442li/gpt-human-eval/human_eval/evaluation.py e080db2eaa81fdb3 ran · our draft was wrong fingerprinted MIT (permissive) here; the stored copy is pointer only
Scaling Sparse Fine-Tuning to Large Language Models 29 Jan 2024 ducdauge/sft-llm/eval/codex_humaneval/evaluation.py e080db2eaa81fdb3 ran · our draft was wrong fingerprinted MIT (permissive) here; the stored copy is pointer only
OOP: Object-Oriented Programming Evaluation Benchmark for Large Language Models 12 Jan 2024 alphadl/oop-eval/oop_evaluate/evaluation.py e080db2eaa81fdb3 ran · our draft was wrong fingerprinted no licence file found · pointer only
InstructCoder: Instruction Tuning Large Language Models for Code Editing 31 Oct 2023 qishenghu/CodeInstruct/edit_eval/evaluation.py e080db2eaa81fdb3 ran · our draft was wrong fingerprinted MIT (permissive) here; the stored copy is pointer only
Lemur: Harmonizing Natural Language and Code for Language Agents 10 Oct 2023 OpenLemur/Lemur/xchat/eval/mbpp/evaluation.py af79b5e511f124e0 ran Apache-2.0 (permissive)
BAMBOO: A Comprehensive Benchmark for Evaluating Long Text Modeling Capacities of Large Language Models 23 Sep 2023 rucaibox/bamboo/private_eval/evaluation.py e080db2eaa81fdb3 ran · our draft was wrong fingerprinted no licence file found · pointer only
YaRN: Efficient Context Window Extension of Large Language Models 31 Aug 2023 qwenlm/qwq/eval/eval/livecodebench_v5_utils/pass_k_utils.py 9df10eb02582b09f ran · our draft was wrong Apache-2.0 (permissive) here; the stored copy is pointer only
Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks 5 Jul 2023 zhaofengwu/counterfactual-evaluation/programming/generation/human_eval/evaluation.py e080db2eaa81fdb3 ran · our draft was wrong fingerprinted MIT (permissive) here; the stored copy is pointer only
Large Language Models of Code Fail at Completing Code with Potential Bugs 6 Jun 2023 amazon-science/buggy-code-completion/src/metrics/evaluation.py af8375df02e6b247 ran Apache-2.0 (permissive)
xCodeEval: A Large Scale Multilingual Multitask Benchmark for Code Understanding, Generation, Translation and Retrieval 6 Mar 2023 ntunlp/execeval/eval_scripts/eval_passk.py 1e081510f01d4d42 ran · our draft was wrong fingerprinted MIT (permissive)
xCodeEval: A Large Scale Multilingual Multitask Benchmark for Code Understanding, Generation, Translation and Retrieval 6 Mar 2023 ntunlp/xCodeEval/evaluation/apr/get_result.py 8c30a56df3c52cef ran MIT (permissive)
LEVER: Learning to Verify Language-to-Code Generation with Execution 16 Feb 2023 niansong1996/lever/few_shot/codex.py 3a39b10c8a429dc8 ran fingerprinted MIT (permissive)
Multi-lingual Evaluation of Code Generation Models 26 Oct 2022 amazon-research/mbxp-exec-eval/mxeval/evaluation.py af79b5e511f124e0 ran Apache-2.0 (permissive)
Efficient Training of Language Models to Fill in the Middle 28 Jul 2022 openai/human-eval-infilling/human_eval_infilling/evaluation.py af79b5e511f124e0 ran MIT (permissive)
Evaluating Large Language Models Trained on Code 7 Jul 2021 2796gaurav/human-eval/human_eval/evaluation.py e080db2eaa81fdb3 ran · our draft was wrong fingerprinted MIT recorded; this copy not marked cleared · pointer only

This site shows no code text; each File cell links to the file on GitHub at the repository's current default branch, which may have changed since the harvest. "Pointer only" means Syntology does not serve that copy's text, for one of six reasons: no licence file was found; the licence was not identified; the licence is recorded as permissive but that copy's record is not marked cleared; the licence is outside the permissive list Syntology serves text under (MIT, Apache-2.0, BSD and similar); this paper's copy was not located, so its licence is not recorded; or this paper's copy is under a permissive licence, but the stored copy of the same code that Syntology serves is not cleared for its text. Some licences outside that list permit redistribution, such as WTFPL, and GPL-3.0 under its conditions; they are simply not on the list. Hover a licence cell for the reason. Per-sample records for a paper are on its paper page under "Code Syntology ran".

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections