{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/dataset/coffe","name":"COFFE","full_name":"COFFE: A Code Efficiency Benchmark for Code Generation","description_markdown":"# COFFE\r\n\r\nCOFFE is a Python benchmark for evaluating the time efficiency of LLM-generated code. It is released by the FSE'25 paper \"[COFFE: A Code Efficiency Benchmark for Code Generation](https://arxiv.org/abs/2502.02827)\". You can also refer to the [project webpage](https://www.yunpeng.site/projects/coffe/) for more details.\r\n\r\n## Data\r\n\r\nCOFFE is designed for evaluating both function-level code and file-level code. It contains selected instances from HumanEval, MBPP, APPS and Code Contests. COFFE keeps the original test cases in these benchmarks as *correctness test cases* and adds new test cases designed for time efficiency evaluation as *stressful test cases*.\r\n\r\n**Statistics:**\r\n\r\n|Category|#Instance|#Solution/Instance|#Correctness/Instance | #Stressful/Instance|\r\n|----|----|----|----|----|\r\n|Function-Level|398|1.00|5.72|4.99|\r\n|File-Level|358|66.93|43.68|4.95|\r\n\r\n**Data Files:**\r\n\r\nAll instances in COFFE are in `Coffe/datasets`, where `Coffe/datasets/function` contains all function-level instances and `Coffe/datasets/file` contains all file-level instances. In each repo:\r\n- `best_solutions.json` contains the best ground truth solution COFFE uses to calculate efficient@1 and speedup.\r\n- `stressful_testcases.json` contains all stressful test cases COFFE adds.\r\n- `solutions.json` contains all ground truth solutions in original benchmarks.\r\n- `testcases.json` contains all correctness test cases in original benchmarks.\r\n\r\n## Installation\r\n\r\n**Requirements:**\r\n- Linux Machine\r\n- Docker \r\n- Python>=3.10\r\n\r\nWe suggest you create a virtural environment before installing COFFE.\r\n\r\n1. To use COFFE, please clone this repo in your current workspace `workspace/` and execute:\r\n```bash\r\ncd Coffe && pip install .\r\n```\r\n2. COFFE comes with a docker image, to install it:\r\n```bash\r\ndocker build . -t coffe\r\n```\r\nNote that if your network requires proxy, please modify the Dockerfile in `Coffe/` to indicate it, otherwise the docker image building process could fail.\r\n\r\n3. Go back to your workspace and initialize COFFE:\r\n```bash\r\ncd .. && coffe init\r\n```\r\nIf your installation succeeds, you could see the statistics of COFFE. \r\n\r\n## Usage\r\n\r\n### Pipeline\r\n\r\nWhen you prepare the predictions from LLMs, COFFE provides a pipeline to calculate the efficient@1 and speedup defined in the paper:\r\n```bash\r\ncoffe pipe <benchmark_name> <output_repo> \r\n-p <pred_file_1>,...,<pred_file_n> \r\n-f <efficient_at_1/speedup> \r\n-n <number_of_workers>\r\n```\r\nThis command has four phases:\r\n1. santize the predictions.\r\n2. select the correct predictions based on correctness test cases.\r\n3. evaluate the GPU instruction count based on stressful test cases.\r\n4. calculate the final metrics.\r\n\r\nFor example:\r\n```bash\r\ncoffe pipe function Coffe/examples/function -p Coffe/examples/function/GPT-4o.json -f efficient_at_1 -n 8\r\n```\r\nThis command evaluates the predictions from GPT-4o on the function-level instances of COFFE. If you want to evaluate other LLMs, please prepare a `JSON` file with the same format as `Coffe/examples/function/GPT-4o.json`. \r\n\r\n**Prediction File Format:**\r\n\r\nIn the `JSON` file, the key is the prompt used to query the LLM for the results, you could get the prompts in `datasets/function/prompts.json` and `datasets/file/prompts.json`. The value contains two objects, the first is a list contains the raw outputs from LLMs and the second is an indicator for the whether the raw output is valid.\r\n\r\n**Note:**\r\n\r\nIn default, COFFE will run all predictions in docker. However, if you could not successfully install the docker or want to run the predictions on the host machine, you can add the `-x` option.\r\n\r\n\r\n### Single Evaluation\r\n\r\nThe `pipe` command provides an entire pipeline for calculating the final metrics. This pipeline could also be completed by executing the following four single evaluation commands.\r\n\r\n1. Sanitize the predictions\r\n```bash\r\ncoffe eval <benchmark_name> <output_repo> \r\n-p <pred_file_1>,...,<pred_file_n> \r\n-m compilable_rate\r\n```\r\nThis commands output a file ending with `SOLUTIONS.json` that contains the predictions without syntax errors.\r\n\r\n2. Select correct predictions\r\n```bash\r\ncoffe eval <benchmark_name> <output_repo> \r\n-p <pred_file_1>,...,<pred_file_n> \r\n-m correctness\r\n-n <number_of_workers>\r\n```\r\nThis commands accept prediction files ending with `SOLUTIONS.json` and output a file ending with `PASSED_SOLUTIONS.json` that contains the predictions pass all correctness solutions.\r\n\r\n**Note:**\r\nThis command will combine all correct solutions and ground truth solutions together into files `<dataset_name>_all_indexes.json` (used in step 4) and `<dataset_name>_all_PASSED_SOLUTIONS.json` for the next step.\r\n\r\n3. Evaluate the GPU instruction count\r\n```bash\r\ncoffe eval <benchmark_name> <output_repo> \r\n-p <pred_file>\r\n-m instr_count\r\n-n <number_of_workers>\r\n```\r\nThis command will evaluate the GPU instruction count each prediction consumes and output a file ending with `STRESSFUL_INSTRUCTION.json`. \r\n\r\n**Note:**\r\nThis command could only accept one single prediction file ending with `PASSED_SOLUTIONS.json`.\r\n\r\n4. Calculating the efficient@1/speedup\r\n```bash\r\ncoffe eval <benchmark_name> <output_repo> \r\n-p <index_file>,<pred_file>\r\n-m instr_count\r\n-f <efficient_at_1/speedup>\r\n```\r\nThis command calculate the efficient@1 or speedup.\r\n\r\n**Note:**\r\nThis command requires the index file and the instruction file as COFFE compares the performance of predictions with grouth truth solutions to calculate the metrics.\r\n\r\n## STGen\r\n\r\nFor details about the stressful test case generation approach STGen, please see `stgen/`.\r\n\r\n\r\n## Cite\r\nIf you use COFFE, please cite us:\r\n```\r\n@misc{peng2025coffe,\r\n      title={COFFE: A Code Efficiency Benchmark for Code Generation}, \r\n      author={Yun Peng and Jun Wan and Yichen Li and Xiaoxue Ren},\r\n      year={2025},\r\n      eprint={2502.02827},\r\n      archivePrefix={arXiv},\r\n      primaryClass={cs.SE},\r\n      url={https://arxiv.org/abs/2502.02827}, \r\n}\r\n```","description_withheld":null,"homepage":"https://www.yunpeng.site/projects/coffe","introduced_date":"2025-02-05","introduced_date_note":null,"introduced_by":{"paper":"/paper/coffe-a-code-efficiency-benchmark-for-code","title":"COFFE: A Code Efficiency Benchmark for Code Generation","first_author":null,"url":null},"license":{"name":"Apache License 2.0","url":"https://github.com/JohnnyPeng18/Coffe/blob/main/LICENSE"},"modalities":[{"name":"Texts","url":"/datasets/modality/texts"}],"tasks":[{"name":"Code Generation","url":"/task/code-generation","datasets_with_task":"/datasets/task/code-generation"}],"languages":[{"name":"English","url":"/datasets/language/english"}],"variants":["COFFE"],"data_loaders":[],"num_papers_in_archive":1,"source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28"},"benchmarks":[],"papers":{"order":"date (newest first), then slug","population":"every paper with a leaderboard row on this dataset's benchmarks (the benchmark-backed subset): the archive's own papers-using-this-dataset list was never published, so this is not that list; num_papers_in_archive is the archive's own count","page":1,"pages_in_order":1,"rows_per_page":100,"shown":0,"of":0,"counts":{"papers_with_a_benchmark_row":0,"with_a_code_link":0,"where_syntology_ran_a_sample":0,"not_listed_spam_title":0,"listed":0,"listed_where_code_ran":0,"where_syntology_ran_a_sample_split":{"with_a_run_with_no_instrument_failure":0,"every_run_a_failure_of_syntologys_instrument":0,"listed_with_a_run_with_no_instrument_failure":0,"listed_every_run_a_failure_of_syntologys_instrument":0,"filter":{"states":["a run with no instrument failure","any run, instrument failures included"],"default":"a run with no instrument failure","note":"on the 'only where code ran' pages the default hides, in the browser, the rows where every run was a failure of Syntology's instrument; the second state shows them again. Rows are hidden, never re-ordered; these twins list every row"}},"definition":"distinct papers with at least one leaderboard row on this dataset's benchmarks; 'where Syntology ran a sample' counts papers with at least one harvested sample that ran, which is not a correctness claim"},"items":[],"full_list":"papers_with_a_benchmark_row: every listed paper, in this order"},"papers_with_a_benchmark_row":[],"papers_where_code_ran":0,"papers_where_code_ran_first_page":null,"papers_where_code_ran_caption":"We ran code from the paper's repository; we did not run it on this dataset or check it against this dataset's benchmarks.","pages":[],"row_route":"each paper's syntology object carries sample_list (this site's paper page, #ran), syntology_url (syntology.ai's page for the paper, only when that route probed live) and mcp (the agent call that lists the paper's harvested samples; not a verdict on the paper)","developers":"https://syntology.ai/developers","syntology_totals":{"read_at":"2026-10-05T00:36:11+00:00","papers_with_samples":0,"samples_harvested":0,"samples_ran":0,"samples_constructed":0,"samples_ran_checked":0,"samples_ran_instrument_failed":0,"samples_unverified":0,"pointer_only_for_licence":0,"papers_with_no_sample_that_ran":0,"papers_with_a_sample_that_ran":0,"papers_with_a_run_with_no_instrument_failure":0,"papers_where_every_run_was_an_instrument_failure":0,"note":"the per-paper counts of every listed paper (all pages), summed; not a rate"},"key_notes":{"samples_ran_checked":"legacy name, kept unchanged so existing readers do not break: it counts the samples that ran with no instrument failure (honoured, violated, and ran with no contract checked); it does not mean a contract was checked, and the pages print it as 'K with no instrument failure', not 'K checked'","samples_constructed":"a sub-count of the samples that ran, never subtracted from them and never a failure: an executed sample whose run returned an instance of its own class (fixture_out_type equals the entry name): the run built an object and did not compute a result (Syntology's RAN record, counts.constructed)"},"since_archive_judged":{"papers":0,"papers_placed":0,"judged_by":{"model":"a model","models_as_recorded":[],"from":"lb_rows_run_verified.jsonl: each record's `model` field (the extractor's call that wrote decisions[])"},"note":"a model's judgement from the paper's own tables that they report on this leaderboard, not a result: no value from a paper that was not placed is printed or carried here; papers counts each paper once over these leaderboards","backfill":"8,886 of 18,637 newer papers checked so far (733 more had nothing to place; 41 could not be read)","backfill_complete":false,"leaderboards":[]},"papers_note":"The archive never published its papers-using-dataset list; these are papers with a leaderboard row on this dataset's benchmarks.","record_sha256":"92f70124478d2de581228c2baaab812d9b5bc1f54bcf419b244a8cd609eb59a3","record_changed_at":"2026-10-05","record_changed_at_basis":"changed"}