{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/dataset/ac-bench","name":"AC-Bench","full_name":null,"description_markdown":"# AC-Bench: A Benchmark for Actual Causality Reasoning\r\n\r\n## Dataset Description\r\n\r\nAC-Bench is designed to evaluate the actual causality (AC) reasoning capabilities of large language models (LLMs). It contains a collection of carefully annotated samples, each consisting of a story, a query related to actual causation, detailed reasoning steps, and a binary answer. The dataset aims to provide a comprehensive benchmark for assessing the ability of LLMs to perform formal and interpretable AC reasoning.\r\n\r\n## Dataset Structure\r\n\r\nEach sample in the dataset is formatted as a JSON object with the following structure:\r\n\r\n```json\r\n{\r\n  \"story\": \"A narrative describing a real-world scenario involving causal events.\",\r\n  \"question\": \"A query asking whether a specific event caused an outcome.\",\r\n  \"reasoning\": {\r\n    \"causal_events\": {\r\n      \"Event Description\": {\r\n        \"occur\": true/false,  // Whether the event actually occurs.\r\n        \"order\": integer,     // The temporal order of the event.\r\n        \"focal\": true/false,  // Whether the event is the focal causal event.\r\n        \"sufficient\": true/false,  // Whether the event is a sufficient cause.\r\n        \"necessary\": true/false,   // Whether the event is a necessary cause.\r\n        \"halpern_pearl\": true/false,  // Whether the event is an actual cause.\r\n        \"norm_violated\": true/false,  // Whether the event violates a norm.\r\n        \"behavior_intended\": true/false  // Whether the event is intentional.\r\n      },\r\n      ...\r\n    },\r\n    \"outcome_event\": {\r\n      \"Outcome Description\": {\r\n        \"occur\": true/false,  // Whether the outcome event occurs.\r\n        \"order\": integer      // The temporal order of the outcome event.\r\n      }\r\n    }\r\n  },\r\n  \"answer\": \"Yes/No\"  // The binary answer to the question.\r\n}\r\n```\r\n\r\n## Example Sample\r\n\r\n```json\r\n{\r\n  \"story\": \"Bill's wife, Sue, is out of town for the weekend. She leaves Bill a message that says, 'I just saw this marvelous bookend. It's called a Bartlett bookend. So pretty! I'm going to go back tomorrow and get one. It will be perfect for the left side of our bookshelf'. Bill goes and visits his friend. Bill and his friend talk for a while, and when Bill asks if his friend is willing to sell the bookend, his friend is happy to sell it. Bill makes an offer, but his friend insists on him not paying so much. Finally, Bill buys the right-side Bartlett bookend from his friend and goes home. Then the next day, Sue goes and buys the left-side Bartlett bookend. So, when Sue got home, they had the paired set of bookends.\",\r\n  \"question\": \"Did Bill cause them to possess the paired set of bookends?\",\r\n  \"reasoning\": {\r\n    \"causal_events\": {\r\n      \"Bill buys the right-side Bartlett bookend\": {\r\n        \"occur\": true,\r\n        \"order\": 0,\r\n        \"focal\": true,\r\n        \"sufficient\": false,\r\n        \"necessary\": true,\r\n        \"halpern_pearl\": true,\r\n        \"norm_violated\": false,\r\n        \"behavior_intended\": true\r\n      },\r\n      \"Sue buys the left-side Bartlett bookend\": {\r\n        \"occur\": true,\r\n        \"order\": 1,\r\n        \"focal\": false,\r\n        \"sufficient\": false,\r\n        \"necessary\": true,\r\n        \"halpern_pearl\": true,\r\n        \"norm_violated\": false,\r\n        \"behavior_intended\": true\r\n      }\r\n    },\r\n    \"outcome_event\": {\r\n      \"Bill and Sue possess the paired set of bookends\": {\r\n        \"occur\": true,\r\n        \"order\": 2\r\n      }\r\n    }\r\n  },\r\n  \"answer\": \"Yes\"\r\n}\r\n```\r\n\r\n## Dataset Features\r\n\r\n- **Comprehensive Annotations**: Each sample includes detailed reasoning steps for actual causality reasoning.\r\n- **Focus on Actual Causation**: The dataset is specifically designed to evaluate reasoning about actual causality, distinguishing it from type causality.\r\n- **Challenging and Diverse Samples**: The dataset includes a variety of causal scenarios, and presents greater challenge for LLMs compared with Big-Bench Hard Causal Judgment.\r\n\r\n## Usage\r\n\r\nThis dataset can be used to train, evaluate, and analyze the performance of LLMs on actual causality reasoning tasks. Researchers can leverage this benchmark to develop more interpretable and accurate causal reasoning models.\r\n\r\n## License\r\n\r\nThis dataset is released under the [Creative Commons Attribution 4.0 International License](https://creativecommons.org/licenses/by/4.0/).\r\n\r\n## Acknowledgments\r\n\r\nThis dataset is based on the work presented in the paper \"AC-Reason: Towards Theory-Guided Actual Causality Reasoning with Large Language Models\" by Yanxi Zhang, Xin Cong, Zhong Zhang, Xiao Liu, Dongyan Zhao, and Yesai Wu. We thank the authors for their contributions to the field of causal reasoning.","description_withheld":null,"homepage":"https://huggingface.co/datasets/zhangyx2001/AC-Bench","introduced_date":"2025-05-13","introduced_date_note":null,"introduced_by":{"paper":"/paper/ac-reason-towards-theory-guided-actual","title":"AC-Reason: Towards Theory-Guided Actual Causality Reasoning with Large Language Models","first_author":"Yanxi Zhang","url":null},"license":{"name":"CC-BY-4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"modalities":[{"name":"Texts","url":"/datasets/modality/texts"}],"tasks":[{"name":"Causal Judgment","url":"/task/causal-judgment","datasets_with_task":"/datasets/task/causal-judgment"}],"languages":[{"name":"English","url":"/datasets/language/english"}],"variants":["AC-Bench"],"data_loaders":[],"num_papers_in_archive":1,"source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28"},"benchmarks":[],"papers":{"order":"date (newest first), then slug","population":"every paper with a leaderboard row on this dataset's benchmarks (the benchmark-backed subset): the archive's own papers-using-this-dataset list was never published, so this is not that list; num_papers_in_archive is the archive's own count","page":1,"pages_in_order":1,"rows_per_page":100,"shown":0,"of":0,"counts":{"papers_with_a_benchmark_row":0,"with_a_code_link":0,"where_syntology_ran_a_sample":0,"not_listed_spam_title":0,"listed":0,"listed_where_code_ran":0,"where_syntology_ran_a_sample_split":{"with_a_run_with_no_instrument_failure":0,"every_run_a_failure_of_syntologys_instrument":0,"listed_with_a_run_with_no_instrument_failure":0,"listed_every_run_a_failure_of_syntologys_instrument":0,"filter":{"states":["a run with no instrument failure","any run, instrument failures included"],"default":"a run with no instrument failure","note":"on the 'only where code ran' pages the default hides, in the browser, the rows where every run was a failure of Syntology's instrument; the second state shows them again. Rows are hidden, never re-ordered; these twins list every row"}},"definition":"distinct papers with at least one leaderboard row on this dataset's benchmarks; 'where Syntology ran a sample' counts papers with at least one harvested sample that ran, which is not a correctness claim"},"items":[],"full_list":"papers_with_a_benchmark_row: every listed paper, in this order"},"papers_with_a_benchmark_row":[],"papers_where_code_ran":0,"papers_where_code_ran_first_page":null,"papers_where_code_ran_caption":"We ran code from the paper's repository; we did not run it on this dataset or check it against this dataset's benchmarks.","pages":[],"row_route":"each paper's syntology object carries sample_list (this site's paper page, #ran), syntology_url (syntology.ai's page for the paper, only when that route probed live) and mcp (the agent call that lists the paper's harvested samples; not a verdict on the paper)","developers":"https://syntology.ai/developers","syntology_totals":{"read_at":"2026-10-05T00:36:11+00:00","papers_with_samples":0,"samples_harvested":0,"samples_ran":0,"samples_constructed":0,"samples_ran_checked":0,"samples_ran_instrument_failed":0,"samples_unverified":0,"pointer_only_for_licence":0,"papers_with_no_sample_that_ran":0,"papers_with_a_sample_that_ran":0,"papers_with_a_run_with_no_instrument_failure":0,"papers_where_every_run_was_an_instrument_failure":0,"note":"the per-paper counts of every listed paper (all pages), summed; not a rate"},"key_notes":{"samples_ran_checked":"legacy name, kept unchanged so existing readers do not break: it counts the samples that ran with no instrument failure (honoured, violated, and ran with no contract checked); it does not mean a contract was checked, and the pages print it as 'K with no instrument failure', not 'K checked'","samples_constructed":"a sub-count of the samples that ran, never subtracted from them and never a failure: an executed sample whose run returned an instance of its own class (fixture_out_type equals the entry name): the run built an object and did not compute a result (Syntology's RAN record, counts.constructed)"},"since_archive_judged":{"papers":0,"papers_placed":0,"judged_by":{"model":"a model","models_as_recorded":[],"from":"lb_rows_run_verified.jsonl: each record's `model` field (the extractor's call that wrote decisions[])"},"note":"a model's judgement from the paper's own tables that they report on this leaderboard, not a result: no value from a paper that was not placed is printed or carried here; papers counts each paper once over these leaderboards","backfill":"8,886 of 18,637 newer papers checked so far (733 more had nothing to place; 41 could not be read)","backfill_complete":false,"leaderboards":[]},"papers_note":"The archive never published its papers-using-dataset list; these are papers with a leaderboard row on this dataset's benchmarks.","record_sha256":"fe7860055610be7be863b7d27692e099654b4cb36677cecc42002848f1aefd1e","record_changed_at":"2026-10-05","record_changed_at_basis":"changed"}