{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/benchie-open-information-extraction","title":"BenchIE: A Framework for Multi-Faceted Fact-Based Open Information Extraction Evaluation","arxiv_id":"2109.06850","date":"2021-09-14","proceeding":"ACL 2022 5","authors":["Kiril Gashteovski","Mingying Yu","Bhushan Kotnis","Carolin Lawrence","Mathias Niepert","Goran Glavaš"],"abstract":"Intrinsic evaluations of OIE systems are carried out either manually -- with human evaluators judging the correctness of extractions -- or automatically, on standardized benchmarks. The latter, while much more cost-effective, is less reliable, primarily because of the incompleteness of the existing OIE benchmarks: the ground truth extractions do not include all acceptable variants of the same fact, leading to unreliable assessment of the models' performance. Moreover, the existing OIE benchmarks are available for English only. In this work, we introduce BenchIE: a benchmark and evaluation framework for comprehensive evaluation of OIE systems for English, Chinese, and German. In contrast to existing OIE benchmarks, BenchIE is fact-based, i.e., it takes into account informational equivalence of extractions: our gold standard consists of fact synsets, clusters in which we exhaustively list all acceptable surface forms of the same fact. Moreover, having in mind common downstream applications for OIE, we make BenchIE multi-faceted; i.e., we create benchmark variants that focus on different facets of OIE evaluation, e.g., compactness or minimality of extractions. We benchmark several state-of-the-art OIE systems using BenchIE and demonstrate that these systems are significantly less effective than indicated by existing OIE benchmarks. We make BenchIE (data and evaluation code) publicly available on https://github.com/gkiril/benchie.","url_abs":"https://arxiv.org/abs/2109.06850v2","url_pdf":"https://arxiv.org/pdf/2109.06850v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"benchie-open-information-extraction","repo_url":"https://github.com/gkiril/benchie","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"NOASSERTION"}}],"tasks":[{"task_slug":"open-information-extraction","task_name":"Open Information Extraction"}],"methods":[],"datasets_introduced":[{"slug":"benchie","name":"BenchIE","full_name":""}],"methods_introduced":[],"results":[{"leaderboard":"/sota/open-information-extraction-on-benchie","task":"Open Information Extraction","dataset":"BenchIE","model":"ClausIE","rank_in_archive_order":1,"of":11,"metrics":{"F1":"0.34","Precision":"0.50","Recall":"0.26"},"uses_additional_data":false},{"leaderboard":"/sota/open-information-extraction-on-benchie","task":"Open Information Extraction","dataset":"BenchIE","model":"MinIE","rank_in_archive_order":2,"of":11,"metrics":{"Precision":"0.43","Recall":"0.28"},"uses_additional_data":false},{"leaderboard":"/sota/open-information-extraction-on-benchie","task":"Open Information Extraction","dataset":"BenchIE","model":"M2OIE (EN)","rank_in_archive_order":4,"of":11,"metrics":{"F1":"0.23","Precision":"0.39"},"uses_additional_data":false},{"leaderboard":"/sota/open-information-extraction-on-benchie","task":"Open Information Extraction","dataset":"BenchIE","model":"ROIE-T","rank_in_archive_order":5,"of":11,"metrics":{"F1":"0.13","Precision":"0.37","Recall":"0.08"},"uses_additional_data":false},{"leaderboard":"/sota/open-information-extraction-on-benchie","task":"Open Information Extraction","dataset":"BenchIE","model":"OpenIE6","rank_in_archive_order":6,"of":11,"metrics":{"F1":"0.25","Precision":"0.31","Recall":"0.21"},"uses_additional_data":false},{"leaderboard":"/sota/open-information-extraction-on-benchie","task":"Open Information Extraction","dataset":"BenchIE","model":"M2OIE (ZH)","rank_in_archive_order":7,"of":11,"metrics":{"F1":"0.17","Precision":"0.26","Recall":"0.13"},"uses_additional_data":false},{"leaderboard":"/sota/open-information-extraction-on-benchie","task":"Open Information Extraction","dataset":"BenchIE","model":"ROIE-N","rank_in_archive_order":8,"of":11,"metrics":{"F1":"0.13","Precision":"0.20","Recall":"0.09"},"uses_additional_data":false},{"leaderboard":"/sota/open-information-extraction-on-benchie","task":"Open Information Extraction","dataset":"BenchIE","model":"Stanford OIE","rank_in_archive_order":9,"of":11,"metrics":{"F1":"0.13","Precision":"0.11","Recall":"0.16"},"uses_additional_data":false},{"leaderboard":"/sota/open-information-extraction-on-benchie","task":"Open Information Extraction","dataset":"BenchIE","model":"M2OIE (DE)","rank_in_archive_order":10,"of":11,"metrics":{"F1":"0.04","Precision":"0.09","Recall":"0.03"},"uses_additional_data":false},{"leaderboard":"/sota/open-information-extraction-on-benchie","task":"Open Information Extraction","dataset":"BenchIE","model":"Naive OIE","rank_in_archive_order":11,"of":11,"metrics":{"F1":"0.03","Precision":"0.03","Recall":"0.02"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2109.06850","atlas_url":"https://app.syntology.ai/?focus=2109.06850","mcp":null,"developers":"https://syntology.ai/developers","held":true,"reason":"arxiv_id_in_atlas","checked_against":{"atlas_ids":549449,"atlas_ids_sha256":"afd72399dafd9906e9062927f70655952385a3d75488558e1f0484e950ab09b7","read_at":"2026-10-05T01:46:21Z"},"agent_calls":[{"tool":"get_citation_path","arguments":{"paper_1":"2109.06850"},"arguments_in":{"paper_2":"another paper's arXiv id or title"},"call":"get_citation_path(paper_1=\"2109.06850\", paper_2=\"…\")"},{"tool":"get_concepts_for_paper","arguments":{"arxiv_id":"2109.06850"},"call":"get_concepts_for_paper(arxiv_id=\"2109.06850\")"}],"by_repo":[{"repo":"gkiril/benchie","kind":"official","samples":0,"ran":0,"instrument":0,"unverified":0}],"by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":null,"arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"4e5f8509e9fb122ff0076b0e5813474c5d53bc6b0c615a9a1fba74e5f7796f81","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}