{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/profiling-entity-matching-benchmark-tasks","title":"Profiling Entity Matching Benchmark Tasks","arxiv_id":null,"date":"2020-10-19","proceeding":"International Conference on Information & Knowledge Management 2020 10","authors":["Anna Primpeli","Christian Bizer"],"abstract":"Entity matching is a central task in data integration which has been researched for decades. Over this time, a wide range of benchmark tasks for evaluating entity matching methods has been developed. This resource paper systematically complements, profiles, and compares 21 entity matching benchmark tasks. In order to better understand the specific challenges associated with different tasks, we define a set of profiling dimensions which capture central aspects of the matching tasks. Using these dimensions, we create groups of benchmark tasks having similar characteristics. Afterwards, we assess the difficulty of the tasks in each group by computing baseline evaluation results using standard feature engineering together with two common classification methods. In order to enable the exact reproducibility of evaluation results, matching tasks need to contain exactly defined sets of matching and non-matching record pairs, as well as a fixed development and test split. As this is not the case for some widely-used benchmark tasks, we complement these tasks with fixed sets of non-matching pairs, as well as fixed splits, and provide the resulting development and test sets for public download.\r\nBy profiling and complementing the benchmark tasks, we support researchers to select challenging as well as diverse tasks and to compare matching systems on clearly defined grounds.","url_abs":"https://dl.acm.org/doi/10.1145/3340531.3412781","url_pdf":"https://dl.acm.org/doi/pdf/10.1145/3340531.3412781","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"profiling-entity-matching-benchmark-tasks","repo_url":"https://github.com/wbsg-uni-mannheim/EntityMatchingTaskProfiler","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"none","reach":null}],"tasks":[{"task_slug":"data-integration","task_name":"Data Integration"},{"task_slug":"entity-resolution","task_name":"Entity Resolution"},{"task_slug":"feature-engineering","task_name":"Feature Engineering"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/entity-resolution-on-abt-buy","task":"Entity Resolution","dataset":"Abt-Buy","model":"Random Forest","rank_in_archive_order":10,"of":16,"metrics":{"F1 (%)":"85.00"},"uses_additional_data":false},{"leaderboard":"/sota/entity-resolution-on-amazon-google","task":"Entity Resolution","dataset":"Amazon-Google","model":"Random Forest","rank_in_archive_order":5,"of":17,"metrics":{"F1 (%)":"79.0"},"uses_additional_data":false},{"leaderboard":"/sota/entity-resolution-on-wdc-computers-xlarge","task":"Entity Resolution","dataset":"WDC Computers-xlarge","model":"Random Forest","rank_in_archive_order":6,"of":6,"metrics":{"F1 (%)":"78.0"},"uses_additional_data":false}],"syntology":{"syntology_url":null,"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers","held":null,"reason":"no_arxiv_id","checked_against":null,"by_repo":[{"repo":"wbsg-uni-mannheim/EntityMatchingTaskProfiler","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0}],"by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":null,"arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"2c7512dcd77588323653eafce5e5b52dcde3c893f280f7d66aac03a2c65bf64c","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}