{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/common-limitations-of-image-processing","title":"Common Limitations of Image Processing Metrics: A Picture Story","arxiv_id":"2104.05642","date":"2021-04-12","proceeding":null,"authors":["Annika Reinke","Minu D. Tizabi","Carole H. Sudre","Matthias Eisenmann","Tim Rädsch","Michael Baumgartner","Laura Acion","Michela Antonelli","Tal Arbel","Spyridon Bakas","Peter Bankhead","Arriel Benis","Matthew Blaschko","Florian Buettner","M. Jorge Cardoso","Jianxu Chen","Veronika Cheplygina","Evangelia Christodoulou","Beth Cimini","Gary S. Collins","Sandy Engelhardt","Keyvan Farahani","Luciana Ferrer","Adrian Galdran","Bram van Ginneken","Ben Glocker","Patrick Godau","Robert Haase","Fred Hamprecht","Daniel A. Hashimoto","Doreen Heckmann-Nötzel","Peter Hirsch","Michael M. Hoffman","Merel Huisman","Fabian Isensee","Pierre Jannin","Charles E. Kahn","Dagmar Kainmueller","Bernhard Kainz","Alexandros Karargyris","Alan Karthikesalingam","A. Emre Kavur","Hannes Kenngott","Jens Kleesiek","Andreas Kleppe","Sven Kohler","Florian Kofler","Annette Kopp-Schneider","Thijs Kooi","Michal Kozubek","Anna Kreshuk","Tahsin Kurc","Bennett A. Landman","Geert Litjens","Amin Madani","Klaus Maier-Hein","Anne L. Martel","Peter Mattson","Erik Meijering","Bjoern Menze","David Moher","Karel G. M. Moons","Henning Müller","Brennan Nichyporuk","Felix Nickel","M. Alican Noyan","Jens Petersen","Gorkem Polat","Susanne M. Rafelski","Nasir Rajpoot","Mauricio Reyes","Nicola Rieke","Michael Riegler","Hassan Rivaz","Julio Saez-Rodriguez","Clara I. Sánchez","Julien Schroeter","Anindo Saha","M. Alper Selver","Lalith Sharan","Shravya Shetty","Maarten van Smeden","Bram Stieltjes","Ronald M. Summers","Abdel A. Taha","Aleksei Tiulpin","Sotirios A. Tsaftaris","Ben van Calster","Gaël Varoquaux","Manuel Wiesenfarth","Ziv R. Yaniv","Paul Jäger","Lena Maier-Hein"],"abstract":"While the importance of automatic image analysis is continuously increasing, recent meta-research revealed major flaws with respect to algorithm validation. Performance metrics are particularly key for meaningful, objective, and transparent performance assessment and validation of the used automatic algorithms, but relatively little attention has been given to the practical pitfalls when using specific metrics for a given image analysis task. These are typically related to (1) the disregard of inherent metric properties, such as the behaviour in the presence of class imbalance or small target structures, (2) the disregard of inherent data set properties, such as the non-independence of the test cases, and (3) the disregard of the actual biomedical domain interest that the metrics should reflect. This living dynamically document has the purpose to illustrate important limitations of performance metrics commonly applied in the field of image analysis. In this context, it focuses on biomedical image analysis problems that can be phrased as image-level classification, semantic segmentation, instance segmentation, or object detection task. The current version is based on a Delphi process on metrics conducted by an international consortium of image analysis experts from more than 60 institutions worldwide.","url_abs":"https://arxiv.org/abs/2104.05642v8","url_pdf":"https://arxiv.org/pdf/2104.05642v8.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"common-limitations-of-image-processing","repo_url":"https://github.com/mathilde-b/SFDA","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"instance-segmentation","task_name":"Instance Segmentation"},{"task_slug":"object-detection","task_name":"Object Detection"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"},{"task_slug":"object-detection-1","task_name":"object-detection"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2104.05642","atlas_url":"https://app.syntology.ai/?focus=2104.05642","mcp":null,"developers":"https://syntology.ai/developers","held":true,"reason":"arxiv_id_in_atlas","checked_against":{"atlas_ids":549449,"atlas_ids_sha256":"afd72399dafd9906e9062927f70655952385a3d75488558e1f0484e950ab09b7","read_at":"2026-10-05T01:46:21Z"},"agent_calls":[{"tool":"get_citation_path","arguments":{"paper_1":"2104.05642"},"arguments_in":{"paper_2":"another paper's arXiv id or title"},"call":"get_citation_path(paper_1=\"2104.05642\", paper_2=\"…\")"},{"tool":"get_concepts_for_paper","arguments":{"arxiv_id":"2104.05642"},"call":"get_concepts_for_paper(arxiv_id=\"2104.05642\")"}],"by_repo":[{"repo":"mathilde-b/SFDA","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0}],"by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":null,"arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"f452e7e1611941c9c4bb7446e824e7cc87eb803e24fd1ac1937baa14b342dec1","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}