{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/chexgenbench-a-unified-benchmark-for-fidelity","title":"CheXGenBench: A Unified Benchmark For Fidelity, Privacy and Utility of Synthetic Chest Radiographs","arxiv_id":"2505.10496","date":"2025-05-15","proceeding":null,"authors":["Raman Dutt","Pedro Sanchez","Yongchen Yao","Steven McDonagh","Sotirios A. Tsaftaris","Timothy Hospedales"],"abstract":"We introduce CheXGenBench, a rigorous and multifaceted evaluation framework for synthetic chest radiograph generation that simultaneously assesses fidelity, privacy risks, and clinical utility across state-of-the-art text-to-image generative models. Despite rapid advancements in generative AI for real-world imagery, medical domain evaluations have been hindered by methodological inconsistencies, outdated architectural comparisons, and disconnected assessment criteria that rarely address the practical clinical value of synthetic samples. CheXGenBench overcomes these limitations through standardised data partitioning and a unified evaluation protocol comprising over 20 quantitative metrics that systematically analyse generation quality, potential privacy vulnerabilities, and downstream clinical applicability across 11 leading text-to-image architectures. Our results reveal critical inefficiencies in the existing evaluation protocols, particularly in assessing generative fidelity, leading to inconsistent and uninformative comparisons. Our framework establishes a standardised benchmark for the medical AI community, enabling objective and reproducible comparisons while facilitating seamless integration of both existing and future generative models. Additionally, we release a high-quality, synthetic dataset, SynthCheX-75K, comprising 75K radiographs generated by the top-performing model (Sana 0.6B) in our benchmark to support further research in this critical domain. Through CheXGenBench, we establish a new state-of-the-art and release our framework, models, and SynthCheX-75K dataset at https://raman1121.github.io/CheXGenBench/","url_abs":"https://arxiv.org/abs/2505.10496v1","url_pdf":"https://arxiv.org/pdf/2505.10496v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"chexgenbench-a-unified-benchmark-for-fidelity","repo_url":"https://github.com/Raman1121/CheXGenBench","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"conditional-text-to-image-synthesis","task_name":"Conditional Text-to-Image Synthesis"}],"methods":[],"datasets_introduced":[{"slug":"synthchex-75k","name":"SynthCheX-75K","full_name":""}],"methods_introduced":[],"results":[{"leaderboard":"/sota/conditional-text-to-image-synthesis-on-mimic","task":"Conditional Text-to-Image Synthesis","dataset":"MIMIC-CXR","model":"Sana","rank_in_archive_order":1,"of":13,"metrics":{"FID (RadDino)":"54.22"},"uses_additional_data":false},{"leaderboard":"/sota/conditional-text-to-image-synthesis-on-mimic","task":"Conditional Text-to-Image Synthesis","dataset":"MIMIC-CXR","model":"Pixart Sigma","rank_in_archive_order":2,"of":13,"metrics":{"FID (RadDino)":"60.15"},"uses_additional_data":false},{"leaderboard":"/sota/conditional-text-to-image-synthesis-on-mimic","task":"Conditional Text-to-Image Synthesis","dataset":"MIMIC-CXR","model":"RadEdit","rank_in_archive_order":3,"of":13,"metrics":{"FID (RadDino)":"69.69"},"uses_additional_data":false},{"leaderboard":"/sota/conditional-text-to-image-synthesis-on-mimic","task":"Conditional Text-to-Image Synthesis","dataset":"MIMIC-CXR","model":"LLM-CXR","rank_in_archive_order":4,"of":13,"metrics":{"FID (RadDino)":"71.24"},"uses_additional_data":false},{"leaderboard":"/sota/conditional-text-to-image-synthesis-on-mimic","task":"Conditional Text-to-Image Synthesis","dataset":"MIMIC-CXR","model":"SD V3.5 Medium (LoRA r128)","rank_in_archive_order":5,"of":13,"metrics":{"FID (RadDino)":"74.58"},"uses_additional_data":false},{"leaderboard":"/sota/conditional-text-to-image-synthesis-on-mimic","task":"Conditional Text-to-Image Synthesis","dataset":"MIMIC-CXR","model":"Lumina 2.0 (LoRA r128)","rank_in_archive_order":6,"of":13,"metrics":{"FID (RadDino)":"88.28"},"uses_additional_data":false},{"leaderboard":"/sota/conditional-text-to-image-synthesis-on-mimic","task":"Conditional Text-to-Image Synthesis","dataset":"MIMIC-CXR","model":"SD V3.5 Medium (LoRA r32)","rank_in_archive_order":7,"of":13,"metrics":{"FID (RadDino)":"93.10"},"uses_additional_data":false},{"leaderboard":"/sota/conditional-text-to-image-synthesis-on-mimic","task":"Conditional Text-to-Image Synthesis","dataset":"MIMIC-CXR","model":"Lumina 2.0 (LoRA r32)","rank_in_archive_order":8,"of":13,"metrics":{"FID (RadDino)":"101.19"},"uses_additional_data":false},{"leaderboard":"/sota/conditional-text-to-image-synthesis-on-mimic","task":"Conditional Text-to-Image Synthesis","dataset":"MIMIC-CXR","model":"SD V1-5","rank_in_archive_order":9,"of":13,"metrics":{"FID (RadDino)":"118.93"},"uses_additional_data":false},{"leaderboard":"/sota/conditional-text-to-image-synthesis-on-mimic","task":"Conditional Text-to-Image Synthesis","dataset":"MIMIC-CXR","model":"Flux.1-Dev (LoRA r32)","rank_in_archive_order":10,"of":13,"metrics":{"FID (RadDino)":"122.40"},"uses_additional_data":false},{"leaderboard":"/sota/conditional-text-to-image-synthesis-on-mimic","task":"Conditional Text-to-Image Synthesis","dataset":"MIMIC-CXR","model":"SD V1-4","rank_in_archive_order":11,"of":13,"metrics":{"FID (RadDino)":"125.18"},"uses_additional_data":false},{"leaderboard":"/sota/conditional-text-to-image-synthesis-on-mimic","task":"Conditional Text-to-Image Synthesis","dataset":"MIMIC-CXR","model":"SD V2-1","rank_in_archive_order":12,"of":13,"metrics":{"FID (RadDino)":"186.53"},"uses_additional_data":false},{"leaderboard":"/sota/conditional-text-to-image-synthesis-on-mimic","task":"Conditional Text-to-Image Synthesis","dataset":"MIMIC-CXR","model":"SD V2","rank_in_archive_order":13,"of":13,"metrics":{"FID (RadDino)":"194.72"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2505.10496","atlas_url":"https://app.syntology.ai/?focus=2505.10496","mcp":null,"developers":"https://syntology.ai/developers","held":true,"reason":"arxiv_id_in_atlas","checked_against":{"atlas_ids":549449,"atlas_ids_sha256":"afd72399dafd9906e9062927f70655952385a3d75488558e1f0484e950ab09b7","read_at":"2026-10-05T01:46:21Z"},"agent_calls":[{"tool":"get_citation_path","arguments":{"paper_1":"2505.10496"},"arguments_in":{"paper_2":"another paper's arXiv id or title"},"call":"get_citation_path(paper_1=\"2505.10496\", paper_2=\"…\")"},{"tool":"get_concepts_for_paper","arguments":{"arxiv_id":"2505.10496"},"call":"get_concepts_for_paper(arxiv_id=\"2505.10496\")"}],"by_repo":[{"repo":"Raman1121/CheXGenBench","kind":"official","samples":0,"ran":0,"instrument":0,"unverified":0}],"by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":null,"arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"7d8b524b5f496962a90ff832e51f0b92b0ef415338376725a97ff7a2b4474281","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}