{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/the-fishyscapes-benchmark-measuring-blind","title":"The Fishyscapes Benchmark: Measuring Blind Spots in Semantic Segmentation","arxiv_id":"1904.03215","date":"2019-04-05","proceeding":null,"authors":["Hermann Blum","Paul-Edouard Sarlin","Juan Nieto","Roland Siegwart","Cesar Cadena"],"abstract":"Deep learning has enabled impressive progress in the accuracy of semantic segmentation. Yet, the ability to estimate uncertainty and detect failure is key for safety-critical applications like autonomous driving. Existing uncertainty estimates have mostly been evaluated on simple tasks, and it is unclear whether these methods generalize to more complex scenarios. We present Fishyscapes, the first public benchmark for uncertainty estimation in a real-world task of semantic segmentation for urban driving. It evaluates pixel-wise uncertainty estimates towards the detection of anomalous objects in front of the vehicle. We~adapt state-of-the-art methods to recent semantic segmentation models and compare approaches based on softmax confidence, Bayesian learning, and embedding density. Our results show that anomaly detection is far from solved even for ordinary situations, while our benchmark allows measuring advancements beyond the state-of-the-art.","url_abs":"https://arxiv.org/abs/1904.03215v4","url_pdf":"https://arxiv.org/pdf/1904.03215v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"the-fishyscapes-benchmark-measuring-blind","repo_url":"https://github.com/hermannsblum/fishyscapes","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"tf","reach":{"status":"ok"}}],"tasks":[{"task_slug":"anomaly-detection","task_name":"Anomaly Detection"},{"task_slug":"autonomous-driving","task_name":"Autonomous Driving"},{"task_slug":"segmentation","task_name":"Segmentation"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"}],"methods":[{"method_slug":"softmax","method_name":"Softmax"}],"datasets_introduced":[{"slug":"fishyscapes","name":"Fishyscapes","full_name":""}],"methods_introduced":[],"results":[{"leaderboard":"/sota/anomaly-detection-on-fishyscapes-l-f","task":"Anomaly Detection","dataset":"Fishyscapes L&F","model":"Dirichlet DeepLab","rank_in_archive_order":13,"of":18,"metrics":{"AP":"34.28","FPR95":"47.43"},"uses_additional_data":true},{"leaderboard":"/sota/anomaly-detection-on-fishyscapes-l-f","task":"Anomaly Detection","dataset":"Fishyscapes L&F","model":"Void Classifier","rank_in_archive_order":15,"of":18,"metrics":{"AP":"10.29","FPR95":"22.11"},"uses_additional_data":true},{"leaderboard":"/sota/anomaly-detection-on-fishyscapes-l-f","task":"Anomaly Detection","dataset":"Fishyscapes L&F","model":"Bayesian DeepLab","rank_in_archive_order":16,"of":18,"metrics":{"AP":"9.8","FPR95":"38.5"},"uses_additional_data":false},{"leaderboard":"/sota/anomaly-detection-on-fishyscapes-l-f","task":"Anomaly Detection","dataset":"Fishyscapes L&F","model":"Learned Embedding Density","rank_in_archive_order":17,"of":18,"metrics":{"AP":"4.7","FPR95":"24.4"},"uses_additional_data":true},{"leaderboard":"/sota/anomaly-detection-on-fishyscapes-l-f","task":"Anomaly Detection","dataset":"Fishyscapes L&F","model":"Softmax Entropy","rank_in_archive_order":18,"of":18,"metrics":{"AP":"2.9","FPR95":"44.8"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/1904.03215","atlas_url":"https://app.syntology.ai/?focus=1904.03215","mcp":null,"developers":"https://syntology.ai/developers","held":true,"reason":"arxiv_id_in_atlas","checked_against":{"atlas_ids":549449,"atlas_ids_sha256":"afd72399dafd9906e9062927f70655952385a3d75488558e1f0484e950ab09b7","read_at":"2026-10-05T01:46:21Z"},"agent_calls":[{"tool":"get_citation_path","arguments":{"paper_1":"1904.03215"},"arguments_in":{"paper_2":"another paper's arXiv id or title"},"call":"get_citation_path(paper_1=\"1904.03215\", paper_2=\"…\")"},{"tool":"get_concepts_for_paper","arguments":{"arxiv_id":"1904.03215"},"call":"get_concepts_for_paper(arxiv_id=\"1904.03215\")"}],"by_repo":[{"repo":"hermannsblum/fishyscapes","kind":"official","samples":0,"ran":0,"instrument":0,"unverified":0}],"by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":null,"arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"92c3e4962047d6e28f2f01f3516ee914289acd63e8f17d6a76b3cab5df4917b5","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}