{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/a-bayesian-evaluation-framework-for-ground","title":"A Bayesian Evaluation Framework for Subjectively Annotated Visual Recognition Tasks","arxiv_id":"2007.06711","date":"2020-06-20","proceeding":null,"authors":["Derek S. Prijatelj","Mel McCurrie","Walter J. Scheirer"],"abstract":"An interesting development in automatic visual recognition has been the emergence of tasks where it is not possible to assign objective labels to images, yet still feasible to collect annotations that reflect human judgements about them. Machine learning-based predictors for these tasks rely on supervised training that models the behavior of the annotators, i.e., what would the average person's judgement be for an image? A key open question for this type of work, especially for applications where inconsistency with human behavior can lead to ethical lapses, is how to evaluate the epistemic uncertainty of trained predictors, i.e., the uncertainty that comes from the predictor's model. We propose a Bayesian framework for evaluating black box predictors in this regime, agnostic to the predictor's internal structure. The framework specifies how to estimate the epistemic uncertainty that comes from the predictor with respect to human labels by approximating a conditional distribution and producing a credible interval for the predictions and their measures of performance. The framework is successfully applied to four image classification tasks that use subjective human judgements: facial beauty assessment, social attribute assignment, apparent age estimation, and ambiguous scene labeling.","url_abs":"https://arxiv.org/abs/2007.06711v2","url_pdf":"https://arxiv.org/pdf/2007.06711v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"a-bayesian-evaluation-framework-for-ground","repo_url":"https://github.com/prijatelj/bayesian_eval_ground_truth-free","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"tf","reach":null}],"tasks":[{"task_slug":"age-estimation","task_name":"Age Estimation"},{"task_slug":"attribute","task_name":"Attribute"},{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"open-question","task_name":"Open-Ended Question Answering"},{"task_slug":"scene-labeling","task_name":"Scene Labeling"},{"task_slug":"image-classification","task_name":"image-classification"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers","held":null,"reason":"arxiv_id_not_in_atlas","checked_against":{"atlas_ids":549449,"atlas_ids_sha256":"afd72399dafd9906e9062927f70655952385a3d75488558e1f0484e950ab09b7","read_at":"2026-10-05T01:46:21Z"},"by_repo":[{"repo":"prijatelj/bayesian_eval_ground_truth-free","kind":"official","samples":0,"ran":0,"instrument":0,"unverified":0}],"by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":null,"arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"b5a38ecb548cd490fcadd5053f0a734dfa42ce5481a27e6e0da10ad9fc38695a","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}