{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/qlevr-a-diagnostic-dataset-for","title":"QLEVR: A Diagnostic Dataset for Quantificational Language and Elementary Visual Reasoning","arxiv_id":"2205.03075","date":"2022-05-06","proceeding":"Findings (NAACL) 2022 7","authors":["Zechen Li","Anders Søgaard"],"abstract":"Synthetic datasets have successfully been used to probe visual question-answering datasets for their reasoning abilities. CLEVR (johnson2017clevr), for example, tests a range of visual reasoning abilities. The questions in CLEVR focus on comparisons of shapes, colors, and sizes, numerical reasoning, and existence claims. This paper introduces a minimally biased, diagnostic visual question-answering dataset, QLEVR, that goes beyond existential and numerical quantification and focus on more complex quantifiers and their combinations, e.g., asking whether there are more than two red balls that are smaller than at least three blue balls in an image. We describe how the dataset was created and present a first evaluation of state-of-the-art visual question-answering models, showing that QLEVR presents a formidable challenge to our current models. Code and Dataset are available at https://github.com/zechenli03/QLEVR","url_abs":"https://arxiv.org/abs/2205.03075v1","url_pdf":"https://arxiv.org/pdf/2205.03075v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"qlevr-a-diagnostic-dataset-for","repo_url":"https://github.com/zechenli03/qlevr","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}}],"tasks":[{"task_slug":"diagnostic","task_name":"Diagnostic"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"visual-question-answering-1","task_name":"Visual Question Answering"},{"task_slug":"visual-question-answering","task_name":"Visual Question Answering (VQA)"},{"task_slug":"visual-reasoning","task_name":"Visual Reasoning"}],"methods":[],"datasets_introduced":[{"slug":"qlevr","name":"QLEVR","full_name":""}],"methods_introduced":[],"results":[{"leaderboard":"/sota/visual-question-answering-on-qlevr","task":"Visual Question Answering (VQA)","dataset":"QLEVR","model":"MAC","rank_in_archive_order":1,"of":5,"metrics":{"Overall Accuracy":"66.5"},"uses_additional_data":false},{"leaderboard":"/sota/visual-question-answering-on-qlevr","task":"Visual Question Answering (VQA)","dataset":"QLEVR","model":"CNN+LSTM","rank_in_archive_order":2,"of":5,"metrics":{"Overall Accuracy":"65.9"},"uses_additional_data":false},{"leaderboard":"/sota/visual-question-answering-on-qlevr","task":"Visual Question Answering (VQA)","dataset":"QLEVR","model":"BERT","rank_in_archive_order":3,"of":5,"metrics":{"Overall Accuracy":" 65.8"},"uses_additional_data":false},{"leaderboard":"/sota/visual-question-answering-on-qlevr","task":"Visual Question Answering (VQA)","dataset":"QLEVR","model":"LSTM","rank_in_archive_order":4,"of":5,"metrics":{"Overall Accuracy":" 64.6"},"uses_additional_data":false},{"leaderboard":"/sota/visual-question-answering-on-qlevr","task":"Visual Question Answering (VQA)","dataset":"QLEVR","model":"Q-type","rank_in_archive_order":5,"of":5,"metrics":{"Overall Accuracy":"50.0"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2205.03075","atlas_url":"https://app.syntology.ai/?focus=2205.03075","mcp":null,"developers":"https://syntology.ai/developers","held":true,"reason":"arxiv_id_in_atlas","checked_against":{"atlas_ids":549449,"atlas_ids_sha256":"afd72399dafd9906e9062927f70655952385a3d75488558e1f0484e950ab09b7","read_at":"2026-10-05T01:46:21Z"},"agent_calls":[{"tool":"get_citation_path","arguments":{"paper_1":"2205.03075"},"arguments_in":{"paper_2":"another paper's arXiv id or title"},"call":"get_citation_path(paper_1=\"2205.03075\", paper_2=\"…\")"},{"tool":"get_concepts_for_paper","arguments":{"arxiv_id":"2205.03075"},"call":"get_concepts_for_paper(arxiv_id=\"2205.03075\")"}],"by_repo":[{"repo":"zechenli03/qlevr","kind":"official","samples":0,"ran":0,"instrument":0,"unverified":0}],"by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":null,"arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"3f1d4bfad1bb428a2f8219e3b9c4baabd660abb52b125ddfb946c5959a275875","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}