{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/dataset/cofar","name":"COFAR","full_name":"Commonsense and Factual Reasoning in Image Search","description_markdown":"The COFAR (COmmonsense and FActual Reasoning) dataset is a collection of images and text queries specifically designed to challenge and evaluate image search models that aim to go beyond simple visual matching. It focuses on the ability of these models to perform **commonsense and factual reasoning**, a capability currently lacking in most existing image search technology.\r\n\r\n**Key Features of COFAR:**\r\n\r\n* **Named Visual Entities as a Gateway to Knowledge:** The dataset consists of images containing prominent named visual entities, such as:\r\n    * **Business Brands:** Examples include \"Rolex\", \"Starbucks\", and \"KFC\". \r\n    * **Celebrities:** Examples include \"Lionel Messi\" and \"J.K. Rowling\". \r\n    * **Landmarks:** Examples include \"Eiffel Tower\", \"Taj Mahal\", and \"Parthenon\".\r\n* **Complex Queries Requiring Reasoning:** Instead of simple object-based queries, COFAR uses carefully crafted textual queries that require a deeper understanding of the image content. These queries incorporate two main elements: \r\n    * **Factual Knowledge:** This refers to specific facts about the named entities present in the image, often derived from external knowledge sources like Wikipedia. For instance, a query might state that \"Lionel Messi is the captain of the Argentina national football team\".\r\n    * **Commonsense Reasoning:** Queries also involve commonsense assumptions about the scene, activities, and relationships depicted in the image. An example is \"a queue of customers patiently waiting to buy ice cream\", where the model needs to infer that people standing in line are likely \"customers\" who are \"waiting\".\r\n* **Structured for Evaluation:**  COFAR is organised into distinct sets to facilitate model training and evaluation:\r\n    * **Training Set:** This subset contains images and corresponding queries used to train image search models, allowing them to learn the relationships between visual content, textual descriptions, and external knowledge.\r\n    * **Gallery Sets:** These sets contain images with entities unseen during training. Models are evaluated on their ability to retrieve relevant images from these galleries given a query, testing their generalisation to new, unseen entities. \r\n\r\n**What Makes COFAR Unique?**\r\n\r\n* **Focus on Reasoning over Visual Recognition:** COFAR shifts the emphasis from purely visual aspects of an image to the reasoning required to understand the query. This means that models cannot rely solely on detecting objects or scenes; they need to incorporate external knowledge and make commonsense inferences.\r\n* **Diversity and Real-World Applicability:** The dataset covers a diverse range of named entities and scenarios, making it more representative of real-world image search tasks. The inclusion of business brands, celebrities, and landmarks makes COFAR relevant to various practical applications, like e-commerce, news search, and tourism.\r\n* **Promoting Advanced Image Search Technology:** COFAR aims to stimulate the development of more sophisticated image search engines capable of understanding user intent and context. By providing a challenging benchmark, COFAR encourages researchers to explore new techniques and architectures that go beyond the limitations of existing methods.\r\n\r\n\r\n**Limitations of COFAR:**\r\n\r\nThe sources also acknowledge some limitations of the COFAR dataset:\r\n\r\n* **Single Entity Focus:** Each image in COFAR contains only one prominent named visual entity. This might not accurately reflect real-world scenarios where multiple entities often co-exist within an image.\r\n* **Dataset Size:** While COFAR is larger than some previous datasets in this area, it is still limited in size compared to datasets used for training large vision-language models. This could potentially limit the generalisability of models trained on COFAR.\r\n\r\n**Overall, COFAR is a valuable resource for researchers working on image search that involves complex reasoning. It presents a challenging benchmark and encourages the development of novel techniques that can leverage external knowledge and commonsense understanding to improve image search accuracy and relevance.**","description_withheld":null,"homepage":"https://vl2g.github.io/projects/cofar/","introduced_date":"2022-10-16","introduced_date_note":null,"introduced_by":{"paper":"/paper/cofar-commonsense-and-factual-reasoning-in","title":"COFAR: Commonsense and Factual Reasoning in Image Search","first_author":"Prajwal Gatti","url":null},"license":{"name":"MIT License","url":"https://github.com/vl2g/cofar?tab=MIT-1-ov-file"},"modalities":[{"name":"Images","url":"/datasets/modality/images"},{"name":"Texts","url":"/datasets/modality/texts"}],"tasks":[{"name":"Image Retrieval","url":"/task/image-retrieval","datasets_with_task":"/datasets/task/image-retrieval"},{"name":"Common Sense Reasoning","url":"/task/common-sense-reasoning","datasets_with_task":"/datasets/task/common-sense-reasoning"}],"languages":[{"name":"English","url":"/datasets/language/english"}],"variants":["COFAR"],"data_loaders":[],"num_papers_in_archive":2,"source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28"},"benchmarks":[{"leaderboard":"/sota/image-retrieval-on-cofar","task":"Image Retrieval","dataset_variant":"COFAR","rows":1,"metrics":["Recall@1","Recall@5"],"first_row_in_archive_order":{"model":"KRAMT","paper":"/paper/cofar-commonsense-and-factual-reasoning-in","metrics":{"Recall@1":"31.6","Recall@5":"64.4"},"code_links":[{"title":"vl2g/cofar","url":"https://github.com/vl2g/cofar"}]},"note":"rows are the archive's own order at snapshot; nothing here re-ranks them"}],"papers":{"order":"date (newest first), then slug","population":"every paper with a leaderboard row on this dataset's benchmarks (the benchmark-backed subset): the archive's own papers-using-this-dataset list was never published, so this is not that list; num_papers_in_archive is the archive's own count","page":1,"pages_in_order":1,"rows_per_page":100,"shown":1,"of":1,"counts":{"papers_with_a_benchmark_row":1,"with_a_code_link":1,"where_syntology_ran_a_sample":0,"not_listed_spam_title":0,"listed":1,"listed_where_code_ran":0,"where_syntology_ran_a_sample_split":{"with_a_run_with_no_instrument_failure":0,"every_run_a_failure_of_syntologys_instrument":0,"listed_with_a_run_with_no_instrument_failure":0,"listed_every_run_a_failure_of_syntologys_instrument":0,"filter":{"states":["a run with no instrument failure","any run, instrument failures included"],"default":"a run with no instrument failure","note":"on the 'only where code ran' pages the default hides, in the browser, the rows where every run was a failure of Syntology's instrument; the second state shows them again. Rows are hidden, never re-ordered; these twins list every row"}},"definition":"distinct papers with at least one leaderboard row on this dataset's benchmarks; 'where Syntology ran a sample' counts papers with at least one harvested sample that ran, which is not a correctness claim"},"items":[{"paper":"/paper/cofar-commonsense-and-factual-reasoning-in","slug":"cofar-commonsense-and-factual-reasoning-in","title":"COFAR: Commonsense and Factual Reasoning in Image Search","date":"2022-10-16","arxiv_id":"2210.08554","rows_on_this_dataset":1,"code_links":1,"syntology":null}],"full_list":"papers_with_a_benchmark_row: every listed paper, in this order"},"papers_with_a_benchmark_row":[{"paper":"/paper/cofar-commonsense-and-factual-reasoning-in","slug":"cofar-commonsense-and-factual-reasoning-in","title":"COFAR: Commonsense and Factual Reasoning in Image Search","date":"2022-10-16","arxiv_id":"2210.08554","rows_on_this_dataset":1,"code_links":1,"syntology":null}],"papers_where_code_ran":0,"papers_where_code_ran_first_page":null,"papers_where_code_ran_caption":"We ran code from the paper's repository; we did not run it on this dataset or check it against this dataset's benchmarks.","pages":[],"row_route":"each paper's syntology object carries sample_list (this site's paper page, #ran), syntology_url (syntology.ai's page for the paper, only when that route probed live) and mcp (the agent call that lists the paper's harvested samples; not a verdict on the paper)","developers":"https://syntology.ai/developers","syntology_totals":{"read_at":"2026-10-05T00:36:11+00:00","papers_with_samples":0,"samples_harvested":0,"samples_ran":0,"samples_constructed":0,"samples_ran_checked":0,"samples_ran_instrument_failed":0,"samples_unverified":0,"pointer_only_for_licence":0,"papers_with_no_sample_that_ran":0,"papers_with_a_sample_that_ran":0,"papers_with_a_run_with_no_instrument_failure":0,"papers_where_every_run_was_an_instrument_failure":0,"note":"the per-paper counts of every listed paper (all pages), summed; not a rate"},"key_notes":{"samples_ran_checked":"legacy name, kept unchanged so existing readers do not break: it counts the samples that ran with no instrument failure (honoured, violated, and ran with no contract checked); it does not mean a contract was checked, and the pages print it as 'K with no instrument failure', not 'K checked'","samples_constructed":"a sub-count of the samples that ran, never subtracted from them and never a failure: an executed sample whose run returned an instance of its own class (fixture_out_type equals the entry name): the run built an object and did not compute a result (Syntology's RAN record, counts.constructed)"},"since_archive_judged":{"papers":0,"papers_placed":0,"judged_by":{"model":"a model","models_as_recorded":[],"from":"lb_rows_run_verified.jsonl: each record's `model` field (the extractor's call that wrote decisions[])"},"note":"a model's judgement from the paper's own tables that they report on this leaderboard, not a result: no value from a paper that was not placed is printed or carried here; papers counts each paper once over these leaderboards","backfill":"8,886 of 18,637 newer papers checked so far (733 more had nothing to place; 41 could not be read)","backfill_complete":false,"leaderboards":[]},"papers_note":"The archive never published its papers-using-dataset list; these are papers with a leaderboard row on this dataset's benchmarks.","record_sha256":"184e4d77753ae71550d4b154054bc417f8b98c84a6334f79b8176611e2f30349","record_changed_at":"2026-10-05","record_changed_at_basis":"changed"}