{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/dataset/conqa","name":"ConQA","full_name":"Conceptual Query Answering","description_markdown":"ConQA is a dataset created using the intersection between VisualGenome and MS-COCO. The goal of this dataset is to provide a new benchmark for text to image retrieval using short and less descriptive queries than the commonly use captions from MS-COCO or Flicker. ConQA consists of 80 queries divided into 50 conceptual and 30 descriptive queries. A descriptive query mentions some of the objects in the image, for instance, `people chopping vegetables`.  While, a conceptual query does not mention objects or only refers to objects in a general context, e.g., `working class life`.\r\n\r\n# Dataset generation\r\nFor the dataset generation, we followed a 3 step workflow: filtering images, generating queries and seeding relevant, and crowd-sourcing extended annotations.\r\n\r\n## Filtering images\r\nThe first step is focused on filtering images that have meaningful scene graphs and captions. To filter the images, we used the following procedure:\r\n\r\n* The image should have one or more captions. Hence, we discarded the YFCC images with no caption, obtaining \r\n images from the MS-COCO subset of Visual Genome.\r\n* The image should describe a complex scene with multiple objects. We filtered all the scene graphs that did not contain any edges. \r\n images pass this filter.\r\n* The relationships should be verbs and not contain nouns or pronouns. To detect this, we generated sentences for each edge as a concatenation of the words on the labels of the nodes and the relationship and applied Part of Speech tagging. We performed the POS Tagging using the model en_core_web_sm provided by SpaCy. We filter all scene graphs that contain an edge not tagged as a verb or that the tag is not in an ad-hoc list of allowed non-verb keywords. The allowed keywords are top, front, end, side, edge, middle, rear, part, bottom, under, next, left, right, center, background, back, and parallel. We allowed these keywords as they represent positional relationships between objects. After filtering, we obtain \r\n images.\r\n\r\n##Generating Queries\r\nTo generate ConQA, the dataset authors worked in three pairs and acted as annotators to manually design the queries, namely 50 conceptual and 30 descriptive queries. After that, we proceeded to use the model \"ViT-B/32\" from CLIP to find relevant images. For conceptual queries, it was challenging to find relevant images directly, so alternative proxy queries were used to identify an initial set of relevant images. These images are the seed for finding other relevant images that were annotated through Amazon Mechanical Turk.\r\n\r\n## Annotation crowdsourcing\r\nHaving the initial relevant set defined by the dataset authors, we expanded the relevant candidates by looking into the top-100 visually closest images according to a pre-trained ResNet152 model for each query. As a result, we increase the number of potentially relevant images to analyze without adding human bias to the task.\r\n\r\nAfter selecting the images to annotate, we set up a set of Human Intelligence Tasks (HITs) on Amazon Mechanical Turk. Each task consisted of a query and 5 potentially relevant images. Then, the workers were instructed to determine whether each image is relevant for the given query. If they were not sure, they could alternatively mark the image as “Unsure”. To reduce presentation bias, we randomize the order of images and the options. Additionally, we include validation tasks with control images to ensure a minimum quality in the annotation process, so workers failing 70% or more of validation queries were excluded.","description_withheld":null,"homepage":"https://github.com/AU-DIS/ConQA","introduced_date":"2024-03-15","introduced_date_note":null,"introduced_by":{"paper":"/paper/does-the-performance-of-text-to-image","title":"Does the Performance of Text-to-Image Retrieval Models Generalize Beyond Captions-as-a-Query?","first_author":"Juan Manuel Rodriguez","url":null},"license":{"name":"Creative Commons Attribution 4.0 International Public License","url":"https://github.com/AU-DIS/ConQA/tree/main?tab=License-1-ov-file#readme"},"modalities":[{"name":"Images","url":"/datasets/modality/images"},{"name":"Texts","url":"/datasets/modality/texts"}],"tasks":[{"name":"Image Retrieval","url":"/task/image-retrieval","datasets_with_task":"/datasets/task/image-retrieval"}],"languages":[{"name":"English","url":"/datasets/language/english"}],"variants":["ConQA","ConQA Descriptive","ConQA Conceptual"],"data_loaders":[],"num_papers_in_archive":1,"source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28"},"benchmarks":[{"leaderboard":"/sota/image-retrieval-on-conqa-conceptual","task":"Image Retrieval","dataset_variant":"ConQA Conceptual","rows":5,"metrics":["Recall@1","Recall@5","Recall@10","R-precision"],"first_row_in_archive_order":{"model":"CLIP","paper":"/paper/does-the-performance-of-text-to-image","metrics":{"R-precision":"6.8","Recall@1":"12.2","Recall@10":"36.7","Recall@5":"30.6"},"code_links":[{"title":"AU-DIS/ConQA","url":"https://github.com/AU-DIS/ConQA"}]},"note":"rows are the archive's own order at snapshot; nothing here re-ranks them"},{"leaderboard":"/sota/image-retrieval-on-conqa-descriptive","task":"Image Retrieval","dataset_variant":"ConQA Descriptive","rows":5,"metrics":["Recall@1","Recall@5","Recall@10","R-precision"],"first_row_in_archive_order":{"model":"CLIP","paper":"/paper/does-the-performance-of-text-to-image","metrics":{"R-precision":"16.5","Recall@1":"20.7","Recall@10":"65.5","Recall@5":"58.3"},"code_links":[{"title":"AU-DIS/ConQA","url":"https://github.com/AU-DIS/ConQA"}]},"note":"rows are the archive's own order at snapshot; nothing here re-ranks them"}],"papers":{"order":"date (newest first), then slug","population":"every paper with a leaderboard row on this dataset's benchmarks (the benchmark-backed subset): the archive's own papers-using-this-dataset list was never published, so this is not that list; num_papers_in_archive is the archive's own count","page":1,"pages_in_order":1,"rows_per_page":100,"shown":1,"of":1,"counts":{"papers_with_a_benchmark_row":1,"with_a_code_link":1,"where_syntology_ran_a_sample":0,"not_listed_spam_title":0,"listed":1,"listed_where_code_ran":0,"where_syntology_ran_a_sample_split":{"with_a_run_with_no_instrument_failure":0,"every_run_a_failure_of_syntologys_instrument":0,"listed_with_a_run_with_no_instrument_failure":0,"listed_every_run_a_failure_of_syntologys_instrument":0,"filter":{"states":["a run with no instrument failure","any run, instrument failures included"],"default":"a run with no instrument failure","note":"on the 'only where code ran' pages the default hides, in the browser, the rows where every run was a failure of Syntology's instrument; the second state shows them again. Rows are hidden, never re-ordered; these twins list every row"}},"definition":"distinct papers with at least one leaderboard row on this dataset's benchmarks; 'where Syntology ran a sample' counts papers with at least one harvested sample that ran, which is not a correctness claim"},"items":[{"paper":"/paper/does-the-performance-of-text-to-image","slug":"does-the-performance-of-text-to-image","title":"Does the Performance of Text-to-Image Retrieval Models Generalize Beyond Captions-as-a-Query?","date":"2024-03-15","arxiv_id":null,"rows_on_this_dataset":10,"code_links":1,"syntology":null}],"full_list":"papers_with_a_benchmark_row: every listed paper, in this order"},"papers_with_a_benchmark_row":[{"paper":"/paper/does-the-performance-of-text-to-image","slug":"does-the-performance-of-text-to-image","title":"Does the Performance of Text-to-Image Retrieval Models Generalize Beyond Captions-as-a-Query?","date":"2024-03-15","arxiv_id":null,"rows_on_this_dataset":10,"code_links":1,"syntology":null}],"papers_where_code_ran":0,"papers_where_code_ran_first_page":null,"papers_where_code_ran_caption":"We ran code from the paper's repository; we did not run it on this dataset or check it against this dataset's benchmarks.","pages":[],"row_route":"each paper's syntology object carries sample_list (this site's paper page, #ran), syntology_url (syntology.ai's page for the paper, only when that route probed live) and mcp (the agent call that lists the paper's harvested samples; not a verdict on the paper)","developers":"https://syntology.ai/developers","syntology_totals":{"read_at":"2026-10-05T00:36:11+00:00","papers_with_samples":0,"samples_harvested":0,"samples_ran":0,"samples_constructed":0,"samples_ran_checked":0,"samples_ran_instrument_failed":0,"samples_unverified":0,"pointer_only_for_licence":0,"papers_with_no_sample_that_ran":0,"papers_with_a_sample_that_ran":0,"papers_with_a_run_with_no_instrument_failure":0,"papers_where_every_run_was_an_instrument_failure":0,"note":"the per-paper counts of every listed paper (all pages), summed; not a rate"},"key_notes":{"samples_ran_checked":"legacy name, kept unchanged so existing readers do not break: it counts the samples that ran with no instrument failure (honoured, violated, and ran with no contract checked); it does not mean a contract was checked, and the pages print it as 'K with no instrument failure', not 'K checked'","samples_constructed":"a sub-count of the samples that ran, never subtracted from them and never a failure: an executed sample whose run returned an instance of its own class (fixture_out_type equals the entry name): the run built an object and did not compute a result (Syntology's RAN record, counts.constructed)"},"since_archive_judged":{"papers":0,"papers_placed":0,"judged_by":{"model":"a model","models_as_recorded":[],"from":"lb_rows_run_verified.jsonl: each record's `model` field (the extractor's call that wrote decisions[])"},"note":"a model's judgement from the paper's own tables that they report on this leaderboard, not a result: no value from a paper that was not placed is printed or carried here; papers counts each paper once over these leaderboards","backfill":"8,886 of 18,637 newer papers checked so far (733 more had nothing to place; 41 could not be read)","backfill_complete":false,"leaderboards":[]},"papers_note":"The archive never published its papers-using-dataset list; these are papers with a leaderboard row on this dataset's benchmarks.","record_sha256":"f17498e6eb9af15a4386ea88241124221df85d69531606ec46976d61473dc787","record_changed_at":"2026-10-05","record_changed_at_basis":"changed"}