{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/dataset/spiqa-a-dataset-for-multimodal-question","name":"SPIQA: A Dataset for Multimodal Question Answering on Scientific Papers","full_name":null,"description_markdown":"# SPIQA Dataset Card\r\n\r\n## Dataset Details\r\n\r\n**Dataset Name**: SPIQA (**S**cientific **P**aper **I**mage **Q**uestion **A**nswering)\r\n\r\n**Paper**: [SPIQA: A Dataset for Multimodal Question Answering on Scientific Papers](https://arxiv.org/abs/2407.09413)\r\n\r\n**Github**: [SPIQA eval and metrics code repo](https://github.com/google/spiqa)\r\n\r\n**Dataset Summary**: SPIQA is a large-scale and challenging QA dataset focused on figures, tables, and text paragraphs from scientific research papers in various computer science domains. The figures cover a wide variety of plots, charts, schematic diagrams, result visualization etc. The dataset is the result of a meticulous curation process, leveraging the breadth of expertise and ability of multimodal large language models (MLLMs) to understand figures. We employ both automatic and manual curation to ensure the highest level of quality and reliability. SPIQA consists of more than 270K questions divided into training, validation, and three different evaluation splits. The purpose of the dataset is to evaluate the ability of Large Multimodal Models to comprehend complex figures and tables with the textual paragraphs of scientific papers.\r\n\r\nThis Data Card describes the structure of the SPIQA dataset, divided into training, validation, and three different evaluation splits. The test-B and test-C splits are filtered from the QASA and QASPER datasets and contain human-written QAs. We collect all scientific papers published at top computer science conferences between 2018 and 2023 from arXiv.\r\n\r\nIf you have any comments or questions, reach out to [Shraman Pramanick](https://shramanpramanick.github.io/) or [Subhashini Venugopalan](https://vsubhashini.github.io/).\r\n\r\n**Supported Tasks**: \r\n- Direct QA with figures and tables\r\n- Direct QA with full paper\r\n- CoT QA (retrieval of helpful figures, tables; then answering)\r\n\r\n**Language**: English\r\n\r\n**Release Date**: SPIQA is released in June 2024.\r\n\r\n## Data Splits\r\n\r\nThe statistics of different splits of SPIQA is shown below. \r\n\r\n| <center>Split</center>  | <center>Papers</center> | <center>Questions</center> | <center>Schematics</center>  | <center>Plots & Charts</center> | <center>Visualizations</center>  | <center>Other figures</center> | <center>Tables</center> |\r\n|--------|----------|---------|--------|----------------|-------|--------|----------|\r\n| <center>Train</center>  | <center>25,459</center>   | <center>262,524</center> | <center>44,008</center> | <center>70,041</center>   | <center>27,297</center>| <center>6,450</center>  | <center>114,728</center>  |\r\n| <center>Val</center>    | <center>200</center>   | <center>2,085</center>   | <center>360</center>     | <center>582</center>   | <center>173</center>    | <center>55</center>  | <center>915</center>  |\r\n|  <center>test-A</center>  |  <center>118</center> |  <center>666</center>  |  <center>154</center>  |  <center>301</center>             |  <center>131</center>    |  <center>95</center>      |  <center>434</center>       |\r\n|  <center>test-B</center>  |  <center>65</center>        |  <center>228</center>      |  <center>147</center>     |  <center>156</center>            |  <center>133</center>    |  <center>17</center>      |  <center>341</center>       |\r\n|  <center>test-C</center>  |  <center>314</center>   |  <center>493</center>      |  <center>415</center>     |  <center>404</center>   |  <center>26</center>     |  <center>66</center>      |  <center>1,332</center>     |\r\n\r\n\r\n## Dataset Structure\r\n\r\nThe contents of this dataset card are structured as follows:\r\n\r\n```bash\r\nSPIQA\r\n    ├── SPIQA_train_val_test-A_extracted_paragraphs.zip\r\n        ├── Extracted textual paragraphs from the papers in SPIQA train, val and test-A splits\r\n    ├── SPIQA_train_val_test-A_raw_tex.zip\r\n        └── The raw tex files from the papers in SPIQA train, val and test-A splits. These files are not required to reproduce our results; we open-source them for future research.\r\n    ├── train_val\r\n        ├── SPIQA_train_val_Images.zip\r\n            └── Full resolution figures and tables from the papers in SPIQA train, val splits\r\n        ├── SPIQA_train.json\r\n            └── SPIQA train metadata\r\n        ├── SPIQA_val.json\r\n            └── SPIQA val metadata\r\n    ├── test-A\r\n        ├── SPIQA_testA_Images.zip\r\n            └── Full resolution figures and tables from the papers in SPIQA test-A split\r\n        ├── SPIQA_testA_Images_224px.zip\r\n            └── 224px figures and tables from the papers in SPIQA test-A split\r\n        ├── SPIQA_testA.json\r\n            └── SPIQA test-A metadata\r\n    ├── test-B\r\n        ├── SPIQA_testB_Images.zip\r\n            └── Full resolution figures and tables from the papers in SPIQA test-B split\r\n        ├── SPIQA_testB_Images_224px.zip\r\n            └── 224px figures and tables from the papers in SPIQA test-B split\r\n        ├── SPIQA_testB.json\r\n            └── SPIQA test-B metadata\r\n    ├── test-C\r\n        ├── SPIQA_testC_Images.zip\r\n            └── Full resolution figures and tables from the papers in SPIQA test-C split\r\n        ├── SPIQA_testC_Images_224px.zip\r\n            └── 224px figures and tables from the papers in SPIQA test-C split\r\n        ├── SPIQA_testC.json\r\n            └── SPIQA test-C metadata\r\n```\r\n\r\nThe `testA_data_viewer.json` file is only for viewing a portion of the data on HuggingFace viewer to get a quick sense of the metadata.\r\n\r\n## Metadata Structure\r\n\r\nThe metadata for every split is provided as dictionary where the keys are arXiv IDs of the papers. The primary contents of each dictionary item are:\r\n\r\n- arXiv ID\r\n- Semantic scholar ID (for test-B)\r\n- Figures and tables\r\n    - Name of the png file\r\n    - Caption\r\n    - Content type (figure or table)\r\n    - Figure type (schematic, plot, photo (visualization), others)\r\n- QAs\r\n    - Question, answer and rationale\r\n    - Reference figures and tables\r\n    - Textual evidence (for test-B and test-C)\r\n- Abstract and full paper text (for test-B and test-C; full paper for other splits are provided as a zip)\r\n\r\n## Dataset Use and Starter Snippets\r\n\r\n#### Downloading the Dataset to Local\r\n\r\nWe recommend the users to download the metadata and images to their local machine. \r\n\r\n- Download the whole dataset (all splits).\r\n```bash\r\nfrom huggingface_hub import snapshot_download\r\nsnapshot_download(repo_id=\"google/spiqa\", repo_type=\"dataset\", local_dir='.') ### Mention the local directory path\r\n```\r\n\r\n- Download specific file.\r\n```bash\r\nfrom huggingface_hub import hf_hub_download\r\nhf_hub_download(repo_id=\"google/spiqa\", filename=\"test-A/SPIQA_testA.json\", repo_type=\"dataset\", local_dir='.') ### Mention the local directory path\r\n```\r\n\r\n#### Questions and Answers from a Specific Paper in test-A\r\n```bash\r\nimport json\r\ntestA_metadata = json.load(open('test-A/SPIQA_testA.json', 'r'))\r\npaper_id = '1702.03584v3'\r\nprint(testA_metadata[paper_id]['qa'])\r\n```\r\n\r\n#### Questions and Answers from a Specific Paper in test-B\r\n```bash\r\nimport json\r\ntestB_metadata = json.load(open('test-B/SPIQA_testB.json', 'r'))\r\npaper_id = '1707.07012'\r\nprint(testB_metadata[paper_id]['question']) ## Questions\r\nprint(testB_metadata[paper_id]['composition']) ## Answers\r\n```\r\n\r\n#### Questions and Answers from a Specific Paper in test-C\r\n```bash\r\nimport json\r\ntestC_metadata = json.load(open('test-C/SPIQA_testC.json', 'r'))\r\npaper_id = '1808.08780'\r\nprint(testC_metadata[paper_id]['question']) ## Questions\r\nprint(testC_metadata[paper_id]['answer']) ## Answers\r\n```\r\n\r\n## Annotation Overview\r\nQuestions and answers for the SPIQA train, validation, and test-A sets were machine-generated. Additionally, the SPIQA test-A set was manually filtered and curated. Questions in the SPIQA test-B set are collected from the QASA dataset, while those in the SPIQA test-C set are from the QASPER dataset. Answering the questions in all splits requires holistic understanding of figures and tables with related text from the scientific papers.\r\n\r\n## Personal and Sensitive Information\r\nWe are not aware of any personal or sensitive information in the dataset.\r\n\r\n## Licensing Information\r\nCC BY 4.0\r\n\r\n## Citation Information\r\n\r\n```bibtex\r\n@article{pramanick2024spiqa,\r\n  title={SPIQA: A Dataset for Multimodal Question Answering on Scientific Papers},\r\n  author={Pramanick, Shraman and Chellappa, Rama and Venugopalan, Subhashini},\r\n  journal={NeurIPS},\r\n  year={2024} \r\n}\r\n```","description_withheld":null,"homepage":"https://huggingface.co/datasets/google/spiqa","introduced_date":"2024-07-12","introduced_date_note":null,"introduced_by":{"paper":"/paper/spiqa-a-dataset-for-multimodal-question","title":"SPIQA: A Dataset for Multimodal Question Answering on Scientific Papers","first_author":"Shraman Pramanick","url":null},"license":{"name":"CC-by-4.0","url":null},"modalities":[{"name":"Images","url":"/datasets/modality/images"},{"name":"Texts","url":"/datasets/modality/texts"}],"tasks":[],"languages":[{"name":"English","url":"/datasets/language/english"}],"variants":["SPIQA: A Dataset for Multimodal Question Answering on Scientific Papers"],"data_loaders":[],"num_papers_in_archive":1,"source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28"},"benchmarks":[],"papers":{"order":"date (newest first), then slug","population":"every paper with a leaderboard row on this dataset's benchmarks (the benchmark-backed subset): the archive's own papers-using-this-dataset list was never published, so this is not that list; num_papers_in_archive is the archive's own count","page":1,"pages_in_order":1,"rows_per_page":100,"shown":0,"of":0,"counts":{"papers_with_a_benchmark_row":0,"with_a_code_link":0,"where_syntology_ran_a_sample":0,"not_listed_spam_title":0,"listed":0,"listed_where_code_ran":0,"where_syntology_ran_a_sample_split":{"with_a_run_with_no_instrument_failure":0,"every_run_a_failure_of_syntologys_instrument":0,"listed_with_a_run_with_no_instrument_failure":0,"listed_every_run_a_failure_of_syntologys_instrument":0,"filter":{"states":["a run with no instrument failure","any run, instrument failures included"],"default":"a run with no instrument failure","note":"on the 'only where code ran' pages the default hides, in the browser, the rows where every run was a failure of Syntology's instrument; the second state shows them again. Rows are hidden, never re-ordered; these twins list every row"}},"definition":"distinct papers with at least one leaderboard row on this dataset's benchmarks; 'where Syntology ran a sample' counts papers with at least one harvested sample that ran, which is not a correctness claim"},"items":[],"full_list":"papers_with_a_benchmark_row: every listed paper, in this order"},"papers_with_a_benchmark_row":[],"papers_where_code_ran":0,"papers_where_code_ran_first_page":null,"papers_where_code_ran_caption":"We ran code from the paper's repository; we did not run it on this dataset or check it against this dataset's benchmarks.","pages":[],"row_route":"each paper's syntology object carries sample_list (this site's paper page, #ran), syntology_url (syntology.ai's page for the paper, only when that route probed live) and mcp (the agent call that lists the paper's harvested samples; not a verdict on the paper)","developers":"https://syntology.ai/developers","syntology_totals":{"read_at":"2026-10-05T00:36:11+00:00","papers_with_samples":0,"samples_harvested":0,"samples_ran":0,"samples_constructed":0,"samples_ran_checked":0,"samples_ran_instrument_failed":0,"samples_unverified":0,"pointer_only_for_licence":0,"papers_with_no_sample_that_ran":0,"papers_with_a_sample_that_ran":0,"papers_with_a_run_with_no_instrument_failure":0,"papers_where_every_run_was_an_instrument_failure":0,"note":"the per-paper counts of every listed paper (all pages), summed; not a rate"},"key_notes":{"samples_ran_checked":"legacy name, kept unchanged so existing readers do not break: it counts the samples that ran with no instrument failure (honoured, violated, and ran with no contract checked); it does not mean a contract was checked, and the pages print it as 'K with no instrument failure', not 'K checked'","samples_constructed":"a sub-count of the samples that ran, never subtracted from them and never a failure: an executed sample whose run returned an instance of its own class (fixture_out_type equals the entry name): the run built an object and did not compute a result (Syntology's RAN record, counts.constructed)"},"since_archive_judged":{"papers":0,"papers_placed":0,"judged_by":{"model":"a model","models_as_recorded":[],"from":"lb_rows_run_verified.jsonl: each record's `model` field (the extractor's call that wrote decisions[])"},"note":"a model's judgement from the paper's own tables that they report on this leaderboard, not a result: no value from a paper that was not placed is printed or carried here; papers counts each paper once over these leaderboards","backfill":"8,886 of 18,637 newer papers checked so far (733 more had nothing to place; 41 could not be read)","backfill_complete":false,"leaderboards":[]},"papers_note":"The archive never published its papers-using-dataset list; these are papers with a leaderboard row on this dataset's benchmarks.","record_sha256":"769c0810d96e8e9e379fea63b0503ba6c3425ae73198c7d7ca41c8e030a8dc9f","record_changed_at":"2026-10-05","record_changed_at_basis":"changed"}