Datasets › CommonsenseQA

CommonsenseQA (CSQA)

Introduced by Alon Talmor et al. in CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge1 Jan 2019 archive 2025-07-28

The CommonsenseQA is a dataset for commonsense question answering task. The dataset consists of 12,247 questions with 5 choices each. The dataset was generated by Amazon Mechanical Turk workers in the following process (an example is provided in parentheses):

  1. a crowd worker observes a source concept from ConceptNet (“River”) and three target concepts (“Waterfall”, “Bridge”, “Valley”) that are all related by the same ConceptNet relation (“AtLocation”),
  2. the worker authors three questions, one per target concept, such that only that particular target concept is the answer, while the other two distractor concepts are not, (“Where on a river can you hold a cup upright to catch water on a sunny day?”, “Where can I stand on a river to see water falling without getting wet?”, “I’m crossing the river, my feet are wet but my body is dry, where am I?”)
  3. for each question, another worker chooses one additional distractor from Concept Net (“pebble”, “stream”, “bank”), and the author another distractor (“mountain”, “bottom”, “island”) manually.

Source: CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge Image Source: CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge

Benchmarks archive 2025-07-28

All 1 leaderboard whose dataset resolves to this page shown (sort by any header). "First row" is the archive's own first row at snapshot, in the archive's row order; nothing here re-ranks and metric direction is not asserted.

Syntology 16 newer papers were judged to report on this dataset's benchmarks so far (Claude Sonnet 4.5, reading their tables: a model's judgement, not a result; 2 placed): Common Sense Reasoning · CommonsenseQA (16 judged, 2 placed). Drawn from 8,886 of 18,637 newer papers checked so far (733 more had nothing to place; 41 could not be read).

First row (archive order)PaperCode
Common Sense Reasoning CommonsenseQA GPT-4o (HPT) Accuracy 92.54 Hierarchical Prompting Taxonomy: A Universal Evaluation... devichand579/HPT 38 Compare

Papers archive 2025-07-28

papers with a benchmark row: 22 · with a code link: 19 · where Syntology ran a sample: 12 (12 with a run with no instrument failure, 0 where every run was a failure of Syntology's instrument) Syntology

Show: all papers with a benchmark rowonly where code ran (12 of 22 with a benchmark row: 12 with a run with no instrument failure, 0 where every run was a failure of Syntology's instrument)

Page 1 of 1: papers 1 to 22 of 22 with a leaderboard row on this dataset's benchmarks, newest first by the archive's date (ties by slug; undated papers last).

The archive's own "papers using this dataset" list was never published, so this is the benchmark-backed subset, not that list; the archive's count for this dataset is 483. The Syntology column is from Syntology's graph, stated per sample; it is not part of any archive number. A line reads “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified” (C is a part of N, never taken away from it); the figure “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. “Ran” means the sample executed on a synthesized fixture, not that the paper's result was reproduced. When the archive marks a repository official for the paper, the cell starts with that repository's state (the archive's flag, not a verdict on who wrote the code; “community repositories only” when every sample that ran came from a community repository, “official: no sample here; runs from other or unrecorded repositories” when some came from a repository the paper names or has in its text, or from none recorded); hover it for the repositories the samples that ran came from. Syntology's record for this page last changed ; when this build read Syntology's graph is in the build record.

PaperCodeResultsDateSamples run Syntology
Hierarchical Prompting Taxonomy: A Universal Evaluation Framework for Large Language Models Aligned with Human Cognitive Principles 1 1 18 Jun 2024 not harvested
PaLM 2 Technical Report 1 1 17 May 2023 not harvested
BloombergGPT: A Large Language Model for Finance 2 4 30 Mar 2023 not harvested
GrapeQA: GRaph Augmentation and Pruning to Enhance Question-Answering 0 1 22 Mar 2023 not harvested
Deep Bidirectional Language-Knowledge Graph Pretraining 2 1 17 Oct 2022 official (archive's flag): 9 ran · 9 ran (of which 0 constructed an object rather than computing a result; 8 with no instrument failure: 1 honoured, 0 violated, 7 with no contract checked; 1 where Syntology's instrument failed) · 9 unverified (1 pointer-only for licence)
UL2: Unifying Language Learning Paradigms 2 3 10 May 2022 community repositories only · 15 ran (of which 0 constructed an object rather than computing a result; 15 with no instrument failure: 0 honoured, 0 violated, 15 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified
STaR: Bootstrapping Reasoning With Reasoning 1 6 28 Mar 2022 official: harvested, nothing ran · 0 ran · 2 unverified
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models 19 1 28 Jan 2022 5 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 0 honoured, 0 violated, 4 with no contract checked; 1 where Syntology's instrument failed) · 2 unverified (1 pointer-only for licence)
Human Parity on CommonsenseQA: Augmenting Self-Attention with External Attention 2 3 6 Dec 2021 not harvested
QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answering 6 1 13 Apr 2021 official (archive's flag): 8 ran · 8 ran (of which 0 constructed an object rather than computing a result; 8 with no instrument failure: 1 honoured, 0 violated, 7 with no contract checked; 0 where Syntology's instrument failed) · 17 unverified (5 pointer-only for licence)
UNICORN on RAINBOW: A Universal Commonsense Reasoning Model on a New Multitask Benchmark 1 1 24 Mar 2021 official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 0 violated, 3 with no contract checked; 0 where Syntology's instrument failed) · 2 unverified
Muppet: Massive Multi-task Representations with Pre-Finetuning 2 1 26 Jan 2021 not harvested
Fusing Context Into Knowledge Graph for Commonsense Question Answering 2 1 9 Dec 2020 not harvested
UnifiedQA: Crossing Format Boundaries With a Single QA System 2 5 2 May 2020 official (archive's flag): 4 ran · 4 ran (of which 3 constructed an object rather than computing a result; 4 with no instrument failure: 0 honoured, 1 violated, 3 with no contract checked; 0 where Syntology's instrument failed) · 3 unverified (1 pointer-only for licence)
Towards Generalizable Neuro-Symbolic Systems for Commonsense Question Answering 0 1 30 Oct 2019 not harvested
ALBERT: A Lite BERT for Self-supervised Learning of Language Representations 48 1 26 Sep 2019 official (archive's flag): 6 ran · 81 ran (of which 17 constructed an object rather than computing a result; 56 with no instrument failure: 4 honoured, 0 violated, 52 with no contract checked; 25 where Syntology's instrument failed) · 45 unverified (28 pointer-only for licence)
Graph-Based Reasoning over Heterogeneous External Knowledge for Commonsense Question Answering 1 1 9 Sep 2019 official (archive's flag): 12 ran · 12 ran (of which 0 constructed an object rather than computing a result; 7 with no instrument failure: 2 honoured, 0 violated, 5 with no contract checked; 5 where Syntology's instrument failed) · 5 unverified (2 pointer-only for licence)
KagNet: Knowledge-Aware Graph Networks for Commonsense Reasoning 2 1 4 Sep 2019 official (archive's flag): 11 ran · 11 ran (of which 0 constructed an object rather than computing a result; 11 with no instrument failure: 1 honoured, 0 violated, 10 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified
Align, Mask and Select: A Simple Method for Incorporating Commonsense Knowledge into Language Representation Models 0 1 19 Aug 2019 not harvested
RoBERTa: A Robustly Optimized BERT Pretraining Approach 67 1 26 Jul 2019 community repositories only · 37 ran (of which 11 constructed an object rather than computing a result; 36 with no instrument failure: 2 honoured, 0 violated, 34 with no contract checked; 1 where Syntology's instrument failed) · 11 unverified (24 pointer-only for licence)
Explain Yourself! Leveraging Language Models for Commonsense Reasoning 1 1 6 Jun 2019 5 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 1 honoured, 0 violated, 0 with no contract checked; 4 where Syntology's instrument failed) · 0 unverified (1 pointer-only for licence)
CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge 4 1 2 Nov 2018 community repositories only · 3 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 0 violated, 3 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified

Dataset loaders archive 2025-07-28

6 loaders as listed in the archive; links are outbound and not re-checked here.

Tasks archive 2025-07-28

License archive 2025-07-28

Unknown

Modalities archive 2025-07-28

Languages archive 2025-07-28

Variants archive 2025-07-28

  • CommonsenseQA

1 variant name, as the archive lists them.

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections