Datasets › GSM8K

GSM8K

Introduced by Karl Cobbe et al. in Training Verifiers to Solve Math Word Problems27 Oct 2021 archive 2025-07-28

GSM8K is a dataset of 8.5K high quality linguistically diverse grade school math word problems created by human problem writers. The dataset is segmented into 7.5K training problems and 1K test problems. These problems take between 2 and 8 steps to solve, and solutions primarily involve performing a sequence of elementary calculations using basic arithmetic operations (+ − ×÷) to reach the final answer. A bright middle school student should be able to solve every problem. It can be used for multi-step mathematical reasoning.

Image source: https://arxiv.org/pdf/2110.14168v1.pdf

Benchmarks archive 2025-07-28

All 7 leaderboards whose dataset resolves to this page shown (sort by any header). "First row" is the archive's own first row at snapshot, in the archive's row order; nothing here re-ranks and metric direction is not asserted.

Syntology 172 newer papers were judged to report on this dataset's benchmarks so far (Claude Sonnet 4.5, reading their tables: a model's judgement, not a result; 0 placed): Arithmetic Reasoning · GSM8K (168 judged, 0 placed), GSM8K · GSM8K (81 judged, 0 placed). Drawn from 8,886 of 18,637 newer papers checked so far (733 more had nothing to place; 41 could not be read).

First row (archive order)PaperCode
Arithmetic Reasoning GSM8K Claude 3.5 Sonnet (HPT) Accuracy 97.72 Hierarchical Prompting Taxonomy: A Universal Evaluation... devichand579/HPT 164 Compare
GSM8K GSM8K Xolver Accuracy 98.1 Xolver: Multi-Agent Reasoning with Holistic Experience... kagnlp/Xolver 3 Compare
GSM8K gsm8k (5-shots) no rows — — 0 Compare
Mathematical Reasoning GSM8K no rows — — 0 Compare
Text Generation GSM8k (5-shot) no rows — — 0 Compare
Text Generation GSM8k TR no rows — — 0 Compare
Text Generation GSM8k TR v0.2 no rows — — 0 Compare

Papers archive 2025-07-28

papers with a benchmark row: 58 · with a code link: 44 · where Syntology ran a sample: 27 (25 with a run with no instrument failure, 2 where every run was a failure of Syntology's instrument) Syntology

Show: all papers with a benchmark rowonly where code ran (27 of 58 with a benchmark row: 25 with a run with no instrument failure, 2 where every run was a failure of Syntology's instrument)

Page 1 of 1: papers 1 to 58 of 58 with a leaderboard row on this dataset's benchmarks, newest first by the archive's date (ties by slug; undated papers last).

The archive's own "papers using this dataset" list was never published, so this is the benchmark-backed subset, not that list; the archive's count for this dataset is 1,881. The Syntology column is from Syntology's graph, stated per sample; it is not part of any archive number. A line reads “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified” (C is a part of N, never taken away from it); the figure “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. “Ran” means the sample executed on a synthesized fixture, not that the paper's result was reproduced. When the archive marks a repository official for the paper, the cell starts with that repository's state (the archive's flag, not a verdict on who wrote the code; “community repositories only” when every sample that ran came from a community repository, “official: no sample here; runs from other or unrecorded repositories” when some came from a repository the paper names or has in its text, or from none recorded); hover it for the repositories the samples that ran came from. Syntology's record for this page last changed ; when this build read Syntology's graph is in the build record.

PaperCodeResultsDateSamples run Syntology
Xolver: Multi-Agent Reasoning with Holistic Experience Learning Just Like an Olympiad Team 1 1 17 Jun 2025 not harvested
CAPO: Cost-Aware Prompt Optimization 2 3 22 Apr 2025 not harvested
MyGO Multiplex CoT: A Method for Self-Reflection in Large Language Models via Double Chain of Thought Thinking 1 1 20 Jan 2025 not harvested
Teaching-Inspired Integrated Prompting Framework: A Novel Approach for Enhancing Reasoning in Large Language Models 1 1 10 Oct 2024 official (archive's flag): 8 ran · 8 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 1 honoured, 2 violated, 0 with no contract checked; 5 where Syntology's instrument failed) · 0 unverified (8 pointer-only for licence)
OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction Data 1 4 2 Oct 2024 12 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 1 honoured, 0 violated, 4 with no contract checked; 7 where Syntology's instrument failed) · 3 unverified
Qwen2 Technical Report 6 1 15 Jul 2024 not harvested
Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMs 1 1 26 Jun 2024 official (archive's flag): 11 ran · 11 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 1 honoured, 0 violated, 4 with no contract checked; 6 where Syntology's instrument failed) · 1 unverified (12 pointer-only for licence)
Breaking the Ceiling of the LLM Community by Treating Token Generation as a Classification for Ensembling 1 1 18 Jun 2024 official (archive's flag): 5 ran · 5 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 1 honoured, 0 violated, 1 with no contract checked; 3 where Syntology's instrument failed) · 2 unverified (1 pointer-only for licence)
DART-Math: Difficulty-Aware Rejection Tuning for Mathematical Problem-Solving 1 8 18 Jun 2024 official (archive's flag): 10 ran · 10 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 1 honoured, 0 violated, 1 with no contract checked; 8 where Syntology's instrument failed) · 2 unverified
Hierarchical Prompting Taxonomy: A Universal Evaluation Framework for Large Language Models Aligned with Human Cognitive Principles 1 1 18 Jun 2024 not harvested
Achieving >97% on GSM8K: Deeply Understanding the Problems Makes LLMs Better Solvers for Math Word Problems 1 1 23 Apr 2024 not harvested
Toward Self-Improvement of LLMs via Imagination, Searching, and Criticizing 1 2 18 Apr 2024 official (archive's flag): 14 ran · 14 ran (of which 1 constructed an object rather than computing a result; 5 with no instrument failure: 1 honoured, 1 violated, 3 with no contract checked; 9 where Syntology's instrument failed) · 4 unverified (18 pointer-only for licence)
Branch-Train-MiX: Mixing Expert LLMs into a Mixture-of-Experts LLM 1 1 12 Mar 2024 not harvested
The Claude 3 Model Family: Opus, Sonnet, Haiku 0 3 4 Mar 2024 not harvested
An Empirical Study of Data Ability Boundary in LLMs' Math Reasoning 1 4 23 Feb 2024 official (archive's flag): 9 ran · 9 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 0 honoured, 2 violated, 3 with no contract checked; 4 where Syntology's instrument failed) · 2 unverified (11 pointer-only for licence)
Orca-Math: Unlocking the potential of SLMs in Grade School Math 0 1 16 Feb 2024 not harvested
OpenMathInstruct-1: A 1.8 Million Math Instruction Tuning Dataset 1 12 15 Feb 2024 not harvested
The Unreasonable Effectiveness of Eccentric Automatic Prompts 0 3 9 Feb 2024 not harvested
DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models 5 1 5 Feb 2024 official (archive's flag): 8 ran · 14 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 0 violated, 3 with no contract checked; 11 where Syntology's instrument failed) · 10 unverified (3 pointer-only for licence)
Parameter-Efficient Sparsity Crafting from Dense to Mixture-of-Experts for Instruction Tuning on General Tasks 2 2 5 Jan 2024 not harvested
Gemini: A Family of Highly Capable Multimodal Models 1 1 19 Dec 2023 not harvested
Fewer is More: Boosting LLM Reasoning with Reinforced Context Pruning 0 1 14 Dec 2023 not harvested
Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations 3 2 14 Dec 2023 not harvested
TinyGSM: achieving >80% on GSM8k with small language models 0 2 14 Dec 2023 not harvested
Frugal LMs Trained to Invoke Symbolic Solvers Achieve Parameter-Efficient Arithmetic Reasoning 1 1 9 Dec 2023 official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 1 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (2 pointer-only for licence)
Orca 2: Teaching Small Language Models How to Reason 0 2 18 Nov 2023 not harvested
OVM, Outcome-supervised Value Models for Planning in Mathematical Reasoning 1 3 16 Nov 2023 official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified (3 pointer-only for licence)
The ART of LLM Refinement: Ask, Refine, and Trust 0 1 14 Nov 2023 not harvested
Llemma: An Open Language Model For Mathematics 4 2 16 Oct 2023 official (archive's flag): 6 ran · 7 ran (of which 0 constructed an object rather than computing a result; 7 with no instrument failure: 0 honoured, 0 violated, 7 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified
KwaiYiiMath: Technical Report 0 1 11 Oct 2023 not harvested
Mistral 7B 6 1 10 Oct 2023 official (archive's flag): 9 ran · 10 ran (of which 0 constructed an object rather than computing a result; 10 with no instrument failure: 0 honoured, 0 violated, 10 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified (2 pointer-only for licence)
MuggleMath: Assessing the Impact of Query and Response Augmentation on Math Reasoning 1 3 9 Oct 2023 not harvested
MathCoder: Seamless Code Integration in LLMs for Enhanced Mathematical Reasoning 1 6 5 Oct 2023 official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified
ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solving 1 6 29 Sep 2023 official (archive's flag): 11 ran · 11 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 0 honoured, 2 violated, 3 with no contract checked; 6 where Syntology's instrument failed) · 1 unverified (9 pointer-only for licence)
MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models 1 4 21 Sep 2023 official (archive's flag): 15 ran · 15 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 1 violated, 0 with no contract checked; 14 where Syntology's instrument failed) · 7 unverified
OpenChat: Advancing Open-source Language Models with Mixed-Quality Data 1 1 20 Sep 2023 official: harvested, nothing ran · 0 ran · 1 unverified
WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct 1 4 18 Aug 2023 10 ran (of which 0 constructed an object rather than computing a result; 7 with no instrument failure: 0 honoured, 0 violated, 7 with no contract checked; 3 where Syntology's instrument failed) · 6 unverified (16 pointer-only for licence)
Scaling Relationship on Learning Mathematical Reasoning with Large Language Models 1 3 3 Aug 2023 official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 1 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified (3 pointer-only for licence)
Llama 2: Open Foundation and Fine-Tuned Chat Models 19 1 18 Jul 2023 community repositories only · 33 ran (of which 7 constructed an object rather than computing a result; 20 with no instrument failure: 1 honoured, 1 violated, 18 with no contract checked; 13 where Syntology's instrument failed) · 19 unverified (20 pointer-only for licence)
Model Card and Evaluations for Claude Models 0 3 11 Jul 2023 not harvested
PaLM 2 Technical Report 1 2 17 May 2023 not harvested
CodeT5+: Open Code Large Language Models for Code Understanding and Generation 2 1 13 May 2023 official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 1 where Syntology's instrument failed) · 1 unverified
Sparks of Artificial General Intelligence: Early experiments with GPT-4 3 1 22 Mar 2023 9 ran (of which 0 constructed an object rather than computing a result; 9 with no instrument failure: 0 honoured, 0 violated, 9 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified
GPT-4 Technical Report 11 1 15 Mar 2023 community repositories only · 5 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 0 honoured, 2 violated, 3 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified (1 pointer-only for licence)
LLaMA: Open and Efficient Foundation Language Models 57 8 27 Feb 2023 official: no sample here; runs from other or unrecorded repositories · 37 ran (of which 9 constructed an object rather than computing a result; 25 with no instrument failure: 3 honoured, 0 violated, 22 with no contract checked; 12 where Syntology's instrument failed) · 21 unverified (4 pointer-only for licence)
LEVER: Learning to Verify Language-to-Code Generation with Execution 1 1 16 Feb 2023 official (archive's flag): 18 ran · 18 ran (of which 0 constructed an object rather than computing a result; 14 with no instrument failure: 0 honoured, 2 violated, 12 with no contract checked; 4 where Syntology's instrument failed) · 4 unverified (1 pointer-only for licence)
Solving math word problems with process- and outcome-based feedback 0 3 25 Nov 2022 not harvested
Solving Math Word Problems via Cooperative Reasoning induced Language Models 1 1 28 Oct 2022 not harvested
Composing Ensembles of Pre-trained Models via Iterative Consensus 0 4 20 Oct 2022 not harvested
Large Language Models Can Self-Improve 0 6 20 Oct 2022 not harvested
Transcending Scaling Laws with 0.1% Extra Compute 0 1 20 Oct 2022 not harvested
Solving Quantitative Reasoning Problems with Language Models 1 10 29 Jun 2022 not harvested
Making Large Language Models Better Reasoners with Step-Aware Verifier 0 1 6 Jun 2022 not harvested
Learning Math Reasoning from Self-Sampled Correct and Partially-Correct Solutions 1 2 28 May 2022 official (archive's flag): 7 ran · 7 ran (of which 0 constructed an object rather than computing a result; 7 with no instrument failure: 0 honoured, 0 violated, 7 with no contract checked; 0 where Syntology's instrument failed) · 4 unverified
Large Language Models are Zero-Shot Reasoners 4 7 24 May 2022 official: harvested, nothing ran · 3 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 0 violated, 3 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified (1 pointer-only for licence)
Least-to-Most Prompting Enables Complex Reasoning in Large Language Models 1 1 21 May 2022 not harvested
UL2: Unifying Language Learning Paradigms 2 2 10 May 2022 community repositories only · 15 ran (of which 0 constructed an object rather than computing a result; 15 with no instrument failure: 0 honoured, 0 violated, 15 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified
Self-Consistency Improves Chain of Thought Reasoning in Language Models 3 1 21 Mar 2022 1 ran (of which 1 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified; the one sample that ran constructed an object rather than computing a result

Dataset loaders archive 2025-07-28

22 loaders as listed in the archive; links are outbound and not re-checked here.

Tasks archive 2025-07-28

License archive 2025-07-28

Unknown

Modalities archive 2025-07-28

Languages archive 2025-07-28

Variants archive 2025-07-28

  • GSM8K
  • GSM8k (5-shot)
  • GSM8k TR v0.2
  • GSM8k TR
  • gsm8k (5-shots)

5 variant names, as the archive lists them.

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections