Datasets › HellaSwag

HellaSwag

Introduced by Rowan Zellers et al. in HellaSwag: Can a Machine Really Finish Your Sentence? archive 2025-07-28

HellaSwag is a challenge dataset for evaluating commonsense NLI that is specially hard for state-of-the-art models, though its questions are trivial for humans (>95% accuracy).

Benchmarks archive 2025-07-28

All 6 leaderboards whose dataset resolves to this page shown (sort by any header). "First row" is the archive's own first row at snapshot, in the archive's row order; nothing here re-ranks and metric direction is not asserted.

Syntology 35 newer papers were judged to report on this dataset's benchmarks so far (Claude Sonnet 4.5, reading their tables: a model's judgement, not a result; 0 placed): Sentence Completion · HellaSwag (29 judged, 0 placed), parameter-efficient fine-tuning · HellaSwag (10 judged, 0 placed), Question Answering · HellaSwag (3 judged, 0 placed). Drawn from 8,886 of 18,637 newer papers checked so far (733 more had nothing to place; 41 could not be read).

First row (archive order)PaperCode
Sentence Completion HellaSwag CompassMTL 567M with Tailor Accuracy 96.1 Task Compass: Scaling Multi-task Pre-training with Task Prefix cooelf/compassmtl 89 Compare
parameter-efficient fine-tuning HellaSwag LLaMA2-7b Accuracy (% ) 76.68 GIFT-SW: Gaussian noise Injected Fine-Tuning of Salient... On-Point-RND/GIFT_SW 3 Compare
Question Answering HellaSwag Shakti-LLM (2.5B) Accuracy 52.4 SHAKTI: A 2.5 Billion Parameter Small Language Model... — 1 Compare
Text Generation HellaSwag no rows — — 0 Compare
Text Generation HellaSwag (10-Shot) no rows — — 0 Compare
Text Generation HellaSwag TR no rows — — 0 Compare

Papers archive 2025-07-28

papers with a benchmark row: 39 · with a code link: 32 · where Syntology ran a sample: 20 (18 with a run with no instrument failure, 2 where every run was a failure of Syntology's instrument) Syntology

Show: all papers with a benchmark rowonly where code ran (20 of 39 with a benchmark row: 18 with a run with no instrument failure, 2 where every run was a failure of Syntology's instrument)

Page 1 of 1: papers 1 to 39 of 39 with a leaderboard row on this dataset's benchmarks, newest first by the archive's date (ties by slug; undated papers last).

The archive's own "papers using this dataset" list was never published, so this is the benchmark-backed subset, not that list; the archive's count for this dataset is 994. The Syntology column is from Syntology's graph, stated per sample; it is not part of any archive number. A line reads “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified” (C is a part of N, never taken away from it); the figure “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. “Ran” means the sample executed on a synthesized fixture, not that the paper's result was reproduced. When the archive marks a repository official for the paper, the cell starts with that repository's state (the archive's flag, not a verdict on who wrote the code; “community repositories only” when every sample that ran came from a community repository, “official: no sample here; runs from other or unrecorded repositories” when some came from a repository the paper names or has in its text, or from none recorded); hover it for the repositories the samples that ran came from. Syntology's record for this page last changed ; when this build read Syntology's graph is in the build record.

PaperCodeResultsDateSamples run Syntology
SHAKTI: A 2.5 Billion Parameter Small Language Model Optimized for Edge AI and Low-Resource Environments 0 1 15 Oct 2024 not harvested
GIFT-SW: Gaussian noise Injected Fine-Tuning of Salient Weights for LLMs 1 1 27 Aug 2024 official (archive's flag): 2 ran · 5 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 2 honoured, 0 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified (5 pointer-only for licence)
Mixture-of-Subspaces in Low-Rank Adaptation 1 1 16 Jun 2024 official (archive's flag): 4 ran · 4 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 3 where Syntology's instrument failed) · 2 unverified (6 pointer-only for licence)
MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Experts 2 3 22 Apr 2024 official (archive's flag): 6 ran · 6 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 1 honoured, 1 violated, 1 with no contract checked; 3 where Syntology's instrument failed) · 5 unverified
DoRA: Weight-Decomposed Low-Rank Adaptation 5 1 14 Feb 2024 official (archive's flag): 7 ran · 8 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 8 where Syntology's instrument failed) · 7 unverified (14 pointer-only for licence)
Parameter-Efficient Sparsity Crafting from Dense to Mixture-of-Experts for Instruction Tuning on General Tasks 2 2 5 Jan 2024 not harvested
LLM in a flash: Efficient Large Language Model Inference with Limited Memory 0 2 12 Dec 2023 not harvested
Mamba: Linear-Time Sequence Modeling with Selective State Spaces 35 2 1 Dec 2023 community repositories only · 28 ran (of which 7 constructed an object rather than computing a result; 20 with no instrument failure: 0 honoured, 0 violated, 20 with no contract checked; 8 where Syntology's instrument failed) · 34 unverified (29 pointer-only for licence)
The Falcon Series of Open Language Models 0 3 28 Nov 2023 not harvested
Mistral 7B 6 1 10 Oct 2023 official (archive's flag): 9 ran · 10 ran (of which 0 constructed an object rather than computing a result; 10 with no instrument failure: 0 honoured, 0 violated, 10 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified (2 pointer-only for licence)
Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning 2 3 10 Oct 2023 official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 1 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified
Llama 2: Open Foundation and Fine-Tuned Chat Models 19 4 18 Jul 2023 community repositories only · 33 ran (of which 7 constructed an object rather than computing a result; 20 with no instrument failure: 1 honoured, 1 violated, 18 with no contract checked; 13 where Syntology's instrument failed) · 19 unverified (20 pointer-only for licence)
Stay on topic with Classifier-Free Guidance 0 3 30 Jun 2023 not harvested
The CoT Collection: Improving Zero-shot and Few-shot Learning of Language Models via Chain-of-Thought Fine-Tuning 2 1 23 May 2023 not harvested
PaLM 2 Technical Report 1 3 17 May 2023 not harvested
LaMini-LM: A Diverse Herd of Distilled Models from Large-Scale Instructions 1 6 27 Apr 2023 not harvested
BloombergGPT: A Large Language Model for Finance 2 4 30 Mar 2023 not harvested
GPT-4 Technical Report 11 2 15 Mar 2023 community repositories only · 5 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 0 honoured, 2 violated, 3 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified (1 pointer-only for licence)
LLaMA: Open and Efficient Foundation Language Models 57 4 27 Feb 2023 official: no sample here; runs from other or unrecorded repositories · 37 ran (of which 9 constructed an object rather than computing a result; 25 with no instrument failure: 3 honoured, 0 violated, 22 with no contract checked; 12 where Syntology's instrument failed) · 21 unverified (4 pointer-only for licence)
Exploring the Benefits of Training Expert Language Models over Instruction Tuning 2 1 7 Feb 2023 official (archive's flag): 2 ran · 3 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified (3 pointer-only for licence)
Two is Better than Many? Binary Classification as an Effective Approach to Multi-Choice Question Answering 1 2 29 Oct 2022 official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 0 where Syntology's instrument failed) · 5 unverified
Knowledge-in-Context: Towards Knowledgeable Semi-Parametric Language Models 0 1 28 Oct 2022 not harvested
DiscoSense: Commonsense Reasoning with Discourse Connectives 1 2 22 Oct 2022 not harvested
Task Compass: Scaling Multi-task Pre-training with Task Prefix 1 3 12 Oct 2022 not harvested
Guess the Instruction! Flipped Learning Makes Language Models Stronger Zero-Shot Learners 1 1 6 Oct 2022 not harvested
PaLM: Scaling Language Modeling with Pathways 7 3 5 Apr 2022 32 ran (of which 16 constructed an object rather than computing a result; 24 with no instrument failure: 2 honoured, 1 violated, 21 with no contract checked; 8 where Syntology's instrument failed) · 5 unverified
Training Compute-Optimal Large Language Models 2 1 29 Mar 2022 8 ran (of which 3 constructed an object rather than computing a result; 5 with no instrument failure: 2 honoured, 0 violated, 3 with no contract checked; 3 where Syntology's instrument failed) · 3 unverified (4 pointer-only for licence)
Efficient Language Modeling with Sparse all-MLP 0 5 14 Mar 2022 not harvested
Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model 2 2 28 Jan 2022 not harvested
Scaling Language Models: Methods, Analysis & Insights from Training Gopher 3 1 8 Dec 2021 not harvested
Finetuned Language Models Are Zero-Shot Learners 8 2 3 Sep 2021 official: harvested for another paper · 0 ran · 1 unverified
LoRA: Low-Rank Adaptation of Large Language Models 74 1 17 Jun 2021 community repositories only · 51 ran (of which 19 constructed an object rather than computing a result; 43 with no instrument failure: 1 honoured, 0 violated, 42 with no contract checked; 8 where Syntology's instrument failed) · 33 unverified (30 pointer-only for licence)
UNICORN on RAINBOW: A Universal Commonsense Reasoning Model on a New Multitask Benchmark 1 1 24 Mar 2021 official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 0 violated, 3 with no contract checked; 0 where Syntology's instrument failed) · 2 unverified
Muppet: Massive Multi-task Representations with Pre-Finetuning 2 1 26 Jan 2021 not harvested
DeBERTa: Decoding-enhanced BERT with Disentangled Attention 14 1 5 Jun 2020 official: no sample here; runs from other or unrecorded repositories · 4 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 2 honoured, 1 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 9 unverified (3 pointer-only for licence)
Language Models are Few-Shot Learners 67 3 28 May 2020 community repositories only · 45 ran (of which 0 constructed an object rather than computing a result; 37 with no instrument failure: 3 honoured, 2 violated, 32 with no contract checked; 8 where Syntology's instrument failed) · 20 unverified (7 pointer-only for licence)
Towards Generalizable Neuro-Symbolic Systems for Commonsense Question Answering 0 1 30 Oct 2019 not harvested
RoBERTa: A Robustly Optimized BERT Pretraining Approach 67 2 26 Jul 2019 community repositories only · 37 ran (of which 11 constructed an object rather than computing a result; 36 with no instrument failure: 2 honoured, 0 violated, 34 with no contract checked; 1 where Syntology's instrument failed) · 11 unverified (24 pointer-only for licence)
HellaSwag: Can a Machine Really Finish Your Sentence? 2 9 19 May 2019 4 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 0 honoured, 0 violated, 4 with no contract checked; 0 where Syntology's instrument failed) · 2 unverified (4 pointer-only for licence)

Dataset loaders archive 2025-07-28

8 loaders as listed in the archive; links are outbound and not re-checked here.

Tasks archive 2025-07-28

License archive 2025-07-28

MIT

Modalities archive 2025-07-28

No modality tagged.

Languages archive 2025-07-28

No language tagged.

Variants archive 2025-07-28

  • HellaSwag
  • HellaSwag (10-Shot)
  • HellaSwag TR

3 variant names, as the archive lists them.

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections