Datasets › WSC

WSC (Winograd Schema Challenge)

Introduced in The Winograd Schema Challenge1 Jan 2012 archive 2025-07-28

The Winograd Schema Challenge was introduced both as an alternative to the Turing Test and as a test of a system’s ability to do commonsense reasoning. A Winograd schema is a pair of sentences differing in one or two words with a highly ambiguous pronoun, resolved differently in the two sentences, that appears to require commonsense knowledge to be resolved correctly. The examples were designed to be easily solvable by humans but difficult for machines, in principle requiring a deep understanding of the content of the text and the situation it describes.

The original Winograd Schema Challenge dataset consisted of 100 Winograd schemas constructed manually by AI experts. As of 2020 there are 285 examples available; however, the last 12 examples were only added recently. To ensure consistency with earlier models, several authors often prefer to report the performance on the first 273 examples only. These datasets are usually referred to as WSC285 and WSC273, respectively.

Source: https://arxiv.org/pdf/2004.13831.pdf Image Source: https://arxiv.org/pdf/1907.11983.pdf

Benchmarks archive 2025-07-28

All 2 leaderboards whose dataset resolves to this page shown (sort by any header). "First row" is the archive's own first row at snapshot, in the archive's row order; nothing here re-ranks and metric direction is not asserted.

Syntology 1 newer paper was judged to report on this dataset's benchmarks so far (Claude Sonnet 4.5, reading their tables: a model's judgement, not a result; 0 placed): Classification · WSC (1 judged, 0 placed). Drawn from 8,886 of 18,637 newer papers checked so far (733 more had nothing to place; 41 could not be read).

First row (archive order)PaperCode
Coreference Resolution Winograd Schema Challenge PaLM 540B (fine-tuned) Accuracy 100 PaLM: Scaling Language Modeling with Pathways lucidrains/CoCa-pytorch +6 82 Compare
Classification WSC OPT-1.3B Test Accuracy 64.16% Achieving Dimension-Free Communication in Federated... ZidongLiu/DeComFL 2 Compare

Papers archive 2025-07-28

papers with a benchmark row: 38 · with a code link: 31 · where Syntology ran a sample: 18 (14 with a run with no instrument failure, 4 where every run was a failure of Syntology's instrument) Syntology

Show: all papers with a benchmark rowonly where code ran (18 of 38 with a benchmark row: 14 with a run with no instrument failure, 4 where every run was a failure of Syntology's instrument)

Page 1 of 1: papers 1 to 38 of 38 with a leaderboard row on this dataset's benchmarks, newest first by the archive's date (ties by slug; undated papers last).

The archive's own "papers using this dataset" list was never published, so this is the benchmark-backed subset, not that list; the archive's count for this dataset is 361. The Syntology column is from Syntology's graph, stated per sample; it is not part of any archive number. A line reads “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified” (C is a part of N, never taken away from it); the figure “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. “Ran” means the sample executed on a synthesized fixture, not that the paper's result was reproduced. When the archive marks a repository official for the paper, the cell starts with that repository's state (the archive's flag, not a verdict on who wrote the code; “community repositories only” when every sample that ran came from a community repository, “official: no sample here; runs from other or unrecorded repositories” when some came from a repository the paper names or has in its text, or from none recorded); hover it for the repositories the samples that ran came from. Syntology's record for this page last changed ; when this build read Syntology's graph is in the build record.

PaperCodeResultsDateSamples run Syntology
Achieving Dimension-Free Communication in Federated Learning via Zeroth-Order Optimization 1 2 24 May 2024 official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 1 honoured, 0 violated, 0 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified
The CoT Collection: Improving Zero-shot and Few-shot Learning of Language Models via Chain-of-Thought Fine-Tuning 2 1 23 May 2023 not harvested
PaLM 2 Technical Report 1 3 17 May 2023 not harvested
LaMini-LM: A Diverse Herd of Distilled Models from Large-Scale Instructions 1 5 27 Apr 2023 not harvested
Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling 4 4 3 Apr 2023 not harvested
Exploring the Benefits of Training Expert Language Models over Instruction Tuning 2 1 7 Feb 2023 official (archive's flag): 2 ran · 3 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified (3 pointer-only for licence)
Hungry Hungry Hippos: Towards Language Modeling with State Space Models 3 3 28 Dec 2022 official (archive's flag): 1 ran · 7 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 1 honoured, 0 violated, 2 with no contract checked; 4 where Syntology's instrument failed) · 8 unverified
Toward Efficient Language Model Pretraining and Downstream Adaptation via Self-Evolution: A Case Study on SuperGLUE 0 2 4 Dec 2022 not harvested
Knowledge-in-Context: Towards Knowledgeable Semi-Parametric Language Models 0 1 28 Oct 2022 not harvested
Scaling Instruction-Finetuned Language Models 9 1 20 Oct 2022 8 ran (of which 1 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 7 where Syntology's instrument failed) · 9 unverified (2 pointer-only for licence)
Guess the Instruction! Flipped Learning Makes Language Models Stronger Zero-Shot Learners 1 1 6 Oct 2022 not harvested
Ask Me Anything: A simple strategy for prompting language models 3 3 5 Oct 2022 official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified
AlexaTM 20B: Few-Shot Learning Using a Large-Scale Multilingual Seq2Seq Model 1 1 2 Aug 2022 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified
N-Grammer: Augmenting Transformers with latent n-grams 2 1 13 Jul 2022 official (archive's flag): 5 ran · 6 ran (of which 0 constructed an object rather than computing a result; 6 with no instrument failure: 0 honoured, 0 violated, 6 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified
UL2: Unifying Language Learning Paradigms 2 2 10 May 2022 community repositories only · 15 ran (of which 0 constructed an object rather than computing a result; 15 with no instrument failure: 0 honoured, 0 violated, 15 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified
PaLM: Scaling Language Modeling with Pathways 7 4 5 Apr 2022 32 ran (of which 16 constructed an object rather than computing a result; 24 with no instrument failure: 2 honoured, 1 violated, 21 with no contract checked; 8 where Syntology's instrument failed) · 5 unverified
ST-MoE: Designing Stable and Transferable Sparse Expert Models 3 2 17 Feb 2022 community repositories only · 5 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 3 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified (5 pointer-only for licence)
On Generalization in Coreference Resolution 2 2 20 Sep 2021 not harvested
Finetuned Language Models Are Zero-Shot Learners 8 2 3 Sep 2021 official: harvested for another paper · 0 ran · 1 unverified
Back to Square One: Artifact Detection, Training and Commonsense Disentanglement in the Winograd Schema 0 7 16 Apr 2021 not harvested
DeBERTa: Decoding-enhanced BERT with Disentangled Attention 14 1 5 Jun 2020 official: no sample here; runs from other or unrecorded repositories · 4 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 2 honoured, 1 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 9 unverified (3 pointer-only for licence)
Language Models are Few-Shot Learners 67 1 28 May 2020 community repositories only · 45 ran (of which 0 constructed an object rather than computing a result; 37 with no instrument failure: 3 honoured, 2 violated, 32 with no contract checked; 8 where Syntology's instrument failed) · 20 unverified (7 pointer-only for licence)
Generative Data Augmentation for Commonsense Reasoning 1 1 24 Apr 2020 official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 2 unverified (4 pointer-only for licence)
TTTTTackling WinoGrande Schemas 0 1 18 Mar 2020 not harvested
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer 57 1 23 Oct 2019 21 ran (of which 0 constructed an object rather than computing a result; 20 with no instrument failure: 1 honoured, 0 violated, 19 with no contract checked; 1 where Syntology's instrument failed) · 10 unverified
A Hybrid Neural Network Model for Commonsense Reasoning 3 1 27 Jul 2019 official: harvested for another paper · 0 ran · 1 unverified
WinoGrande: An Adversarial Winograd Schema Challenge at Scale 10 4 24 Jul 2019 not harvested
Attention Is (not) All You Need for Commonsense Reasoning 2 3 31 May 2019 not harvested
A Surprisingly Robust Trick for Winograd Schema Challenge 2 4 15 May 2019 not harvested
SocialIQA: Commonsense Reasoning about Social Interactions 1 2 22 Apr 2019 5 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 0 honoured, 0 violated, 5 with no contract checked; 0 where Syntology's instrument failed) · 5 unverified
Unsupervised Deep Structured Semantic Models for Commonsense Reasoning 0 5 3 Apr 2019 not harvested
Language Models are Unsupervised Multitask Learners 21 1 14 Feb 2019 not harvested
How Reasonable are Common-Sense Reasoning Tasks: A Case-Study on the Winograd Schema Challenge and SWAG 1 4 5 Nov 2018 not harvested
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding 534 1 11 Oct 2018 official: no sample here; runs from other or unrecorded repositories · 300 ran (of which 75 constructed an object rather than computing a result; 234 with no instrument failure: 18 honoured, 5 violated, 211 with no contract checked; 66 where Syntology's instrument failed) · 359 unverified (164 pointer-only for licence)
A Knowledge Hunting Framework for Common Sense Reasoning 0 1 2 Oct 2018 not harvested
A Simple Method for Commonsense Reasoning 2 3 7 Jun 2018 4 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 0 honoured, 0 violated, 4 with no contract checked; 0 where Syntology's instrument failed) · 3 unverified
Attention Is All You Need 595 1 12 Jun 2017 official: no sample here; runs from other or unrecorded repositories · 610 ran (of which 293 constructed an object rather than computing a result; 529 with no instrument failure: 45 honoured, 15 violated, 469 with no contract checked; 81 where Syntology's instrument failed) · 336 unverified (451 pointer-only for licence)
Commonsense Knowledge Enhanced Embeddings for Solving Pronoun Disambiguation Problems in Winograd Schema Challenge 0 1 13 Nov 2016 not harvested

Dataset loaders archive 2025-07-28

6 loaders as listed in the archive; links are outbound and not re-checked here.

Tasks archive 2025-07-28

License archive 2025-07-28

CC BY 4.0

Modalities archive 2025-07-28

Languages archive 2025-07-28

Variants archive 2025-07-28

  • Winograd Schema Challenge
  • WSC
  • winograd_wsc

3 variant names, as the archive lists them.

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections