Papers › Know What You Don't Know: Unanswerable Questions for SQuAD

Know What You Don't Know: Unanswerable Questions for SQuAD

11 Jun 2018ACL 2018 7arXiv:1806.03822archive 2025-07-28

Pranav Rajpurkar, Robin Jia, Percy Liang

Extractive reading comprehension systems can often locate the correct answer to a question in a context document, but they also tend to make unreliable guesses on questions for which the correct answer is not stated in the context. Existing datasets either focus exclusively on answerable questions, or use automatically generated unanswerable questions that are easy to identify. To address these weaknesses, we present SQuAD 2.0, the latest version of the Stanford Question Answering Dataset (SQuAD). SQuAD 2.0 combines existing SQuAD data with over 50,000 unanswerable questions written adversarially by crowdworkers to look similar to answerable ones. To do well on SQuAD 2.0, systems must not only answer questions when possible, but also determine when no answer is supported by the paragraph and abstain from answering. SQuAD 2.0 is a challenging natural language understanding task for existing models: a strong neural system that gets 86% F1 on SQuAD 1.1 achieves only 66% F1 on SQuAD 2.0.

PaperPDFConference PDFCode

In Syntology View this paper on Syntology, its page in Syntology's graph. That page lists the repositories linked to the paper, the abstract and the calls for agents.

Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.

For agents, on Syntology's MCP service (how to connect):

Code

CLUEbenchmark/LGEB mentioned on GitHubpytorchSyntology: not harvested report
ElizaLo/Question-Answering-based-on-SQuAD mentioned on GitHubpytorchSyntology: not harvested report
expertailab/spaceqa mentioned on GitHubpytorchSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to RoBERTa: A Robustly Optimized BERT Pretraining Approach (arXiv:1907.11692; that paper's own run record: 37 ran (of which 11 constructed an object rather than computing a result; 36 with no instrument failure: 2 honoured, 0 violated, 34 with no contract checked; 1 where Syntology's instrument failed) · 11 unverified); nothing checks that they implement this paper's method. report
jopokemine/Dissertation mentioned on GitHubpytorchSyntology: not harvested report
leozhoujf/DataSciComp mentioned on GitHubpaddleSyntology: not harvested report
mtzcorporations/nlp_teamjodka mentioned on GitHubpytorchSyntology: not harvested report
okanvk/ENELPI---Soru-Cevap-Sistemi mentioned on GitHubpytorchMITSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators (arXiv:2003.10555; that paper's own run record: 31 ran (of which 7 constructed an object rather than computing a result; 18 with no instrument failure: 2 honoured, 2 violated, 14 with no contract checked; 13 where Syntology's instrument failed) · 9 unverified); nothing checks that they implement this paper's method. report
okanvk/Turkish-Reading-Comprehension-Question-Answering-Dataset mentioned on GitHubpytorchSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (arXiv:1810.04805; that paper's own run record: 300 ran (of which 75 constructed an object rather than computing a result; 234 with no instrument failure: 18 honoured, 5 violated, 211 with no contract checked; 66 where Syntology's instrument failed) · 359 unverified) and ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators (arXiv:2003.10555; that paper's own run record: 31 ran (of which 7 constructed an object rather than computing a result; 18 with no instrument failure: 2 honoured, 2 violated, 14 with no contract checked; 13 where Syntology's instrument failed) · 9 unverified); nothing checks that they implement this paper's method. report
pablonm3/cs224u mentioned on GitHubpytorchSyntology: not harvested report
zhuzhicai/SQuAD2.0-Baseline-Test-with-BiDAF-No-Answer mentioned on GitHubtfSyntology: not harvested report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

A paper named beside a repository with no sample linked to this paper is shown with that paper's own run record, not this paper's: “ran” means executed on a synthesized input, not that the code is correct, and “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. Papers are listed in arXiv-id order, at most three per repository.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Natural Language UnderstandingQuestion AnsweringReading Comprehension

Results from the paper archive 2025-07-28

No leaderboard rows for this paper in the archive.

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections