Papers › Language Models are Unsupervised Multitask Learners

Language Models are Unsupervised Multitask Learners

14 Feb 2019Preprint 2019 2archive 2025-07-28

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever

Natural language processing tasks, such as question answering, machine translation, reading comprehension, and summarization, are typically approached with supervised learning on taskspecific datasets. We demonstrate that language models begin to learn these tasks without any explicit supervision when trained on a new dataset of millions of webpages called WebText. When conditioned on a document plus questions, the answers generated by the language model reach 55 F1 on the CoQA dataset - matching or exceeding the performance of 3 out of 4 baseline systems without using the 127,000+ training examples. The capacity of the language model is essential to the success of zero-shot task transfer and increasing it improves performance in a log-linear fashion across tasks. Our largest model, GPT-2, is a 1.5B parameter Transformer that achieves state of the art results on 7 out of 8 tested language modeling datasets in a zero-shot setting but still underfits WebText. Samples from the model reflect these improvements and contain coherent paragraphs of text. These findings suggest a promising path towards building language processing systems which learn to perform tasks from their naturally occurring demonstrations.

PaperPDFCode

Syntology This paper has no arXiv id in the archive, and this site joins papers to Syntology's graph by arXiv id, so this page does not say whether the graph holds it.

Code

21 repositories listed; official and paper-mentioned ones first.

openai/gpt-2 officialmentioned in papertfNOASSERTIONSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to Selecting Language Models for Social Science: Start Small, Start Open, and Validate (arXiv:2601.10926; that paper's own run record: 5 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 1 honoured, 0 violated, 0 with no contract checked; 4 where Syntology's instrument failed) · 2 unverified); nothing checks that they implement this paper's method. report
2023-MindSpore-1/ms-code-154 mindsporeSyntology: not harvested report
MS-P3/code5 mindsporeSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to Instruct-MusicGen: Unlocking Text-to-Music Editing for Music Language Models via... (arXiv:2405.18386; that paper's own run record: 2 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified); nothing checks that they implement this paper's method. report
PaddlePaddle/PaddleNLP paddleSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to ERNIE 3.0 Tiny: Frustratingly Simple Method to Improve Task-Agnostic Distillation... (arXiv:2301.03416; that paper's own run record: 2 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified); nothing checks that they implement this paper's method. report
VachanVY/gpt.jax jaxMITSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to Language Models are Few-Shot Learners (arXiv:2005.14165; that paper's own run record: 45 ran (of which 0 constructed an object rather than computing a result; 37 with no instrument failure: 3 honoured, 2 violated, 32 with no contract checked; 8 where Syntology's instrument failed) · 20 unverified); nothing checks that they implement this paper's method. report
aananda-giri/gpt2-nepali pytorchSyntology: not harvested report
abhaskumarsinha/Corpus2GPT pytorchSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to Language Models are Few-Shot Learners (arXiv:2005.14165; that paper's own run record: 45 ran (of which 0 constructed an object rather than computing a result; 37 with no instrument failure: 3 honoured, 2 violated, 32 with no contract checked; 8 where Syntology's instrument failed) · 20 unverified) and LLaMA: Open and Efficient Foundation Language Models (arXiv:2302.13971; that paper's own run record: 37 ran (of which 9 constructed an object rather than computing a result; 25 with no instrument failure: 3 honoured, 0 violated, 22 with no contract checked; 12 where Syntology's instrument failed) · 21 unverified); nothing checks that they implement this paper's method. report
abhaskumarsinha/MinimalGPT tfSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to Attention Is All You Need (arXiv:1706.03762; that paper's own run record: 610 ran (of which 293 constructed an object rather than computing a result; 529 with no instrument failure: 45 honoured, 15 violated, 469 with no contract checked; 81 where Syntology's instrument failed) · 336 unverified) and Language Models are Few-Shot Learners (arXiv:2005.14165; that paper's own run record: 45 ran (of which 0 constructed an object rather than computing a result; 37 with no instrument failure: 3 honoured, 2 violated, 32 with no contract checked; 8 where Syntology's instrument failed) · 20 unverified); nothing checks that they implement this paper's method. report
affjljoo3581/GPT2 pytorchSyntology: not harvested report
akanyaani/gpt-2-tensorflow2.0 tfSyntology: not harvested report
akanyaani/minGPTF tfSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to Attention Is All You Need (arXiv:1706.03762; that paper's own run record: 610 ran (of which 293 constructed an object rather than computing a result; 529 with no instrument failure: 45 honoured, 15 violated, 469 with no contract checked; 81 where Syntology's instrument failed) · 336 unverified); nothing checks that they implement this paper's method. report
huggingface/swift-coreml-transformers pytorchApache-2.0Syntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter (arXiv:1910.01108; that paper's own run record: 21 ran (of which 5 constructed an object rather than computing a result; 12 with no instrument failure: 3 honoured, 1 violated, 8 with no contract checked; 9 where Syntology's instrument failed) · 6 unverified); nothing checks that they implement this paper's method. report
huggingface/transformers pytorchSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to Attention Is All You Need (arXiv:1706.03762; that paper's own run record: 610 ran (of which 293 constructed an object rather than computing a result; 529 with no instrument failure: 45 honoured, 15 violated, 469 with no contract checked; 81 where Syntology's instrument failed) · 336 unverified), HuggingFace's Transformers: State-of-the-art Natural Language Processing (arXiv:1910.03771; that paper's own run record: 3 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified), DeBERTa: Decoding-enhanced BERT with Disentangled Attention (arXiv:2006.03654; that paper's own run record: 4 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 2 honoured, 1 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 9 unverified) and 1 more (every one in this page's JSON twin); nothing checks that they implement this paper's method. report
imcaspar/gpt2-ml tfSyntology: not harvested report
jankrepl/mildlyoverfitted jaxSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to Axiomatic Attribution for Deep Networks (arXiv:1703.01365; that paper's own run record: 36 ran (of which 13 constructed an object rather than computing a result; 21 with no instrument failure: 3 honoured, 1 violated, 17 with no contract checked; 15 where Syntology's instrument failed) · 20 unverified), mixup: Beyond Empirical Risk Minimization (arXiv:1710.09412; that paper's own run record: 36 ran (of which 5 constructed an object rather than computing a result; 14 with no instrument failure: 0 honoured, 0 violated, 14 with no contract checked; 22 where Syntology's instrument failed) · 11 unverified), The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks (arXiv:1803.03635; that paper's own run record: 12 ran (of which 2 constructed an object rather than computing a result; 10 with no instrument failure: 0 honoured, 0 violated, 10 with no contract checked; 2 where Syntology's instrument failed) · 13 unverified) and 6 more (every one in this page's JSON twin); nothing checks that they implement this paper's method. report
lvyufeng/bert4ms mindsporeApache-2.0Syntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to RoBERTa: A Robustly Optimized BERT Pretraining Approach (arXiv:1907.11692; that paper's own run record: 37 ran (of which 11 constructed an object rather than computing a result; 36 with no instrument failure: 2 honoured, 0 violated, 34 with no contract checked; 1 where Syntology's instrument failed) · 11 unverified); nothing checks that they implement this paper's method. report
milmor/GPT tfMITSyntology: not harvested report
mindspore-ai/models mindsporeSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to AdaBin: Improving Binary Neural Networks with Adaptive Binary Sets (arXiv:2208.08084; that paper's own run record: 4 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 0 honoured, 0 violated, 4 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified); nothing checks that they implement this paper's method. report
minimaxir/gpt-2-simple tfNOASSERTIONSyntology: not harvested report
ramanakshay/nanogpt pytorchMITSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to Language Models are Few-Shot Learners (arXiv:2005.14165; that paper's own run record: 45 ran (of which 0 constructed an object rather than computing a result; 37 with no instrument failure: 3 honoured, 2 violated, 32 with no contract checked; 8 where Syntology's instrument failed) · 20 unverified); nothing checks that they implement this paper's method. report
varun-suresh/experiments-with-gpt2 pytorchSyntology: not harvested report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

A paper named beside a repository with no sample linked to this paper is shown with that paper's own run record, not this paper's: “ran” means executed on a synthesized input, not that the code is correct, and “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. Papers are listed in arXiv-id order, at most three per repository.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Common Sense ReasoningCoreference ResolutionData-to-Text GenerationDialogue State TrackingDocument SummarizationLanguage ModelingLanguage ModellingMachine TranslationMulti-Task LearningMulti-task Language UnderstandingQuestion AnsweringReading ComprehensionResponse GenerationSentiment AnalysisText GenerationTranslation

Datasets

Introduced by this paper, per the archive.

WebText

Results from the paper archive 2025-07-28

TaskDatasetModelMetricValueRank at snapshotLeaderboardReport
Coreference Resolution Winograd Schema Challenge GPT-2-XL 1.5B Accuracy 70.7 #33 of 82 Archive leaderboard report
Dialogue State Tracking SIMMC2.0 GPT-2 Act F1 94.5 #4 of 5 Archive leaderboard report
Dialogue State Tracking SIMMC2.0 GPT-2 Slot F1 81.7 #4 of 5 Archive leaderboard report
Document Summarization CNN / Daily Mail GPT-2 ROUGE-1 29.34 #26 of 26 Archive leaderboard report
Document Summarization CNN / Daily Mail GPT-2 ROUGE-2 8.27 #26 of 26 Archive leaderboard report
Document Summarization CNN / Daily Mail GPT-2 ROUGE-L 26.58 #26 of 26 Archive leaderboard report
Language Modelling LAMBADA GPT-2 1.5B (Zero Shot) Accuracy 63.24 #29 of 37 Archive leaderboard report
Language Modelling LAMBADA GPT-2 1.5B (Zero Shot) Perplexity 8.63 #29 of 37 Archive leaderboard report
Language Modelling One Billion Word GPT-2 Number of params 1.54B #23 of 27 Archive leaderboard report
Language Modelling One Billion Word GPT-2 PPL 42.16 #23 of 27 Archive leaderboard report
Language Modelling Penn Treebank (Word Level) GPT-2 Params 1542M #3 of 43 Archive leaderboard report
Language Modelling Penn Treebank (Word Level) GPT-2 Test perplexity 35.76 #3 of 43 Archive leaderboard report
Language Modelling Text8 GPT-2 Bit per Character (BPC) 0.98 #1 of 24 Archive leaderboard report
Language Modelling Text8 GPT-2 Number of params 1542M #1 of 24 Archive leaderboard report
Language Modelling WikiText-103 GPT-2 Full Number of params 1542M #25 of 89 Archive leaderboard report
Language Modelling WikiText-103 GPT-2 Full Test perplexity 17.48 #25 of 89 Archive leaderboard report
Language Modelling WikiText-103 GPT-2 Large Number of params 774M #46 of 89 Archive leaderboard report
Language Modelling WikiText-103 GPT-2 Large Test perplexity 22.05 #46 of 89 Archive leaderboard report
Language Modelling WikiText-103 GPT-2 Medium Number of params 355M #63 of 89 Archive leaderboard report
Language Modelling WikiText-103 GPT-2 Medium Test perplexity 26.37 #63 of 89 Archive leaderboard report
Language Modelling WikiText-103 GPT-2 Small Number of params 124M #79 of 89 Archive leaderboard report
Language Modelling WikiText-103 GPT-2 Small Test perplexity 37.50 #79 of 89 Archive leaderboard report
Language Modelling WikiText-2 GPT-2 Number of params 1542M #6 of 38 Archive leaderboard report
Language Modelling WikiText-2 GPT-2 Test perplexity 18.34 #6 of 38 Archive leaderboard report
Language Modelling WikiText-2 GPT-2 (large) Number of params 762M #7 of 38 Archive leaderboard report
Language Modelling WikiText-2 GPT-2 (large) Test perplexity 19.93 #7 of 38 Archive leaderboard report
Language Modelling WikiText-2 GPT-2 (medium) Number of params 345M #8 of 38 Archive leaderboard report
Language Modelling WikiText-2 GPT-2 (medium) Test perplexity 22.76 #8 of 38 Archive leaderboard report
Language Modelling WikiText-2 GPT-2 (small) Number of params 117M #9 of 38 Archive leaderboard report
Language Modelling WikiText-2 GPT-2 (small) Test perplexity 29.41 #9 of 38 Archive leaderboard report
Language Modelling enwik8 GPT-2 (48 layers, h=1600) Bit per Character (BPC) 0.93 #1 of 42 Archive leaderboard report
Language Modelling enwik8 GPT-2 (48 layers, h=1600) Number of params 1542M #1 of 42 Archive leaderboard report
Question Answering FEVER Zero-shot EM 50 #7 of 8 Archive leaderboard report
Question Answering WebQuestions Zero-shot EM 43 #12 of 37 Archive leaderboard report
Response Generation SIMMC2.0 GPT-2 BLEU 19.2 #5 of 5 Archive leaderboard report
Sentiment Analysis IMDb GPT-2 Finetuned Accuracy 92.36 #31 of 49 Archive leaderboard report
Text Generation OpenWebText GPT2-124M eval_loss 3.12 #3 of 3 Archive leaderboard report

Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.

Methods

Introduced by this paper: GPT-2

AdamAttentionAttention DropoutBPECosine AnnealingDense ConnectionsDiscriminative Fine-TuningDropoutGPT-2Layer NormalizationLinear LayerLinear Warmup With Cosine AnnealingMulti-Head AttentionResidual ConnectionSoftmaxWeight Decay

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections