Datasets › Penn Treebank

Penn Treebank

Introduced in Building a Large Annotated Corpus of English: The Penn Treebank1 Jan 1993 archive 2025-07-28

The English Penn Treebank (PTB) corpus, and in particular the section of the corpus corresponding to the articles of Wall Street Journal (WSJ), is one of the most known and used corpus for the evaluation of models for sequence labelling. The task consists of annotating each word with its Part-of-Speech tag. In the most common split of this corpus, sections from 0 to 18 are used for training (38 219 sentences, 912 344 tokens), sections from 19 to 21 are used for validation (5 527 sentences, 131 768 tokens), and sections from 22 to 24 are used for testing (5 462 sentences, 129 654 tokens). The corpus is also commonly used for character-level and word-level Language Modelling.

Source: Seq2Biseq: Bidirectional Output-wise Recurrent Neural Networks for Sequence Modelling Image Source: https://dl.acm.org/doi/10.5555/972470.972475

Benchmarks archive 2025-07-28

All 10 leaderboards whose dataset resolves to this page shown (sort by any header). "First row" is the archive's own first row at snapshot, in the archive's row order; nothing here re-ranks and metric direction is not asserted.

First row (archive order)PaperCode
Language Modelling Penn Treebank (Word Level) GPT-3 (Zero-Shot) Test perplexity 20.5 Language Models are Few-Shot Learners ggml-org/llama.cpp +66 43 Compare
Constituency Parsing Penn Treebank Hashing + XLNet F1 score 96.43 To be Continuous, or to be Discrete, Those are Bits of Questions speedcell4/parserker 27 Compare
Dependency Parsing Penn Treebank Label Attention Layer + HPSG + XLNet LAS 96.26 Rethinking Self-Attention: Towards Interpretability in... KhalilMrini/LAL-Parser +1 22 Compare
Language Modelling Penn Treebank (Character Level) Mogrifier LSTM + dynamic eval Bit per Character (BPC) 1.083 Mogrifier LSTM deepmind/lamb +2 20 Compare
Part-Of-Speech Tagging Penn Treebank SALE-BART encoder Accuracy 98.15 Sequence Alignment Ensemble with a Single Neural Network... — 20 Compare
Chunking Penn Treebank ACE F1 score 97.3 Automated Concatenation of Embeddings for Structured Prediction Alibaba-NLP/ACE +1 8 Compare
Unsupervised Dependency Parsing Penn Treebank Ensemble (selected w/ society entropy) UAS 67.3 Error Diversity Matters: An Error-Resistant Ensemble... manga-uofa/ed4udp 6 Compare
Open Information Extraction Penn Treebank Deepstruct zero-shot F1 51 DeepStruct: Pretraining of Language Models for Structure... cgraywang/deepstruct 4 Compare
Stochastic Optimization Penn Treebank (Character Level) 3x1000 LSTM - 500 Epochs AvaGrad Bit per Character (BPC) 1.175 Domain-independent Dominance of Adaptive Methods lolemacs/avagrad 4 Compare
Constituency Grammar Induction Penn Treebank no rows — — 0 Compare

Papers archive 2025-07-28

papers with a benchmark row: 108 · with a code link: 93 · where Syntology ran a sample: 41 (34 with a run with no instrument failure, 7 where every run was a failure of Syntology's instrument) Syntology

Show: all papers with a benchmark rowonly where code ran (41 of 108 with a benchmark row: 34 with a run with no instrument failure, 7 where every run was a failure of Syntology's instrument)

Page 1 of 2: papers 1 to 100 of 108 with a leaderboard row on this dataset's benchmarks, newest first by the archive's date (ties by slug; undated papers last).

The archive's own "papers using this dataset" list was never published, so this is the benchmark-backed subset, not that list; the archive's count for this dataset is 1,006. The Syntology column is from Syntology's graph, stated per sample; it is not part of any archive number. A line reads “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified” (C is a part of N, never taken away from it); the figure “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. “Ran” means the sample executed on a synthesized fixture, not that the paper's result was reproduced. When the archive marks a repository official for the paper, the cell starts with that repository's state (the archive's flag, not a verdict on who wrote the code; “community repositories only” when every sample that ran came from a community repository, “official: no sample here; runs from other or unrecorded repositories” when some came from a repository the paper names or has in its text, or from none recorded); hover it for the repositories the samples that ran came from. Syntology's record for this page last changed ; when this build read Syntology's graph is in the build record.

PaperCodeResultsDateSamples run Syntology
Error Diversity Matters: An Error-Resistant Ensemble Method for Unsupervised Dependency Parsing 1 1 16 Dec 2024 not harvested
To be Continuous, or to be Discrete, Those are Bits of Questions 1 2 12 Jun 2024 not harvested
Advancing State of the Art in Language Modeling 1 1 28 Nov 2023 not harvested
Enhancing Structure-aware Encoder with Extremely Limited Data for Graph-based Dependency Parsing 1 1 1 Oct 2022 not harvested
Sequence Alignment Ensemble with a Single Neural Network for Sequence Labeling 0 1 7 Jul 2022 not harvested
DeepStruct: Pretraining of Language Models for Structure Prediction 1 3 21 May 2022 official (archive's flag): 11 ran · 11 ran (of which 0 constructed an object rather than computing a result; 10 with no instrument failure: 0 honoured, 1 violated, 9 with no contract checked; 1 where Syntology's instrument failed) · 2 unverified (3 pointer-only for licence)
Sequential Alignment Methods for Ensemble Part-of-Speech Tagging 1 1 23 Mar 2022 not harvested
Investigating Non-local Features for Neural Constituency Parsing 1 1 27 Sep 2021 not harvested
Zero-Shot Information Extraction as a Unified Text-to-Triple Translation 1 1 23 Sep 2021 not harvested
N-ary Constituent Tree Parsing with Recursive Semi-Markov Model 1 1 26 Jul 2021 not harvested
AutoDropout: Learning Dropout Patterns to Regularize Deep Networks 1 1 5 Jan 2021 not harvested
Learning Associative Inference Using Fast Weight Memory 1 1 16 Nov 2020 official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 2 honoured, 0 violated, 0 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified
Strongly Incremental Constituency Parsing with Graph Neural Networks 3 1 27 Oct 2020 official (archive's flag): 2 ran · 2 ran (of which 1 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 1 where Syntology's instrument failed) · 3 unverified
Improving Constituency Parsing with Span Attention 1 1 15 Oct 2020 official (archive's flag): 1 ran · 1 ran (of which 1 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified; the one sample that ran constructed an object rather than computing a result
Automated Concatenation of Embeddings for Structured Prediction 2 2 10 Oct 2020 not harvested
Second-Order Neural Dependency Parsing with Message Passing and End-to-End Training 1 1 10 Oct 2020 not harvested
Fast and Accurate Neural CRF Constituency Parsing 2 3 9 Aug 2020 official (archive's flag): 1 ran · 3 ran (of which 2 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified
Language Models are Few-Shot Learners 67 1 28 May 2020 community repositories only · 45 ran (of which 0 constructed an object rather than computing a result; 37 with no instrument failure: 3 honoured, 2 violated, 32 with no contract checked; 8 where Syntology's instrument failed) · 20 unverified (7 pointer-only for licence)
Efficient Second-Order TreeCRF for Neural Dependency Parsing 2 1 3 May 2020 official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 1 unverified
Recursive Non-Autoregressive Graph-to-Graph Transformer for Dependency Parsing with Iterative Refinement 1 1 29 Mar 2020 not harvested
Addressing Some Limitations of Transformers with Feedback Memory 4 1 21 Feb 2020 official: no sample here; runs from other or unrecorded repositories · 3 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 2 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (2 pointer-only for licence)
Recurrent Highway Networks with Grouped Auxiliary Memory 4 1 13 Dec 2019 not harvested
Domain-independent Dominance of Adaptive Methods 1 4 4 Dec 2019 not harvested
Gating Revisited: Deep Multi-layer RNNs That Can Be Trained 3 1 25 Nov 2019 not harvested
Seq-U-Net: A One-Dimensional Causal U-Net for Efficient Sequence Modelling 1 4 14 Nov 2019 official (archive's flag): 4 ran · 4 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 0 honoured, 0 violated, 4 with no contract checked; 0 where Syntology's instrument failed) · 2 unverified
Generalizing Natural Language Analysis through Span-relation Representations 3 3 10 Nov 2019 not harvested
Rethinking Self-Attention: Towards Interpretability in Neural Parsing 2 2 10 Nov 2019 official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 2 unverified (3 pointer-only for licence)
Deep Independently Recurrent Neural Network (IndRNN) 1 3 11 Oct 2019 not harvested
Mogrifier LSTM 3 3 4 Sep 2019 not harvested
Deep Equilibrium Models 11 1 3 Sep 2019 community repositories only · 9 ran (of which 0 constructed an object rather than computing a result; 7 with no instrument failure: 1 honoured, 0 violated, 6 with no contract checked; 2 where Syntology's instrument failed) · 3 unverified (4 pointer-only for licence)
Hierarchically-Refined Label Attention Network for Sequence Labeling 2 1 23 Aug 2019 not harvested
R-Transformer: Recurrent Neural Network Enhanced Transformer 2 2 12 Jul 2019 official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified (2 pointer-only for licence)
Head-Driven Phrase Structure Grammar Parsing on Penn Treebank 1 3 5 Jul 2019 official (archive's flag): 8 ran · 8 ran (of which 0 constructed an object rather than computing a result; 7 with no instrument failure: 1 honoured, 0 violated, 6 with no contract checked; 1 where Syntology's instrument failed) · 1 unverified (1 pointer-only for licence)
Graph-based Dependency Parsing with Graph Neural Networks 1 1 1 Jul 2019 not harvested
Improving Neural Language Modeling via Adversarial Training 1 1 10 Jun 2019 not harvested
Discrete Flows: Invertible Generative Models of Discrete Data 2 1 24 May 2019 not harvested
Deep Residual Output Layers for Neural Language Generation 1 2 14 May 2019 not harvested
Tetra-Tagging: Word-Synchronous Parsing with Linear-Time Inference 2 1 22 Apr 2019 not harvested
Language Models with Transformers 1 1 20 Apr 2019 official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 1 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (2 pointer-only for licence)
Left-to-Right Dependency Parsing with Pointer Networks 2 1 20 Mar 2019 not harvested
Cloze-driven Pretraining of Self-attention Networks 0 1 19 Mar 2019 not harvested
Partially Shuffling the Training Data to Improve Language Models 1 2 11 Mar 2019 not harvested
Language Models are Unsupervised Multitask Learners 21 1 14 Feb 2019 not harvested
Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context 37 1 9 Jan 2019 official (archive's flag): 8 ran · 67 ran (of which 37 constructed an object rather than computing a result; 49 with no instrument failure: 4 honoured, 1 violated, 44 with no contract checked; 18 where Syntology's instrument failed) · 76 unverified (43 pointer-only for licence)
Learning Better Internal Structure of Words for Sequence Labeling 0 2 29 Oct 2018 not harvested
Trellis Networks for Sequence Modeling 1 2 15 Oct 2018 official (archive's flag): 7 ran · 7 ran (of which 0 constructed an object rather than computing a result; 6 with no instrument failure: 2 honoured, 0 violated, 4 with no contract checked; 1 where Syntology's instrument failed) · 1 unverified (2 pointer-only for licence)
Semi-Supervised Sequence Modeling with Cross-View Training 2 2 22 Sep 2018 not harvested
FRAGE: Frequency-Agnostic Word Representation 2 1 18 Sep 2018 not harvested
Direct Output Connection for a High-Rank Language Model 1 3 30 Aug 2018 not harvested
Improved Language Modeling by Decoding the Past 0 2 14 Aug 2018 not harvested
Contextual String Embeddings for Sequence Labeling 1 2 1 Aug 2018 not harvested
An improved neural network model for joint POS tagging and dependency parsing 1 1 11 Jul 2018 not harvested
An Empirical Study of Building a Strong Baseline for Constituency Parsing 1 1 1 Jul 2018 not harvested
DARTS: Differentiable Architecture Search 59 1 24 Jun 2018 official: harvested, nothing ran · 74 ran (of which 30 constructed an object rather than computing a result; 59 with no instrument failure: 10 honoured, 1 violated, 48 with no contract checked; 15 where Syntology's instrument failed) · 82 unverified (54 pointer-only for licence)
NCRF++: An Open-source Neural Sequence Labeling Toolkit 3 2 14 Jun 2018 community repositories only · 9 ran (of which 0 constructed an object rather than computing a result; 8 with no instrument failure: 0 honoured, 0 violated, 8 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified
Pushing the bounds of dropout 1 1 23 May 2018 not harvested
Morphosyntactic Tagging with a Meta-BiLSTM Model over Context Sensitive Token Encodings 2 1 21 May 2018 not harvested
Sentence-State LSTM for Text Representation 2 1 7 May 2018 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (1 pointer-only for licence)
Stack-Pointer Networks for Dependency Parsing 3 1 3 May 2018 not harvested
Constituency Parsing with a Self-Attentive Encoder 5 1 2 May 2018 community repositories only · 11 ran (of which 0 constructed an object rather than computing a result; 11 with no instrument failure: 0 honoured, 0 violated, 11 with no contract checked; 0 where Syntology's instrument failed) · 3 unverified
An Analysis of Neural Language Modeling at Multiple Scales 12 2 22 Mar 2018 community repositories only · 14 ran (of which 0 constructed an object rather than computing a result; 13 with no instrument failure: 2 honoured, 0 violated, 11 with no contract checked; 1 where Syntology's instrument failed) · 4 unverified (4 pointer-only for licence)
Independently Recurrent Neural Network (IndRNN): Building A Longer and Deeper RNN 11 1 13 Mar 2018 community repositories only · 5 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 0 honoured, 0 violated, 4 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (1 pointer-only for licence)
An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling 35 3 4 Mar 2018 community repositories only · 9 ran (of which 0 constructed an object rather than computing a result; 8 with no instrument failure: 0 honoured, 1 violated, 7 with no contract checked; 1 where Syntology's instrument failed) · 1 unverified (1 pointer-only for licence)
Efficient Neural Architecture Search via Parameter Sharing 28 1 9 Feb 2018 18 ran (of which 10 constructed an object rather than computing a result; 12 with no instrument failure: 1 honoured, 0 violated, 11 with no contract checked; 6 where Syntology's instrument failed) · 20 unverified (25 pointer-only for licence)
Robust Multilingual Part-of-Speech Tagging via Adversarial Training 1 1 14 Nov 2017 not harvested
Breaking the Softmax Bottleneck: A High-Rank RNN Language Model 9 2 10 Nov 2017 official (archive's flag): 2 ran · 12 ran (of which 0 constructed an object rather than computing a result; 11 with no instrument failure: 0 honoured, 0 violated, 11 with no contract checked; 1 where Syntology's instrument failed) · 11 unverified (1 pointer-only for licence)
Fraternal Dropout 1 1 31 Oct 2017 not harvested
Dynamic Evaluation of Neural Sequence Models 3 1 21 Sep 2017 official: harvested, nothing ran · 0 ran · 1 unverified
Empower Sequence Labeling with Task-Aware Neural Language Model 3 1 13 Sep 2017 not harvested
Gradual Learning of Recurrent Neural Networks 1 1 29 Aug 2017 not harvested
Regularizing and Optimizing LSTM Language Models 45 2 7 Aug 2017 community repositories only · 6 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 2 honoured, 0 violated, 0 with no contract checked; 4 where Syntology's instrument failed) · 1 unverified (7 pointer-only for licence)
In-Order Transition-based Constituent Parsing 2 1 17 Jul 2017 not harvested
Improving Neural Parsing by Disentangling Model Combination and Reranking Effects 0 1 10 Jul 2017 not harvested
Attention Is All You Need 595 1 12 Jun 2017 official: no sample here; runs from other or unrecorded repositories · 610 ran (of which 293 constructed an object rather than computing a result; 529 with no instrument failure: 45 honoured, 15 violated, 469 with no contract checked; 81 where Syntology's instrument failed) · 336 unverified (451 pointer-only for licence)
Fast-Slow Recurrent Neural Networks 1 2 24 May 2017 not harvested
Semi-supervised Multitask Learning for Sequence Labeling 3 1 24 Apr 2017 not harvested
Transfer Learning for Sequence Tagging with Hierarchical Recurrent Networks 4 1 18 Mar 2017 official (archive's flag): 8 ran · 8 ran (of which 0 constructed an object rather than computing a result; 8 with no instrument failure: 0 honoured, 0 violated, 8 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified
What Do Recurrent Neural Network Grammars Learn About Syntax? 1 1 17 Nov 2016 not harvested
Attending to Characters in Neural Sequence Labeling Models 0 1 14 Nov 2016 not harvested
Deep Biaffine Attention for Neural Dependency Parsing 26 2 6 Nov 2016 14 ran (of which 0 constructed an object rather than computing a result; 12 with no instrument failure: 0 honoured, 1 violated, 11 with no contract checked; 2 where Syntology's instrument failed) · 5 unverified (2 pointer-only for licence)
A Joint Many-Task Model: Growing a Neural Network for Multiple NLP Tasks 2 1 5 Nov 2016 community repositories only · 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (1 pointer-only for licence)
Neural Architecture Search with Reinforcement Learning 12 2 5 Nov 2016 community repositories only · 5 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 2 honoured, 1 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified (3 pointer-only for licence)
Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modeling 5 1 4 Nov 2016 not harvested
Parsing as Language Modeling 1 1 1 Nov 2016 not harvested
HyperNetworks 10 1 27 Sep 2016 3 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 1 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 1 unverified (3 pointer-only for licence)
Distilling an Ensemble of Greedy Dependency Parsers into One MST Parser 1 1 24 Sep 2016 not harvested
Deep multi-task learning with low level tasks supervised at lower layers 0 1 1 Aug 2016 not harvested
Recurrent Highway Networks 6 1 12 Jul 2016 official: harvested, nothing ran · 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 4 unverified
Supertagging With LSTMs 0 1 1 Jun 2016 not harvested
Multilingual Part-of-Speech Tagging with Bidirectional Long Short-Term Memory Models and Auxiliary Loss 3 1 19 Apr 2016 official (archive's flag): 1 ran · 4 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 0 honoured, 1 violated, 3 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified (1 pointer-only for licence)
Globally Normalized Transition-Based Neural Networks 1 1 19 Mar 2016 not harvested
Simple and Accurate Dependency Parsing Using Bidirectional LSTM Feature Representations 1 2 14 Mar 2016 official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 0 violated, 3 with no contract checked; 0 where Syntology's instrument failed) · 4 unverified
Training with Exploration Improves a Greedy Stack-LSTM Parser 0 1 11 Mar 2016 not harvested
End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF 25 1 4 Mar 2016 24 ran (of which 0 constructed an object rather than computing a result; 22 with no instrument failure: 2 honoured, 0 violated, 20 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified (3 pointer-only for licence)
Recurrent Neural Network Grammars 6 1 25 Feb 2016 official: no sample here; runs from other or unrecorded repositories · 8 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 1 violated, 2 with no contract checked; 5 where Syntology's instrument failed) · 2 unverified (6 pointer-only for licence)
A Theoretically Grounded Application of Dropout in Recurrent Neural Networks 14 2 16 Dec 2015 3 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 0 violated, 3 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified
Syntactic Parse Fusion 1 1 1 Sep 2015 not harvested
Bidirectional LSTM-CRF Models for Sequence Tagging 25 1 9 Aug 2015 8 ran (of which 0 constructed an object rather than computing a result; 8 with no instrument failure: 0 honoured, 0 violated, 8 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified
Finding Function in Form: Compositional Character Models for Open Vocabulary Word Representation 1 2 9 Aug 2015 not harvested
Structured Training for Neural Network Transition-Based Parsing 0 1 19 Jun 2015 not harvested

Dataset loaders archive 2025-07-28

2 loaders as listed in the archive; links are outbound and not re-checked here.

Tasks archive 2025-07-28

License archive 2025-07-28

Custom

Modalities archive 2025-07-28

Languages archive 2025-07-28

No language tagged.

Variants archive 2025-07-28

  • Penn Treebank
  • Penn Treebank (Word Level)
  • Penn Treebank (Character Level)
  • Penn Treebank (Character Level) 3x1000 LSTM - 500 Epochs

4 variant names, as the archive lists them.

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections