{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/sota/natural-language-inference-on-snli","task":{"name":"Natural Language Inference","url":"/task/natural-language-inference","note":null},"dataset":{"name":"SNLI","url":"/dataset/snli"},"category":"Natural Language Processing","categories":["Natural Language Processing","Reasoning"],"category_note":null,"description":"**Natural language inference (NLI)** is the task of determining whether a \"hypothesis\" is \r\ntrue (entailment), false (contradiction), or undetermined (neutral) given a \"premise\".\r\n\r\nExample:\r\n\r\n| Premise | Label | Hypothesis |\r\n| --- | ---| --- |\r\n| A man inspects the uniform of a figure in some East Asian country. | contradiction | The man is sleeping. |\r\n| An older and younger man smiling. | neutral  | Two men are smiling and laughing at the cats playing on the floor. |\r\n| A soccer game with multiple males playing. | entailment | Some men are playing a sport. |\r\n\r\nApproaches used for NLI include earlier symbolic and statistical approaches to more recent deep learning approaches. Benchmark datasets used for NLI include [SNLI](/dataset/snli), [MultiNLI](/dataset/multinli), [SciTail](/dataset/scitail), among others. You can get hands-on practice on the SNLI task by following this [d2l.ai chapter](https://d2l.ai/chapter_natural-language-processing-applications/natural-language-inference-and-dataset.html).\r\n\r\nFurther readings:\r\n\r\n- [Recent Advances in Natural Language Inference: A Survey of Benchmarks, Resources, and Approaches](https://arxiv.org/abs/1904.01172)","description_from":"task","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","rank":"the archive's row order at snapshot; not re-ranked","rows_end_at":"2025-07-28","rows_withheld_as_spam":0,"metric_values":"the archive's strings, untouched"},"metrics":["% Test Accuracy","% Train Accuracy","Parameters","Dev Accuracy","% Dev Accuracy","Accuracy"],"metric_direction":{"note":"inferred from the metric name only (the archive records no direction; the tokens 'score' and 'time' are not used on this page); null = not inferred, chart draws points only","by_metric":{"% Test Accuracy":"higher","% Train Accuracy":"higher","Parameters":null,"Dev Accuracy":"higher","% Dev Accuracy":"higher","Accuracy":"higher"}},"counts":{"rows":98,"rows_with_code":73,"rows_with_paper_page":94,"rows_dated":94,"rows_using_additional_data":2},"rows":[{"rank_in_archive_order":1,"model":"UnitedSynT5 (3B)","metrics":{"% Test Accuracy":"94.7"},"uses_additional_data":true,"paper_date":"2024-12-12","paper":"/paper/first-train-to-generate-then-generate-to","paper_url":"https://arxiv.org/abs/2412.09263v2","paper_title":"First Train to Generate, then Generate to Train: UnitedSynT5 for Few-Shot NLI","code":null,"n_code_links":0,"official":null,"syntology":null},{"rank_in_archive_order":2,"model":"UnitedSynT5 (335M)","metrics":{"% Test Accuracy":"93.5"},"uses_additional_data":true,"paper_date":"2024-12-12","paper":"/paper/first-train-to-generate-then-generate-to","paper_url":"https://arxiv.org/abs/2412.09263v2","paper_title":"First Train to Generate, then Generate to Train: UnitedSynT5 for Few-Shot NLI","code":null,"n_code_links":0,"official":null,"syntology":null},{"rank_in_archive_order":3,"model":"Neural Tree Indexers for Text Understanding","metrics":{"% Test Accuracy":"93.1","Parameters":"355"},"uses_additional_data":false,"paper_date":"2021-04-29","paper":"/paper/entailment-as-few-shot-learner","paper_url":"https://arxiv.org/abs/2104.14690v1","paper_title":"Entailment as Few-Shot Learner","code":"https://github.com/PaddlePaddle/PaddleNLP/tree/develop/examples/few_shot/efl","n_code_links":3,"official":null,"syntology":{"n_ran":1,"n_constructed":0,"n_ran_checked":1,"n_instrument":0,"n_unverified":2,"n_samples":3,"n_pointer_only_licence":0,"sample_list":"/paper/entailment-as-few-shot-learner#ran","syntology_url":"https://syntology.ai/paper/2104.14690","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2104.14690"}}}},{"rank_in_archive_order":4,"model":"EFL (Entailment as Few-shot Learner) + RoBERTa-large","metrics":{"% Test Accuracy":"93.1","% Train Accuracy":"?","Parameters":"355m"},"uses_additional_data":false,"paper_date":"2021-04-29","paper":"/paper/entailment-as-few-shot-learner","paper_url":"https://arxiv.org/abs/2104.14690v1","paper_title":"Entailment as Few-Shot Learner","code":"https://github.com/PaddlePaddle/PaddleNLP/tree/develop/examples/few_shot/efl","n_code_links":3,"official":null,"syntology":{"n_ran":1,"n_constructed":0,"n_ran_checked":1,"n_instrument":0,"n_unverified":2,"n_samples":3,"n_pointer_only_licence":0,"sample_list":"/paper/entailment-as-few-shot-learner#ran","syntology_url":"https://syntology.ai/paper/2104.14690","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2104.14690"}}}},{"rank_in_archive_order":5,"model":"RoBERTa-large+Self-Explaining","metrics":{"% Test Accuracy":"92.3","Parameters":"340"},"uses_additional_data":false,"paper_date":"2020-12-03","paper":"/paper/self-explaining-structures-improve-nlp-models","paper_url":"https://arxiv.org/abs/2012.01786v2","paper_title":"Self-Explaining Structures Improve NLP Models","code":"https://github.com/ShannonAI/Self_Explaining_Structures_Improve_NLP_Models","n_code_links":1,"official":null,"syntology":null},{"rank_in_archive_order":6,"model":"RoBERTa-large + self-explaining layer","metrics":{"% Test Accuracy":"92.3","% Train Accuracy":"?","Parameters":"355m+"},"uses_additional_data":false,"paper_date":"2020-12-03","paper":"/paper/self-explaining-structures-improve-nlp-models","paper_url":"https://arxiv.org/abs/2012.01786v2","paper_title":"Self-Explaining Structures Improve NLP Models","code":"https://github.com/ShannonAI/Self_Explaining_Structures_Improve_NLP_Models","n_code_links":1,"official":null,"syntology":null},{"rank_in_archive_order":7,"model":"CA-MTL","metrics":{"% Test Accuracy":"92.1","% Train Accuracy":"92.6","Parameters":"340m"},"uses_additional_data":false,"paper_date":"2020-09-19","paper":"/paper/conditionally-adaptive-multi-task-learning","paper_url":"https://arxiv.org/abs/2009.09139v3","paper_title":"Conditionally Adaptive Multi-Task Learning: Improving Transfer Learning in NLP Using Fewer Parameters & Less Data","code":"https://github.com/CAMTL/CA-MTL","n_code_links":1,"official":null,"syntology":null},{"rank_in_archive_order":8,"model":"SemBERT","metrics":{"% Test Accuracy":"91.9","% Train Accuracy":"94.4","Parameters":"339m"},"uses_additional_data":false,"paper_date":"2019-09-05","paper":"/paper/semantics-aware-bert-for-language","paper_url":"https://arxiv.org/abs/1909.02209v3","paper_title":"Semantics-aware BERT for Language Understanding","code":"https://github.com/cooelf/SemBERT","n_code_links":1,"official":{"repos":["cooelf/SemBERT"],"state":"official (archive's flag): 7 ran","n_ran":7,"n_constructed":0,"n_ran_no_instrument_failure":5,"n_unverified":5,"ran_from_kinds":["official"]},"syntology":{"n_ran":7,"n_constructed":0,"n_ran_checked":5,"n_instrument":2,"n_unverified":5,"n_samples":12,"n_pointer_only_licence":3,"sample_list":"/paper/semantics-aware-bert-for-language#ran","syntology_url":"https://syntology.ai/paper/1909.02209","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1909.02209"}}}},{"rank_in_archive_order":9,"model":"MT-DNN-SMARTLARGEv0","metrics":{"% Dev Accuracy":"92.6","% Test Accuracy":"91.7"},"uses_additional_data":false,"paper_date":"2019-11-08","paper":"/paper/smart-robust-and-efficient-fine-tuning-for","paper_url":"https://arxiv.org/abs/1911.03437v5","paper_title":"SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimization","code":"https://github.com/namisan/mt-dnn","n_code_links":6,"official":{"repos":["namisan/mt-dnn"],"state":"official: no sample here; runs from other or unrecorded repositories","n_ran":0,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":0,"ran_from_kinds":["listed","unlocated"]},"syntology":{"n_ran":8,"n_constructed":0,"n_ran_checked":5,"n_instrument":3,"n_unverified":0,"n_samples":8,"n_pointer_only_licence":1,"sample_list":"/paper/smart-robust-and-efficient-fine-tuning-for#ran","syntology_url":"https://syntology.ai/paper/1911.03437","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1911.03437"}}}},{"rank_in_archive_order":10,"model":"MT-DNN","metrics":{"% Test Accuracy":"91.6","% Train Accuracy":"97.2","Parameters":"330m"},"uses_additional_data":false,"paper_date":"2019-01-31","paper":"/paper/multi-task-deep-neural-networks-for-natural","paper_url":"https://arxiv.org/abs/1901.11504v2","paper_title":"Multi-Task Deep Neural Networks for Natural Language Understanding","code":"https://github.com/namisan/mt-dnn","n_code_links":7,"official":{"repos":["namisan/mt-dnn"],"state":"community repositories only","n_ran":0,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":0,"ran_from_kinds":["listed"]},"syntology":{"n_ran":9,"n_constructed":0,"n_ran_checked":4,"n_instrument":5,"n_unverified":4,"n_samples":13,"n_pointer_only_licence":4,"sample_list":"/paper/multi-task-deep-neural-networks-for-natural#ran","syntology_url":"https://syntology.ai/paper/1901.11504","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1901.11504"}}}},{"rank_in_archive_order":11,"model":"SJRC (BERT-Large +SRL)","metrics":{"% Test Accuracy":"91.3","% Train Accuracy":"95.7","Parameters":"308m"},"uses_additional_data":false,"paper_date":"2018-09-08","paper":"/paper/i-know-what-you-want-semantic-learning-for","paper_url":"https://arxiv.org/abs/1809.02794v3","paper_title":"Explicit Contextual Semantics for Text Comprehension","code":null,"n_code_links":0,"official":null,"syntology":null},{"rank_in_archive_order":12,"model":"Ntumpha","metrics":{"% Test Accuracy":"90.5","% Train Accuracy":"99.1","Parameters":"220"},"uses_additional_data":false,"paper_date":"2019-01-31","paper":"/paper/multi-task-deep-neural-networks-for-natural","paper_url":"https://arxiv.org/abs/1901.11504v2","paper_title":"Multi-Task Deep Neural Networks for Natural Language Understanding","code":"https://github.com/namisan/mt-dnn","n_code_links":7,"official":{"repos":["namisan/mt-dnn"],"state":"community repositories only","n_ran":0,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":0,"ran_from_kinds":["listed"]},"syntology":{"n_ran":9,"n_constructed":0,"n_ran_checked":4,"n_instrument":5,"n_unverified":4,"n_samples":13,"n_pointer_only_licence":4,"sample_list":"/paper/multi-task-deep-neural-networks-for-natural#ran","syntology_url":"https://syntology.ai/paper/1901.11504","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1901.11504"}}}},{"rank_in_archive_order":13,"model":"Densely-Connected Recurrent and Co-Attentive Network Ensemble","metrics":{"% Test Accuracy":"90.1","% Train Accuracy":"95.0","Parameters":"53.3m"},"uses_additional_data":false,"paper_date":"2018-05-29","paper":"/paper/semantic-sentence-matching-with-densely","paper_url":"http://arxiv.org/abs/1805.11360v2","paper_title":"Semantic Sentence Matching with Densely-connected Recurrent and Co-attentive Information","code":null,"n_code_links":0,"official":null,"syntology":null},{"rank_in_archive_order":14,"model":"MFAE","metrics":{"% Test Accuracy":"90.07","% Train Accuracy":"93.18"},"uses_additional_data":false,"paper_date":"2020-05-07","paper":"/paper/what-do-questions-exactly-ask-mfae-duplicate","paper_url":"https://epubs.siam.org/doi/10.1137/1.9781611976236.26","paper_title":"What Do Questions Exactly Ask? MFAE: Duplicate Question Identification with Multi-Fusion Asking Emphasis","code":"https://github.com/rzhangpku/MFAE","n_code_links":1,"official":null,"syntology":null},{"rank_in_archive_order":15,"model":"Fine-Tuned LM-Pretrained Transformer","metrics":{"% Test Accuracy":"89.9","% Train Accuracy":"96.6","Parameters":"85m"},"uses_additional_data":false,"paper_date":"2018-06-11","paper":"/paper/improving-language-understanding-by","paper_url":"https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf","paper_title":"Improving Language Understanding by Generative Pre-Training","code":"https://github.com/huggingface/transformers","n_code_links":13,"official":null,"syntology":null},{"rank_in_archive_order":16,"model":"300D DMAN Ensemble","metrics":{"% Test Accuracy":"89.6","% Train Accuracy":"96.1","Parameters":"79m"},"uses_additional_data":false,"paper_date":"2019-07-23","paper":"/paper/discourse-marker-augmented-network-with-1","paper_url":"https://arxiv.org/abs/1907.09692v1","paper_title":"Discourse Marker Augmented Network with Reinforcement Learning for Natural Language Inference","code":"https://github.com/ZJULearning/DMP","n_code_links":1,"official":null,"syntology":null},{"rank_in_archive_order":17,"model":"300D DMAN Ensemble","metrics":{"% Test Accuracy":"89.6","% Train Accuracy":"96.1","Parameters":"79m"},"uses_additional_data":false,"paper_date":null,"paper":null,"paper_url":null,"paper_title":"","code":null,"n_code_links":0,"official":null,"syntology":null},{"rank_in_archive_order":18,"model":"150D Multiway Attention Network Ensemble","metrics":{"% Test Accuracy":"89.4","% Train Accuracy":"95.5","Parameters":"58m"},"uses_additional_data":false,"paper_date":"2018-07-01","paper":"/paper/multiway-attention-networks-for-modeling","paper_url":"https://www.ijcai.org/proceedings/2018/0613","paper_title":"Multiway Attention Networks for Modeling Sentence Pairs","code":"https://github.com/zsweet/zsw_AI_model","n_code_links":1,"official":null,"syntology":null},{"rank_in_archive_order":19,"model":"450D DR-BiLSTM Ensemble","metrics":{"% Test Accuracy":"89.3","% Train Accuracy":"94.8","Parameters":"45m"},"uses_additional_data":false,"paper_date":"2018-02-15","paper":"/paper/dr-bilstm-dependent-reading-bidirectional","paper_url":"http://arxiv.org/abs/1802.05577v2","paper_title":"DR-BiLSTM: Dependent Reading Bidirectional LSTM for Natural Language Inference","code":null,"n_code_links":0,"official":null,"syntology":null},{"rank_in_archive_order":20,"model":"300D CAFE Ensemble","metrics":{"% Test Accuracy":"89.3","% Train Accuracy":"92.5","Parameters":"17.5m"},"uses_additional_data":false,"paper_date":"2017-12-30","paper":"/paper/compare-compress-and-propagate-enhancing","paper_url":"http://arxiv.org/abs/1801.00102v2","paper_title":"Compare, Compress and Propagate: Enhancing Neural Architectures with Alignment Factorization for Natural Language Inference","code":null,"n_code_links":0,"official":null,"syntology":null},{"rank_in_archive_order":21,"model":"ESIM + ELMo Ensemble","metrics":{"% Test Accuracy":"89.3","% Train Accuracy":"92.1","Parameters":"40m"},"uses_additional_data":false,"paper_date":"2018-02-15","paper":"/paper/deep-contextualized-word-representations","paper_url":"http://arxiv.org/abs/1802.05365v2","paper_title":"Deep contextualized word representations","code":"https://github.com/flairNLP/flair","n_code_links":46,"official":null,"syntology":{"n_ran":30,"n_constructed":12,"n_ran_checked":24,"n_instrument":6,"n_unverified":28,"n_samples":58,"n_pointer_only_licence":25,"sample_list":"/paper/deep-contextualized-word-representations#ran","syntology_url":"https://syntology.ai/paper/1802.05365","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1802.05365"}}}},{"rank_in_archive_order":22,"model":"KIM Ensemble","metrics":{"% Test Accuracy":"89.1","% Train Accuracy":"93.6","Parameters":"43m"},"uses_additional_data":false,"paper_date":"2017-11-12","paper":"/paper/neural-natural-language-inference-models","paper_url":"http://arxiv.org/abs/1711.04289v3","paper_title":"Neural Natural Language Inference Models Enhanced with External Knowledge","code":"https://github.com/lukecq1231/kim","n_code_links":2,"official":null,"syntology":null},{"rank_in_archive_order":23,"model":"SLRC","metrics":{"% Test Accuracy":"89.1","% Train Accuracy":"89.1","Parameters":"6.1m"},"uses_additional_data":false,"paper_date":"2018-09-08","paper":"/paper/i-know-what-you-want-semantic-learning-for","paper_url":"https://arxiv.org/abs/1809.02794v3","paper_title":"Explicit Contextual Semantics for Text Comprehension","code":null,"n_code_links":0,"official":null,"syntology":null},{"rank_in_archive_order":24,"model":"RE2","metrics":{"% Test Accuracy":"88.9","% Train Accuracy":"94.0","Parameters":"2.8m"},"uses_additional_data":false,"paper_date":"2019-08-01","paper":"/paper/simple-and-effective-text-matching-with-1","paper_url":"https://arxiv.org/abs/1908.00300v1","paper_title":"Simple and Effective Text Matching with Richer Alignment Features","code":"https://github.com/alibaba-edu/simple-effective-text-matching","n_code_links":3,"official":{"repos":["hitvoice/RE2"],"state":"community repositories only","n_ran":0,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":0,"ran_from_kinds":["listed"]},"syntology":{"n_ran":1,"n_constructed":0,"n_ran_checked":1,"n_instrument":0,"n_unverified":0,"n_samples":1,"n_pointer_only_licence":1,"sample_list":"/paper/simple-and-effective-text-matching-with-1#ran","syntology_url":"https://syntology.ai/paper/1908.00300","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1908.00300"}}}},{"rank_in_archive_order":25,"model":"Densely-Connected Recurrent and Co-Attentive Network","metrics":{"% Test Accuracy":"88.9","% Train Accuracy":"93.1","Parameters":"6.7m"},"uses_additional_data":false,"paper_date":"2018-05-29","paper":"/paper/semantic-sentence-matching-with-densely","paper_url":"http://arxiv.org/abs/1805.11360v2","paper_title":"Semantic Sentence Matching with Densely-connected Recurrent and Co-attentive Information","code":null,"n_code_links":0,"official":null,"syntology":null},{"rank_in_archive_order":26,"model":"DEIM","metrics":{"% Test Accuracy":"88.9","% Train Accuracy":"92.6","Parameters":"22m"},"uses_additional_data":false,"paper_date":"2022-03-20","paper":"/paper/deim-an-effective-deep-encoding-and","paper_url":"https://arxiv.org/abs/2203.10482v1","paper_title":"DEIM: An effective deep encoding and interaction model for sentence matching","code":null,"n_code_links":0,"official":null,"syntology":null},{"rank_in_archive_order":27,"model":"448D Densely Interactive Inference Network (DIIN, code) Ensemble","metrics":{"% Test Accuracy":"88.9","% Train Accuracy":"92.3","Parameters":"17m"},"uses_additional_data":false,"paper_date":"2017-09-13","paper":"/paper/natural-language-inference-over-interaction-1","paper_url":"http://arxiv.org/abs/1709.04348v2","paper_title":"Natural Language Inference over Interaction Space","code":"https://github.com/YichenGong/Densely-Interactive-Inference-Network","n_code_links":2,"official":{"repos":["YichenGong/Densely-Interactive-Inference-Network"],"state":"community repositories only","n_ran":0,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":0,"ran_from_kinds":["listed"]},"syntology":{"n_ran":1,"n_constructed":0,"n_ran_checked":0,"n_instrument":1,"n_unverified":0,"n_samples":1,"n_pointer_only_licence":0,"sample_list":"/paper/natural-language-inference-over-interaction-1#ran","syntology_url":"https://syntology.ai/paper/1709.04348","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1709.04348"}}}},{"rank_in_archive_order":28,"model":"300D DMAN","metrics":{"% Test Accuracy":"88.8","% Train Accuracy":"95.4","Parameters":"9.2m"},"uses_additional_data":false,"paper_date":"2019-07-23","paper":"/paper/discourse-marker-augmented-network-with-1","paper_url":"https://arxiv.org/abs/1907.09692v1","paper_title":"Discourse Marker Augmented Network with Reinforcement Learning for Natural Language Inference","code":"https://github.com/ZJULearning/DMP","n_code_links":1,"official":null,"syntology":null},{"rank_in_archive_order":29,"model":"300D DMAN","metrics":{"% Test Accuracy":"88.8","% Train Accuracy":"95.4","Parameters":"9.2m"},"uses_additional_data":false,"paper_date":null,"paper":null,"paper_url":null,"paper_title":"","code":null,"n_code_links":0,"official":null,"syntology":null},{"rank_in_archive_order":30,"model":"BiMPM Ensemble","metrics":{"% Test Accuracy":"88.8","% Train Accuracy":"93.2","Parameters":"6.4m"},"uses_additional_data":false,"paper_date":"2017-02-13","paper":"/paper/bilateral-multi-perspective-matching-for","paper_url":"http://arxiv.org/abs/1702.03814v3","paper_title":"Bilateral Multi-Perspective Matching for Natural Language Sentences","code":"https://github.com/google-research-datasets/paws","n_code_links":10,"official":null,"syntology":{"n_ran":1,"n_constructed":0,"n_ran_checked":0,"n_instrument":1,"n_unverified":3,"n_samples":4,"n_pointer_only_licence":1,"sample_list":"/paper/bilateral-multi-perspective-matching-for#ran","syntology_url":"https://syntology.ai/paper/1702.03814","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1702.03814"}}}},{"rank_in_archive_order":31,"model":"ESIM + ELMo","metrics":{"% Test Accuracy":"88.7","% Train Accuracy":"91.6","Parameters":"8.0m"},"uses_additional_data":false,"paper_date":"2018-02-15","paper":"/paper/deep-contextualized-word-representations","paper_url":"http://arxiv.org/abs/1802.05365v2","paper_title":"Deep contextualized word representations","code":"https://github.com/flairNLP/flair","n_code_links":46,"official":null,"syntology":{"n_ran":30,"n_constructed":12,"n_ran_checked":24,"n_instrument":6,"n_unverified":28,"n_samples":58,"n_pointer_only_licence":25,"sample_list":"/paper/deep-contextualized-word-representations#ran","syntology_url":"https://syntology.ai/paper/1802.05365","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1802.05365"}}}},{"rank_in_archive_order":32,"model":"KIM","metrics":{"% Test Accuracy":"88.6","% Train Accuracy":"94.1","Parameters":"4.3m"},"uses_additional_data":false,"paper_date":"2017-11-12","paper":"/paper/neural-natural-language-inference-models","paper_url":"http://arxiv.org/abs/1711.04289v3","paper_title":"Neural Natural Language Inference Models Enhanced with External Knowledge","code":"https://github.com/lukecq1231/kim","n_code_links":2,"official":null,"syntology":null},{"rank_in_archive_order":33,"model":"600D ESIM + 300D Syntactic TreeLSTM","metrics":{"% Test Accuracy":"88.6","% Train Accuracy":"93.5","Parameters":"7.7m"},"uses_additional_data":false,"paper_date":"2016-09-20","paper":"/paper/enhanced-lstm-for-natural-language-inference","paper_url":"http://arxiv.org/abs/1609.06038v3","paper_title":"Enhanced LSTM for Natural Language Inference","code":"https://github.com/coetaur0/ESIM","n_code_links":12,"official":null,"syntology":null},{"rank_in_archive_order":34,"model":"450D DR-BiLSTM","metrics":{"% Test Accuracy":"88.5","% Train Accuracy":"94.1","Parameters":"7.5m"},"uses_additional_data":false,"paper_date":"2018-02-15","paper":"/paper/dr-bilstm-dependent-reading-bidirectional","paper_url":"http://arxiv.org/abs/1802.05577v2","paper_title":"DR-BiLSTM: Dependent Reading Bidirectional LSTM for Natural Language Inference","code":null,"n_code_links":0,"official":null,"syntology":null},{"rank_in_archive_order":35,"model":"Stochastic Answer Network","metrics":{"% Test Accuracy":"88.5","% Train Accuracy":"93.3","Parameters":"3.5m"},"uses_additional_data":false,"paper_date":"2018-04-21","paper":"/paper/stochastic-answer-networks-for-natural","paper_url":"http://arxiv.org/abs/1804.07888v2","paper_title":"Stochastic Answer Networks for Natural Language Inference","code":"https://github.com/kevinduh/san_mrc","n_code_links":3,"official":null,"syntology":null},{"rank_in_archive_order":36,"model":"300D CAFE","metrics":{"% Test Accuracy":"88.5","% Train Accuracy":"89.8","Parameters":"4.7m"},"uses_additional_data":false,"paper_date":"2017-12-30","paper":"/paper/compare-compress-and-propagate-enhancing","paper_url":"http://arxiv.org/abs/1801.00102v2","paper_title":"Compare, Compress and Propagate: Enhancing Neural Architectures with Alignment Factorization for Natural Language Inference","code":null,"n_code_links":0,"official":null,"syntology":null},{"rank_in_archive_order":37,"model":"150D Multiway Attention Network","metrics":{"% Test Accuracy":"88.3","% Train Accuracy":"94.5","Parameters":"14m"},"uses_additional_data":false,"paper_date":"2018-07-01","paper":"/paper/multiway-attention-networks-for-modeling","paper_url":"https://www.ijcai.org/proceedings/2018/0613","paper_title":"Multiway Attention Networks for Modeling Sentence Pairs","code":"https://github.com/zsweet/zsw_AI_model","n_code_links":1,"official":null,"syntology":null},{"rank_in_archive_order":38,"model":"Biattentive Classification Network + CoVe + Char","metrics":{"% Test Accuracy":"88.1","% Train Accuracy":"88.5","Parameters":"22m"},"uses_additional_data":false,"paper_date":"2017-08-01","paper":"/paper/learned-in-translation-contextualized-word","paper_url":"http://arxiv.org/abs/1708.00107v2","paper_title":"Learned in Translation: Contextualized Word Vectors","code":"https://github.com/salesforce/cove","n_code_links":5,"official":{"repos":["salesforce/cove"],"state":"community repositories only","n_ran":0,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":0,"ran_from_kinds":["listed"]},"syntology":{"n_ran":3,"n_constructed":0,"n_ran_checked":2,"n_instrument":1,"n_unverified":0,"n_samples":3,"n_pointer_only_licence":2,"sample_list":"/paper/learned-in-translation-contextualized-word#ran","syntology_url":"https://syntology.ai/paper/1708.00107","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1708.00107"}}}},{"rank_in_archive_order":39,"model":"aESIM","metrics":{"% Test Accuracy":"88.1"},"uses_additional_data":false,"paper_date":"2018-12-05","paper":"/paper/attention-boosted-sequential-inference-model","paper_url":"http://arxiv.org/abs/1812.01840v2","paper_title":"Attention Boosted Sequential Inference Model","code":null,"n_code_links":0,"official":null,"syntology":null},{"rank_in_archive_order":40,"model":"448D Densely Interactive Inference Network (DIIN, code)","metrics":{"% Test Accuracy":"88.0","% Train Accuracy":"91.2","Parameters":"4.4m"},"uses_additional_data":false,"paper_date":"2017-09-13","paper":"/paper/natural-language-inference-over-interaction-1","paper_url":"http://arxiv.org/abs/1709.04348v2","paper_title":"Natural Language Inference over Interaction Space","code":"https://github.com/YichenGong/Densely-Interactive-Inference-Network","n_code_links":2,"official":{"repos":["YichenGong/Densely-Interactive-Inference-Network"],"state":"community repositories only","n_ran":0,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":0,"ran_from_kinds":["listed"]},"syntology":{"n_ran":1,"n_constructed":0,"n_ran_checked":0,"n_instrument":1,"n_unverified":0,"n_samples":1,"n_pointer_only_licence":0,"sample_list":"/paper/natural-language-inference-over-interaction-1#ran","syntology_url":"https://syntology.ai/paper/1709.04348","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1709.04348"}}}},{"rank_in_archive_order":41,"model":"Enhanced Sequential Inference Model (Chen et al., [2017a])","metrics":{"% Test Accuracy":"88.0"},"uses_additional_data":false,"paper_date":"2016-09-20","paper":"/paper/enhanced-lstm-for-natural-language-inference","paper_url":"http://arxiv.org/abs/1609.06038v3","paper_title":"Enhanced LSTM for Natural Language Inference","code":"https://github.com/coetaur0/ESIM","n_code_links":12,"official":null,"syntology":null},{"rank_in_archive_order":42,"model":"BiMPM","metrics":{"% Test Accuracy":"87.5","% Train Accuracy":"90.9","Parameters":"1.6m"},"uses_additional_data":false,"paper_date":"2017-02-13","paper":"/paper/bilateral-multi-perspective-matching-for","paper_url":"http://arxiv.org/abs/1702.03814v3","paper_title":"Bilateral Multi-Perspective Matching for Natural Language Sentences","code":"https://github.com/google-research-datasets/paws","n_code_links":10,"official":null,"syntology":{"n_ran":1,"n_constructed":0,"n_ran_checked":0,"n_instrument":1,"n_unverified":3,"n_samples":4,"n_pointer_only_licence":1,"sample_list":"/paper/bilateral-multi-perspective-matching-for#ran","syntology_url":"https://syntology.ai/paper/1702.03814","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1702.03814"}}}},{"rank_in_archive_order":43,"model":"300D re-read LSTM","metrics":{"% Test Accuracy":"87.5","% Train Accuracy":"90.7","Parameters":"2.0m"},"uses_additional_data":false,"paper_date":null,"paper":null,"paper_url":null,"paper_title":"","code":null,"n_code_links":0,"official":null,"syntology":null},{"rank_in_archive_order":44,"model":"300D re-read LSTM","metrics":{"% Test Accuracy":"87.5","% Train Accuracy":"90.7","Parameters":"2.0m"},"uses_additional_data":false,"paper_date":"2016-12-01","paper":"/paper/reading-and-thinking-re-read-lstm-unit-for","paper_url":"https://aclanthology.org/C16-1270","paper_title":"Reading and Thinking: Re-read LSTM Unit for Textual Entailment Recognition","code":null,"n_code_links":0,"official":null,"syntology":null},{"rank_in_archive_order":45,"model":"2400D Multiple-Dynamic Self-Attention Model","metrics":{"% Test Accuracy":"87.4","% Train Accuracy":"89.0","Parameters":"7.0m"},"uses_additional_data":false,"paper_date":"2018-08-22","paper":"/paper/dynamic-self-attention-computing-attention","paper_url":"http://arxiv.org/abs/1808.07383v1","paper_title":"Dynamic Self-Attention : Computing Attention over Words Dynamically for Sentence Embedding","code":"https://github.com/dsindex/iclassifier","n_code_links":1,"official":null,"syntology":null},{"rank_in_archive_order":46,"model":"300D Full tree matching NTI-SLSTM-LSTM w/ global attention","metrics":{"% Test Accuracy":"87.3","% Train Accuracy":"88.5","Parameters":"3.2m"},"uses_additional_data":false,"paper_date":"2016-07-15","paper":"/paper/neural-tree-indexers-for-text-understanding","paper_url":"http://arxiv.org/abs/1607.04492v2","paper_title":"Neural Tree Indexers for Text Understanding","code":"https://bitbucket.org/tsendeemts/nti","n_code_links":1,"official":null,"syntology":null},{"rank_in_archive_order":47,"model":"300D 2-layer Bi-CAS-LSTM","metrics":{"% Test Accuracy":"87"},"uses_additional_data":false,"paper_date":"2018-09-07","paper":"/paper/cell-aware-stacked-lstms-for-modeling","paper_url":"https://arxiv.org/abs/1809.02279v2","paper_title":"Cell-aware Stacked LSTMs for Modeling Sentences","code":null,"n_code_links":0,"official":null,"syntology":null},{"rank_in_archive_order":48,"model":"200D decomposable attention feed-forward model with intra-sentence attention","metrics":{"% Test Accuracy":"86.8","% Train Accuracy":"90.5","Parameters":"580k"},"uses_additional_data":false,"paper_date":"2016-06-06","paper":"/paper/a-decomposable-attention-model-for-natural","paper_url":"http://arxiv.org/abs/1606.01933v2","paper_title":"A Decomposable Attention Model for Natural Language Inference","code":"https://github.com/dmlc/gluon-nlp","n_code_links":10,"official":null,"syntology":{"n_ran":4,"n_constructed":0,"n_ran_checked":4,"n_instrument":0,"n_unverified":2,"n_samples":6,"n_pointer_only_licence":0,"sample_list":"/paper/a-decomposable-attention-model-for-natural#ran","syntology_url":"https://syntology.ai/paper/1606.01933","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1606.01933"}}}},{"rank_in_archive_order":49,"model":"200D decomposable attention model with intra-sentence attention","metrics":{"% Test Accuracy":"86.8","% Train Accuracy":"90.5","Parameters":"580k"},"uses_additional_data":false,"paper_date":"2016-06-06","paper":"/paper/a-decomposable-attention-model-for-natural","paper_url":"http://arxiv.org/abs/1606.01933v2","paper_title":"A Decomposable Attention Model for Natural Language Inference","code":"https://github.com/dmlc/gluon-nlp","n_code_links":10,"official":null,"syntology":{"n_ran":4,"n_constructed":0,"n_ran_checked":4,"n_instrument":0,"n_unverified":2,"n_samples":6,"n_pointer_only_licence":0,"sample_list":"/paper/a-decomposable-attention-model-for-natural#ran","syntology_url":"https://syntology.ai/paper/1606.01933","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1606.01933"}}}},{"rank_in_archive_order":50,"model":"600D Dynamic Self-Attention Model","metrics":{"% Test Accuracy":"86.8","% Train Accuracy":"87.3","Parameters":"2.1m"},"uses_additional_data":false,"paper_date":"2018-08-22","paper":"/paper/dynamic-self-attention-computing-attention","paper_url":"http://arxiv.org/abs/1808.07383v1","paper_title":"Dynamic Self-Attention : Computing Attention over Words Dynamically for Sentence Embedding","code":"https://github.com/dsindex/iclassifier","n_code_links":1,"official":null,"syntology":null},{"rank_in_archive_order":51,"model":"CBS-1 + ESIM","metrics":{"% Test Accuracy":"86.73"},"uses_additional_data":false,"paper_date":"2018-12-04","paper":"/paper/parameter-re-initialization-through-cyclical","paper_url":"http://arxiv.org/abs/1812.01216v1","paper_title":"Parameter Re-Initialization through Cyclical Batch Size Schedules","code":null,"n_code_links":0,"official":null,"syntology":null},{"rank_in_archive_order":52,"model":"512D Dynamic Meta-Embeddings","metrics":{"% Test Accuracy":"86.7","% Train Accuracy":"91.6","Parameters":"9m"},"uses_additional_data":false,"paper_date":"2018-04-21","paper":"/paper/dynamic-meta-embeddings-for-improved-sentence","paper_url":"http://arxiv.org/abs/1804.07983v2","paper_title":"Dynamic Meta-Embeddings for Improved Sentence Representations","code":"https://github.com/facebookresearch/DME","n_code_links":3,"official":null,"syntology":null},{"rank_in_archive_order":53,"model":"600D BiLSTM with generalized pooling","metrics":{"% Test Accuracy":"86.6","% Train Accuracy":"94.9","Parameters":"65m"},"uses_additional_data":false,"paper_date":"2018-06-26","paper":"/paper/enhancing-sentence-embedding-with-generalized","paper_url":"http://arxiv.org/abs/1806.09828v1","paper_title":"Enhancing Sentence Embedding with Generalized Pooling","code":"https://github.com/lukecq1231/generalized-pooling","n_code_links":1,"official":null,"syntology":null},{"rank_in_archive_order":54,"model":"600D Hierarchical BiLSTM with Max Pooling (HBMP, code)","metrics":{"% Test Accuracy":"86.6","% Train Accuracy":"89.9","Parameters":"22m"},"uses_additional_data":false,"paper_date":"2018-08-27","paper":"/paper/natural-language-inference-with-hierarchical","paper_url":"https://arxiv.org/abs/1808.08762v2","paper_title":"Sentence Embeddings in NLI with Iterative Refinement Encoders","code":"https://github.com/Helsinki-NLP/HBMP","n_code_links":1,"official":null,"syntology":null},{"rank_in_archive_order":55,"model":"Densely-Connected Recurrent and Co-Attentive Network (encoder)","metrics":{"% Test Accuracy":"86.5","% Train Accuracy":"91.4","Parameters":"5.6m"},"uses_additional_data":false,"paper_date":"2018-05-29","paper":"/paper/semantic-sentence-matching-with-densely","paper_url":"http://arxiv.org/abs/1805.11360v2","paper_title":"Semantic Sentence Matching with Densely-connected Recurrent and Co-attentive Information","code":null,"n_code_links":0,"official":null,"syntology":null},{"rank_in_archive_order":56,"model":"300D Reinforced Self-Attention Network","metrics":{"% Test Accuracy":"86.3","% Train Accuracy":"92.6","Parameters":"3.1m"},"uses_additional_data":false,"paper_date":"2018-01-31","paper":"/paper/reinforced-self-attention-network-a-hybrid-of","paper_url":"http://arxiv.org/abs/1801.10296v2","paper_title":"Reinforced Self-Attention Network: a Hybrid of Hard and Soft Attention for Sequence Modeling","code":"https://github.com/taoshen58/DiSAN","n_code_links":1,"official":null,"syntology":null},{"rank_in_archive_order":57,"model":"Distance-based Self-Attention Network","metrics":{"% Test Accuracy":"86.3","% Train Accuracy":"89.6","Parameters":"4.7m"},"uses_additional_data":false,"paper_date":"2017-12-06","paper":"/paper/distance-based-self-attention-network-for","paper_url":"http://arxiv.org/abs/1712.02047v1","paper_title":"Distance-based Self-Attention Network for Natural Language Inference","code":null,"n_code_links":0,"official":null,"syntology":null},{"rank_in_archive_order":58,"model":"200D decomposable attention feed-forward model","metrics":{"% Test Accuracy":"86.3","% Train Accuracy":"89.5","Parameters":"380k"},"uses_additional_data":false,"paper_date":"2016-06-06","paper":"/paper/a-decomposable-attention-model-for-natural","paper_url":"http://arxiv.org/abs/1606.01933v2","paper_title":"A Decomposable Attention Model for Natural Language Inference","code":"https://github.com/dmlc/gluon-nlp","n_code_links":10,"official":null,"syntology":{"n_ran":4,"n_constructed":0,"n_ran_checked":4,"n_instrument":0,"n_unverified":2,"n_samples":6,"n_pointer_only_licence":0,"sample_list":"/paper/a-decomposable-attention-model-for-natural#ran","syntology_url":"https://syntology.ai/paper/1606.01933","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1606.01933"}}}},{"rank_in_archive_order":59,"model":"200D decomposable attention model","metrics":{"% Test Accuracy":"86.3","% Train Accuracy":"89.5","Parameters":"380k"},"uses_additional_data":false,"paper_date":"2016-06-06","paper":"/paper/a-decomposable-attention-model-for-natural","paper_url":"http://arxiv.org/abs/1606.01933v2","paper_title":"A Decomposable Attention Model for Natural Language Inference","code":"https://github.com/dmlc/gluon-nlp","n_code_links":10,"official":null,"syntology":{"n_ran":4,"n_constructed":0,"n_ran_checked":4,"n_instrument":0,"n_unverified":2,"n_samples":6,"n_pointer_only_licence":0,"sample_list":"/paper/a-decomposable-attention-model-for-natural#ran","syntology_url":"https://syntology.ai/paper/1606.01933","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1606.01933"}}}},{"rank_in_archive_order":60,"model":"450D LSTMN with deep attention fusion","metrics":{"% Test Accuracy":"86.3","% Train Accuracy":"88.5","Parameters":"3.4m"},"uses_additional_data":false,"paper_date":"2016-01-25","paper":"/paper/long-short-term-memory-networks-for-machine","paper_url":"http://arxiv.org/abs/1601.06733v7","paper_title":"Long Short-Term Memory-Networks for Machine Reading","code":"https://github.com/JRC1995/Abstractive-Summarization","n_code_links":3,"official":null,"syntology":{"n_ran":2,"n_constructed":0,"n_ran_checked":1,"n_instrument":1,"n_unverified":0,"n_samples":2,"n_pointer_only_licence":2,"sample_list":"/paper/long-short-term-memory-networks-for-machine#ran","syntology_url":"https://syntology.ai/paper/1601.06733","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1601.06733"}}}},{"rank_in_archive_order":61,"model":"300D mLSTM word-by-word attention model","metrics":{"% Test Accuracy":"86.1","% Train Accuracy":"92.0","Parameters":"1.9m"},"uses_additional_data":false,"paper_date":"2015-12-30","paper":"/paper/learning-natural-language-inference-with-lstm","paper_url":"http://arxiv.org/abs/1512.08849v2","paper_title":"Learning Natural Language Inference with LSTM","code":"https://github.com/shuohangwang/SeqMatchSeq","n_code_links":4,"official":{"repos":["shuohangwang/SeqMatchSeq"],"state":"community repositories only","n_ran":0,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":0,"ran_from_kinds":["listed"]},"syntology":{"n_ran":1,"n_constructed":0,"n_ran_checked":0,"n_instrument":1,"n_unverified":0,"n_samples":1,"n_pointer_only_licence":1,"sample_list":"/paper/learning-natural-language-inference-with-lstm#ran","syntology_url":"https://syntology.ai/paper/1512.08849","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1512.08849"}}}},{"rank_in_archive_order":62,"model":"600D Gumbel TreeLSTM encoders","metrics":{"% Test Accuracy":"86.0","% Train Accuracy":"93.1","Parameters":"10m"},"uses_additional_data":false,"paper_date":"2017-07-10","paper":"/paper/learning-to-compose-task-specific-tree","paper_url":"http://arxiv.org/abs/1707.02786v4","paper_title":"Learning to Compose Task-Specific Tree Structures","code":"https://github.com/jihunchoi/unsupervised-treelstm","n_code_links":1,"official":null,"syntology":null},{"rank_in_archive_order":63,"model":"600D Residual stacked encoders","metrics":{"% Test Accuracy":"86.0","% Train Accuracy":"91.0","Parameters":"29m"},"uses_additional_data":false,"paper_date":"2017-08-07","paper":"/paper/shortcut-stacked-sentence-encoders-for-multi","paper_url":"http://arxiv.org/abs/1708.02312v2","paper_title":"Shortcut-Stacked Sentence Encoders for Multi-Domain Inference","code":"https://github.com/easonnie/multiNLI_encoder","n_code_links":3,"official":null,"syntology":null},{"rank_in_archive_order":64,"model":"Star-Transformer (no cross sentence attention)","metrics":{"% Test Accuracy":"86.0"},"uses_additional_data":false,"paper_date":"2019-02-25","paper":"/paper/star-transformer","paper_url":"https://arxiv.org/abs/1902.09113v3","paper_title":"Star-Transformer","code":"https://github.com/dmlc/dgl","n_code_links":2,"official":{"repos":["dmlc/dgl"],"state":"official: harvested for another paper","n_ran":0,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":0,"ran_from_kinds":[]},"syntology":{"n_ran":0,"n_constructed":0,"n_ran_checked":0,"n_instrument":0,"n_unverified":1,"n_samples":1,"n_pointer_only_licence":0,"sample_list":"/paper/star-transformer#ran","syntology_url":"https://syntology.ai/paper/1902.09113","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1902.09113"}}}},{"rank_in_archive_order":65,"model":"300D CAFE (no cross-sentence attention)","metrics":{"% Test Accuracy":"85.9","% Train Accuracy":"87.3","Parameters":"3.7m"},"uses_additional_data":false,"paper_date":"2017-12-30","paper":"/paper/compare-compress-and-propagate-enhancing","paper_url":"http://arxiv.org/abs/1801.00102v2","paper_title":"Compare, Compress and Propagate: Enhancing Neural Architectures with Alignment Factorization for Natural Language Inference","code":null,"n_code_links":0,"official":null,"syntology":null},{"rank_in_archive_order":66,"model":"1200D REGMAPR (Base+Reg)","metrics":{"% Test Accuracy":"85.9","% Train Accuracy":"–","Parameters":"–"},"uses_additional_data":false,"paper_date":null,"paper":null,"paper_url":null,"paper_title":"","code":null,"n_code_links":0,"official":null,"syntology":null},{"rank_in_archive_order":67,"model":"300D Residual stacked encoders","metrics":{"% Test Accuracy":"85.7","% Train Accuracy":"89.8","Parameters":"9.7m"},"uses_additional_data":false,"paper_date":"2017-08-07","paper":"/paper/shortcut-stacked-sentence-encoders-for-multi","paper_url":"http://arxiv.org/abs/1708.02312v2","paper_title":"Shortcut-Stacked Sentence Encoders for Multi-Domain Inference","code":"https://github.com/easonnie/multiNLI_encoder","n_code_links":3,"official":null,"syntology":null},{"rank_in_archive_order":68,"model":"300D LSTMN with deep attention fusion","metrics":{"% Test Accuracy":"85.7","% Train Accuracy":"87.3","Parameters":"1.7m"},"uses_additional_data":false,"paper_date":"2016-01-25","paper":"/paper/long-short-term-memory-networks-for-machine","paper_url":"http://arxiv.org/abs/1601.06733v7","paper_title":"Long Short-Term Memory-Networks for Machine Reading","code":"https://github.com/JRC1995/Abstractive-Summarization","n_code_links":3,"official":null,"syntology":{"n_ran":2,"n_constructed":0,"n_ran_checked":1,"n_instrument":1,"n_unverified":0,"n_samples":2,"n_pointer_only_licence":2,"sample_list":"/paper/long-short-term-memory-networks-for-machine#ran","syntology_url":"https://syntology.ai/paper/1601.06733","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1601.06733"}}}},{"rank_in_archive_order":69,"model":"300D Gumbel TreeLSTM encoders","metrics":{"% Test Accuracy":"85.6","% Train Accuracy":"91.2","Parameters":"2.9m"},"uses_additional_data":false,"paper_date":"2017-07-10","paper":"/paper/learning-to-compose-task-specific-tree","paper_url":"http://arxiv.org/abs/1707.02786v4","paper_title":"Learning to Compose Task-Specific Tree Structures","code":"https://github.com/jihunchoi/unsupervised-treelstm","n_code_links":1,"official":null,"syntology":null},{"rank_in_archive_order":70,"model":"300D Directional self-attention network encoders","metrics":{"% Test Accuracy":"85.6","% Train Accuracy":"91.1","Parameters":"2.4m"},"uses_additional_data":false,"paper_date":"2017-09-14","paper":"/paper/disan-directional-self-attention-network-for","paper_url":"http://arxiv.org/abs/1709.04696v3","paper_title":"DiSAN: Directional Self-Attention Network for RNN/CNN-Free Language Understanding","code":"https://github.com/taoshen58/DiSAN","n_code_links":3,"official":null,"syntology":null},{"rank_in_archive_order":71,"model":"600D (300+300) Deep Gated Attn. BiLSTM encoders","metrics":{"% Test Accuracy":"85.5","% Train Accuracy":"90.5","Parameters":"12m"},"uses_additional_data":false,"paper_date":"2017-08-04","paper":"/paper/recurrent-neural-network-based-sentence","paper_url":"http://arxiv.org/abs/1708.01353v1","paper_title":"Recurrent Neural Network-Based Sentence Encoder with Gated Attention for Natural Language Inference","code":"https://github.com/lukecq1231/enc_nli","n_code_links":2,"official":null,"syntology":null},{"rank_in_archive_order":72,"model":"300D MMA-NSE encoders with attention","metrics":{"% Test Accuracy":"85.4","% Train Accuracy":"86.9","Parameters":"3.2m"},"uses_additional_data":false,"paper_date":"2016-07-14","paper":"/paper/neural-semantic-encoders","paper_url":"http://arxiv.org/abs/1607.04315v3","paper_title":"Neural Semantic Encoders","code":"https://github.com/Smerity/keras_snli","n_code_links":3,"official":null,"syntology":null},{"rank_in_archive_order":73,"model":"50D stacked TC-LSTMs","metrics":{"% Test Accuracy":"85.1","% Train Accuracy":"86.7","Parameters":"190k"},"uses_additional_data":false,"paper_date":"2016-05-18","paper":"/paper/modelling-interaction-of-sentence-pair-with","paper_url":"http://arxiv.org/abs/1605.05573v2","paper_title":"Modelling Interaction of Sentence Pair with coupled-LSTMs","code":null,"n_code_links":0,"official":null,"syntology":null},{"rank_in_archive_order":74,"model":"600D (300+300) BiLSTM encoders with intra-attention and symbolic preproc.","metrics":{"% Test Accuracy":"85.0","% Train Accuracy":"85.9","Parameters":"2.8m"},"uses_additional_data":false,"paper_date":"2016-05-30","paper":"/paper/learning-natural-language-inference-using","paper_url":"http://arxiv.org/abs/1605.09090v1","paper_title":"Learning Natural Language Inference using Bidirectional LSTM model and Inner-Attention","code":"https://github.com/Smerity/keras_snli","n_code_links":2,"official":null,"syntology":null},{"rank_in_archive_order":75,"model":"Stacked Bi-LSTMs (shortcut connections, max-pooling)","metrics":{"% Test Accuracy":"84.8"},"uses_additional_data":false,"paper_date":"2018-11-02","paper":"/paper/combining-similarity-features-and-deep","paper_url":"http://arxiv.org/abs/1811.00706v1","paper_title":"Combining Similarity Features and Deep Representation Learning for Stance Detection in the Context of Checking Fake News","code":"https://github.com/imran3180/pytorch-nli","n_code_links":3,"official":null,"syntology":null},{"rank_in_archive_order":76,"model":"300D NSE encoders","metrics":{"% Test Accuracy":"84.6","% Train Accuracy":"86.2","Parameters":"3.0m"},"uses_additional_data":false,"paper_date":"2016-07-14","paper":"/paper/neural-semantic-encoders","paper_url":"http://arxiv.org/abs/1607.04315v3","paper_title":"Neural Semantic Encoders","code":"https://github.com/Smerity/keras_snli","n_code_links":3,"official":null,"syntology":null},{"rank_in_archive_order":77,"model":"100D DF-LSTM","metrics":{"% Test Accuracy":"84.6","% Train Accuracy":"85.2","Parameters":"320k"},"uses_additional_data":false,"paper_date":"2016-08-01","paper":"/paper/deep-fusion-lstms-for-text-semantic-matching","paper_url":"https://aclanthology.org/P16-1098","paper_title":"Deep Fusion LSTMs for Text Semantic Matching","code":null,"n_code_links":0,"official":null,"syntology":null},{"rank_in_archive_order":78,"model":"4096D BiLSTM with max-pooling","metrics":{"% Test Accuracy":"84.5","% Train Accuracy":"85.6","Parameters":"40m"},"uses_additional_data":false,"paper_date":"2017-05-05","paper":"/paper/supervised-learning-of-universal-sentence","paper_url":"http://arxiv.org/abs/1705.02364v5","paper_title":"Supervised Learning of Universal Sentence Representations from Natural Language Inference Data","code":"https://github.com/facebookresearch/InferSent","n_code_links":23,"official":{"repos":["facebookresearch/InferSent","facebookresearch/SentEval"],"state":"official (archive's flag): 3 ran","n_ran":3,"n_constructed":0,"n_ran_no_instrument_failure":1,"n_unverified":0,"ran_from_kinds":["listed","official"]},"syntology":{"n_ran":6,"n_constructed":0,"n_ran_checked":2,"n_instrument":4,"n_unverified":1,"n_samples":7,"n_pointer_only_licence":7,"sample_list":"/paper/supervised-learning-of-universal-sentence#ran","syntology_url":"https://syntology.ai/paper/1705.02364","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1705.02364"}}}},{"rank_in_archive_order":79,"model":"Bi-LSTM sentence encoder (max-pooling)","metrics":{"% Test Accuracy":"84.5"},"uses_additional_data":false,"paper_date":"2018-11-02","paper":"/paper/combining-similarity-features-and-deep","paper_url":"http://arxiv.org/abs/1811.00706v1","paper_title":"Combining Similarity Features and Deep Representation Learning for Stance Detection in the Context of Checking Fake News","code":"https://github.com/imran3180/pytorch-nli","n_code_links":3,"official":null,"syntology":null},{"rank_in_archive_order":80,"model":"Stacked Bi-LSTMs (shortcut connections, max-pooling, attention)","metrics":{"% Test Accuracy":"84.4"},"uses_additional_data":false,"paper_date":"2018-11-02","paper":"/paper/combining-similarity-features-and-deep","paper_url":"http://arxiv.org/abs/1811.00706v1","paper_title":"Combining Similarity Features and Deep Representation Learning for Stance Detection in the Context of Checking Fake News","code":"https://github.com/imran3180/pytorch-nli","n_code_links":3,"official":null,"syntology":null},{"rank_in_archive_order":81,"model":"600D (300+300) BiLSTM encoders with intra-attention","metrics":{"% Test Accuracy":"84.2","% Train Accuracy":"84.5","Parameters":"2.8m"},"uses_additional_data":false,"paper_date":"2016-05-30","paper":"/paper/learning-natural-language-inference-using","paper_url":"http://arxiv.org/abs/1605.09090v1","paper_title":"Learning Natural Language Inference using Bidirectional LSTM model and Inner-Attention","code":"https://github.com/Smerity/keras_snli","n_code_links":2,"official":null,"syntology":null},{"rank_in_archive_order":82,"model":"SWEM-max","metrics":{"% Test Accuracy":"83.8"},"uses_additional_data":false,"paper_date":"2018-05-24","paper":"/paper/baseline-needs-more-love-on-simple-word","paper_url":"http://arxiv.org/abs/1805.09843v1","paper_title":"Baseline Needs More Love: On Simple Word-Embedding-Based Models and Associated Pooling Mechanisms","code":"https://github.com/dinghanshen/SWEM","n_code_links":2,"official":null,"syntology":null},{"rank_in_archive_order":83,"model":"100D LSTMs w/ word-by-word attention","metrics":{"% Test Accuracy":"83.5","% Train Accuracy":"85.3","Parameters":"250k"},"uses_additional_data":false,"paper_date":"2015-09-22","paper":"/paper/reasoning-about-entailment-with-neural","paper_url":"http://arxiv.org/abs/1509.06664v4","paper_title":"Reasoning about Entailment with Neural Attention","code":"https://github.com/shyamupa/snli-entailment","n_code_links":7,"official":null,"syntology":{"n_ran":1,"n_constructed":0,"n_ran_checked":0,"n_instrument":1,"n_unverified":0,"n_samples":1,"n_pointer_only_licence":1,"sample_list":"/paper/reasoning-about-entailment-with-neural#ran","syntology_url":"https://syntology.ai/paper/1509.06664","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1509.06664"}}}},{"rank_in_archive_order":84,"model":"300D NTI-SLSTM-LSTM encoders","metrics":{"% Test Accuracy":"83.4","% Train Accuracy":"82.5","Parameters":"4.0m"},"uses_additional_data":false,"paper_date":"2016-07-15","paper":"/paper/neural-tree-indexers-for-text-understanding","paper_url":"http://arxiv.org/abs/1607.04492v2","paper_title":"Neural Tree Indexers for Text Understanding","code":"https://bitbucket.org/tsendeemts/nti","n_code_links":1,"official":null,"syntology":null},{"rank_in_archive_order":85,"model":"600D (300+300) BiLSTM encoders","metrics":{"% Test Accuracy":"83.3","% Train Accuracy":"86.4","Parameters":"2.0m"},"uses_additional_data":false,"paper_date":"2016-05-30","paper":"/paper/learning-natural-language-inference-using","paper_url":"http://arxiv.org/abs/1605.09090v1","paper_title":"Learning Natural Language Inference using Bidirectional LSTM model and Inner-Attention","code":"https://github.com/Smerity/keras_snli","n_code_links":2,"official":null,"syntology":null},{"rank_in_archive_order":86,"model":"300D SPINN-PI encoders","metrics":{"% Test Accuracy":"83.2","% Train Accuracy":"89.2","Parameters":"3.7m"},"uses_additional_data":false,"paper_date":"2016-03-19","paper":"/paper/a-fast-unified-model-for-parsing-and-sentence","paper_url":"http://arxiv.org/abs/1603.06021v3","paper_title":"A Fast Unified Model for Parsing and Sentence Understanding","code":"https://github.com/stanfordnlp/spinn","n_code_links":3,"official":{"repos":["stanfordnlp/spinn"],"state":"official (archive's flag): 3 ran","n_ran":3,"n_constructed":0,"n_ran_no_instrument_failure":3,"n_unverified":0,"ran_from_kinds":["listed","official"]},"syntology":{"n_ran":6,"n_constructed":0,"n_ran_checked":6,"n_instrument":0,"n_unverified":0,"n_samples":6,"n_pointer_only_licence":0,"sample_list":"/paper/a-fast-unified-model-for-parsing-and-sentence#ran","syntology_url":"https://syntology.ai/paper/1603.06021","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1603.06021"}}}},{"rank_in_archive_order":87,"model":"300D Tree-based CNN encoders","metrics":{"% Test Accuracy":"82.1","% Train Accuracy":"83.3","Parameters":"3.5m"},"uses_additional_data":false,"paper_date":"2015-12-28","paper":"/paper/natural-language-inference-by-tree-based","paper_url":"http://arxiv.org/abs/1512.08422v3","paper_title":"Natural Language Inference by Tree-Based Convolution and Heuristic Matching","code":null,"n_code_links":0,"official":null,"syntology":null},{"rank_in_archive_order":88,"model":"1024D GRU encoders w/ unsupervised 'skip-thoughts' pre-training","metrics":{"% Test Accuracy":"81.4","% Train Accuracy":"98.8","Parameters":"15m"},"uses_additional_data":false,"paper_date":"2015-11-19","paper":"/paper/order-embeddings-of-images-and-language","paper_url":"http://arxiv.org/abs/1511.06361v6","paper_title":"Order-Embeddings of Images and Language","code":"https://github.com/ivendrov/order-embedding","n_code_links":2,"official":null,"syntology":null},{"rank_in_archive_order":89,"model":"DELTA (LSTM)","metrics":{"% Test Accuracy":"80.7"},"uses_additional_data":false,"paper_date":"2019-08-02","paper":"/paper/delta-a-deep-learning-based-language","paper_url":"https://arxiv.org/abs/1908.01853v1","paper_title":"DELTA: A DEep learning based Language Technology plAtform","code":"https://github.com/didi/delta","n_code_links":2,"official":null,"syntology":null},{"rank_in_archive_order":90,"model":"300D LSTM encoders","metrics":{"% Test Accuracy":"80.6","% Train Accuracy":"83.9","Parameters":"3.0m"},"uses_additional_data":false,"paper_date":"2016-03-19","paper":"/paper/a-fast-unified-model-for-parsing-and-sentence","paper_url":"http://arxiv.org/abs/1603.06021v3","paper_title":"A Fast Unified Model for Parsing and Sentence Understanding","code":"https://github.com/stanfordnlp/spinn","n_code_links":3,"official":{"repos":["stanfordnlp/spinn"],"state":"official (archive's flag): 3 ran","n_ran":3,"n_constructed":0,"n_ran_no_instrument_failure":3,"n_unverified":0,"ran_from_kinds":["listed","official"]},"syntology":{"n_ran":6,"n_constructed":0,"n_ran_checked":6,"n_instrument":0,"n_unverified":0,"n_samples":6,"n_pointer_only_licence":0,"sample_list":"/paper/a-fast-unified-model-for-parsing-and-sentence#ran","syntology_url":"https://syntology.ai/paper/1603.06021","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1603.06021"}}}},{"rank_in_archive_order":91,"model":"+ Unigram and bigram features","metrics":{"% Test Accuracy":"78.2","% Train Accuracy":"99.7"},"uses_additional_data":false,"paper_date":"2015-08-21","paper":"/paper/a-large-annotated-corpus-for-learning-natural","paper_url":"http://arxiv.org/abs/1508.05326v1","paper_title":"A large annotated corpus for learning natural language inference","code":"https://github.com/kawine/dataset_difficulty","n_code_links":3,"official":null,"syntology":{"n_ran":3,"n_constructed":0,"n_ran_checked":1,"n_instrument":2,"n_unverified":0,"n_samples":3,"n_pointer_only_licence":3,"sample_list":"/paper/a-large-annotated-corpus-for-learning-natural#ran","syntology_url":"https://syntology.ai/paper/1508.05326","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1508.05326"}}}},{"rank_in_archive_order":92,"model":"100D LSTM encoders","metrics":{"% Test Accuracy":"77.6","% Train Accuracy":"84.8","Parameters":"220k"},"uses_additional_data":false,"paper_date":"2015-08-21","paper":"/paper/a-large-annotated-corpus-for-learning-natural","paper_url":"http://arxiv.org/abs/1508.05326v1","paper_title":"A large annotated corpus for learning natural language inference","code":"https://github.com/kawine/dataset_difficulty","n_code_links":3,"official":null,"syntology":{"n_ran":3,"n_constructed":0,"n_ran_checked":1,"n_instrument":2,"n_unverified":0,"n_samples":3,"n_pointer_only_licence":3,"sample_list":"/paper/a-large-annotated-corpus-for-learning-natural#ran","syntology_url":"https://syntology.ai/paper/1508.05326","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1508.05326"}}}},{"rank_in_archive_order":93,"model":"Unlexicalized features","metrics":{"% Test Accuracy":"50.4","% Train Accuracy":"49.4"},"uses_additional_data":false,"paper_date":"2015-08-21","paper":"/paper/a-large-annotated-corpus-for-learning-natural","paper_url":"http://arxiv.org/abs/1508.05326v1","paper_title":"A large annotated corpus for learning natural language inference","code":"https://github.com/kawine/dataset_difficulty","n_code_links":3,"official":null,"syntology":{"n_ran":3,"n_constructed":0,"n_ran_checked":1,"n_instrument":2,"n_unverified":0,"n_samples":3,"n_pointer_only_licence":3,"sample_list":"/paper/a-large-annotated-corpus-for-learning-natural#ran","syntology_url":"https://syntology.ai/paper/1508.05326","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1508.05326"}}}},{"rank_in_archive_order":94,"model":"MT-DNN-SMART_100%ofTrainingData","metrics":{"Dev Accuracy":"91.6"},"uses_additional_data":false,"paper_date":"2019-11-08","paper":"/paper/smart-robust-and-efficient-fine-tuning-for","paper_url":"https://arxiv.org/abs/1911.03437v5","paper_title":"SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimization","code":"https://github.com/namisan/mt-dnn","n_code_links":6,"official":{"repos":["namisan/mt-dnn"],"state":"official: no sample here; runs from other or unrecorded repositories","n_ran":0,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":0,"ran_from_kinds":["listed","unlocated"]},"syntology":{"n_ran":8,"n_constructed":0,"n_ran_checked":5,"n_instrument":3,"n_unverified":0,"n_samples":8,"n_pointer_only_licence":1,"sample_list":"/paper/smart-robust-and-efficient-fine-tuning-for#ran","syntology_url":"https://syntology.ai/paper/1911.03437","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1911.03437"}}}},{"rank_in_archive_order":95,"model":"MT-DNN-SMART_10%ofTrainingData","metrics":{"Dev Accuracy":"88.7"},"uses_additional_data":false,"paper_date":"2019-11-08","paper":"/paper/smart-robust-and-efficient-fine-tuning-for","paper_url":"https://arxiv.org/abs/1911.03437v5","paper_title":"SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimization","code":"https://github.com/namisan/mt-dnn","n_code_links":6,"official":{"repos":["namisan/mt-dnn"],"state":"official: no sample here; runs from other or unrecorded repositories","n_ran":0,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":0,"ran_from_kinds":["listed","unlocated"]},"syntology":{"n_ran":8,"n_constructed":0,"n_ran_checked":5,"n_instrument":3,"n_unverified":0,"n_samples":8,"n_pointer_only_licence":1,"sample_list":"/paper/smart-robust-and-efficient-fine-tuning-for#ran","syntology_url":"https://syntology.ai/paper/1911.03437","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1911.03437"}}}},{"rank_in_archive_order":96,"model":"MT-DNN-SMART_1%ofTrainingData","metrics":{"Dev Accuracy":"86"},"uses_additional_data":false,"paper_date":"2019-11-08","paper":"/paper/smart-robust-and-efficient-fine-tuning-for","paper_url":"https://arxiv.org/abs/1911.03437v5","paper_title":"SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimization","code":"https://github.com/namisan/mt-dnn","n_code_links":6,"official":{"repos":["namisan/mt-dnn"],"state":"official: no sample here; runs from other or unrecorded repositories","n_ran":0,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":0,"ran_from_kinds":["listed","unlocated"]},"syntology":{"n_ran":8,"n_constructed":0,"n_ran_checked":5,"n_instrument":3,"n_unverified":0,"n_samples":8,"n_pointer_only_licence":1,"sample_list":"/paper/smart-robust-and-efficient-fine-tuning-for#ran","syntology_url":"https://syntology.ai/paper/1911.03437","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1911.03437"}}}},{"rank_in_archive_order":97,"model":"MT-DNN-SMART_0.1%ofTrainingData","metrics":{"Dev Accuracy":"82.7"},"uses_additional_data":false,"paper_date":"2019-11-08","paper":"/paper/smart-robust-and-efficient-fine-tuning-for","paper_url":"https://arxiv.org/abs/1911.03437v5","paper_title":"SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimization","code":"https://github.com/namisan/mt-dnn","n_code_links":6,"official":{"repos":["namisan/mt-dnn"],"state":"official: no sample here; runs from other or unrecorded repositories","n_ran":0,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":0,"ran_from_kinds":["listed","unlocated"]},"syntology":{"n_ran":8,"n_constructed":0,"n_ran_checked":5,"n_instrument":3,"n_unverified":0,"n_samples":8,"n_pointer_only_licence":1,"sample_list":"/paper/smart-robust-and-efficient-fine-tuning-for#ran","syntology_url":"https://syntology.ai/paper/1911.03437","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1911.03437"}}}},{"rank_in_archive_order":98,"model":"SplitEE-S","metrics":{"Accuracy":"79.0"},"uses_additional_data":false,"paper_date":"2023-09-17","paper":"/paper/splitee-early-exit-in-deep-neural-networks","paper_url":"https://arxiv.org/abs/2309.09195v1","paper_title":"SplitEE: Early Exit in Deep Neural Networks with Split Computing","code":"https://github.com/Div290/SplitEE/blob/main/README.md","n_code_links":1,"official":null,"syntology":null}],"since_archive":{"claim":"Results that newer papers report for their own method, placed here by Syntology. A model pointed at the cell in the paper's own table; the number was read from that cell and checked against this leaderboard's metric, dataset, split and scale; an independent check that saw this leaderboard's other rows and every other leaderboard on the same dataset accepted it. Not reviewed by the paper's authors or by the archive's editors, and not ranked against the archive rows.","extraction_file_present":true,"measurement":{"test_papers":883,"papers_with_output":881,"judged_true":108,"judged":110,"wilson95_lower":0.9361,"adjudicator":"Claude Opus 5.5","adjudicator_from":"groundwork/59_code_with_papers/LB_EXTRACTOR_PREREG.md (not recorded in lb_measurement.json; the file's frozen_commit, measured_on and counts equal the run the pre-registration records)","adjudicator_sensitivity":[69,70],"adjudicator_specificity":[101,103],"measured_on":"2026-09-24","frozen_commit":"0e3de0df94"},"measurement_note":"blind adjudication by a model (adjudicator) of accepted entries on a held-out split of archive papers, rules frozen before the test; adjudicator_sensitivity and adjudicator_specificity are its own controls as [judged right, of]; it was not measured on the newer papers placed here","measurement_sentence":"Measured before publishing, on held-out archive papers: the extractor was given 883 of them and produced output for 881 (the other 2 had no title and were excluded before any model call, so they are not part of this measurement). A model adjudicator (Claude Opus 5.5), blind to which entries were real, judged 108 of 110 accepted entries correct on those held-out archive papers; its own controls: 69 of 70 archive-confirmed entries judged true, 101 of 103 corrupted copies judged false. It was not measured on the newer papers whose results are placed on this site.","coverage":{"sentence":"Syntology has checked 8,886 of the 18,637 papers on this site that are newer than the archive (for 3,717 of them no archive leaderboard matched the paper's tables, so there was nothing further to check); 733 were read and have nothing to place (arXiv has no HTML version of the paper, or that version has no tables), 41 could not be read (the extractor's reply could not be parsed), and results from the other 8,977 appear after they are checked.","complete":false,"papers_newer_than_archive":18637,"papers_checked":8886,"papers_checked_no_leaderboard_matched":3717,"papers_read_nothing_to_place":733,"papers_could_not_be_read":41,"papers_could_not_be_read_by_status":{"unparseable":41},"papers_extracted_not_yet_verified":0,"boards_without_verdict":3,"papers_not_yet_extracted":8975},"board_status":{"sentence":"Of the 8,888 newer papers whose tables Syntology has read (8,886 checked in full and 2 with a leaderboard still awaiting a verdict), 11 had this board among the leaderboards the extractor considered for their tables (it keeps at most 10 boards per paper and 4 per dataset; the lists of 1,586 of the 8,888 were already full without this board, so if it was crowded out of them it is not counted), and 4 proposed a result; of those 4, 4 gave no cell that passed the extractor's own checks, 0 were rejected by the independent check, 0 await a verdict and 0 were placed.","note":"papers, from the extractor's own records, over the newer papers whose tables were read; proposed = no_cell_passed_checks + rejected_by_independent_check + accepted_not_placed + awaiting_verdict + placed; counts, never a rate","eligible":true,"eligible_note":"false when the extractor never considers this board: it loads only boards with archive rows and at least one metric column (lb_extract.py Boards), so no newer paper can be proposed here; archive_rows and metric_columns say which","archive_rows":98,"metric_columns":6,"papers_read":8888,"papers_read_checked":8886,"papers_read_awaiting_verdict":2,"papers_read_verified":8888,"papers_read_not_yet_verified":0,"papers_read_at_candidate_cap":1596,"papers_read_list_full_without_this_board":1586,"candidates":11,"candidates_capped_note":"candidates = papers that had this board among the leaderboards the extractor considered for their tables; the extractor keeps at most 10 per paper and 4 per dataset, so a board crowded out of a paper's list is not counted and candidates is a floor; papers_read_list_full_without_this_board counts the read papers whose list did not hold this board and had reached 10, or held 4 boards of this board's dataset (the only papers that could have crowded it out); papers_read_at_candidate_cap counts, over every board, the papers whose list reached 10","proposed":4,"no_cell_passed_checks":4,"rejected_by_independent_check":0,"accepted_not_placed":0,"awaiting_verdict":0,"placed":0,"placed_entries_listed":0,"file_lines_unreadable_as_read":0,"since_line":"Since the archive (Syntology): none placed; 4 newer papers judged to report on this board so far"},"judged_to_report":{"heading":"Newer papers whose tables Claude Sonnet 4.5 judged to report on this benchmark (a model's judgement, not a result; newest first by arXiv id)","judged_by":{"model":"Claude Sonnet 4.5","models_as_recorded":["global.anthropic.claude-sonnet-4-5-20250929-v1:0"],"a_record_without_a_model":false,"from":"lb_rows_run_verified.jsonl: each record's `model` field (the extractor's call that wrote decisions[])"},"note":"a model's judgement from the paper's own tables that they report on this leaderboard, not a result: no value from a paper that was not placed is printed or carried here","order":"newest first by arXiv id (the identifier's order, not a rank)","outcomes":{"placed":"placed","no_cell_passed_checks":"no cell passed the extractor's checks","refused_by_rule":"refused by a rule","rejected":"rejected by the independent check","accepted_not_placed":"accepted by the independent check but not placed by this site","awaiting":"awaiting a verdict"},"reason_note":"for refused_by_rule, the rules' plain words for the reason codes the record gives (never the quoted or the actual cell text); null for every other outcome","total":4,"shown_on_page":4,"no_cell_split":{"no_cell_passed_checks_in_ledger":4,"rows_refused_by_rule":4,"rows_no_cell_passed_checks":0},"rows":[{"arxiv_id":"2602.03120","outcome":"refused_by_rule","reason":"the table's column header does not name the leaderboard's metric"},{"arxiv_id":"2511.17399","outcome":"refused_by_rule","reason":"the value is not on the scale of the leaderboard column's archive values"},{"arxiv_id":"2509.17786","outcome":"refused_by_rule","reason":"the table's column header does not name the leaderboard's metric"},{"arxiv_id":"2508.21083","outcome":"refused_by_rule","reason":"the table's column header does not name the leaderboard's metric"}]},"order":"newest first by month (arXiv date, else the arXiv-id month), then arXiv id descending","columns":[],"entries":[]},"syntology":{"read_at":"2026-10-05T00:36:11+00:00","claim":"Per row: of M harvested code samples from that row's paper, n_ran executed on a synthesized fixture (n_ran = n_ran_checked + n_instrument; n_instrument counts failures of Syntology's instrument, not of the code); the other M - n_ran are unverified. Not a reproduction of the row's number; not a correctness claim. n_pointer_only_licence counts samples the site points at rather than redistributes (a licence axis, independent of ran/unverified).","key_notes":{"n_ran_checked":"legacy name, kept unchanged so existing readers do not break: it counts the samples that ran with no instrument failure (honoured, violated, and ran with no contract checked); it does not mean a contract was checked, and the pages print it as 'K with no instrument failure', not 'K checked'","n_constructed":"a sub-count of the samples that ran, never subtracted from them and never a failure: an executed sample whose run returned an instance of its own class (fixture_out_type equals the entry name): the run built an object and did not compute a result (Syntology's RAN record, counts.constructed)"},"rows_with_graph_line":33,"rows_with_any_sample_ran":32,"rows_with_a_run_with_no_instrument_failure":26,"official":{"state_rule":"the archive's official repositories have samples for this paper that ran: 'official (archive's flag): N ran' (N counts them, instrument failures included); they have samples, none ran: 'official: harvested, nothing ran'; they have no sample for this paper and another sample ran: 'community repositories only' when every sample that ran came from a repository the archive lists for the paper but neither marks official nor records as mentioned in it, or one Syntology harvested outside the archive's links, else 'official: no sample here; runs from other or unrecorded repositories' (a repository named in the paper or found in its text by Syntology, or no repository recorded); otherwise 'official: harvested for another paper' or 'official: not harvested'. Absent when the archive marks no repository official. Official is the archive's is_official flag, not an authorship verdict","flag":"the archive's is_official flag for the paper's code link; not a verdict on who wrote the code","rows_with_an_archive_official_repository":17,"rows_whose_official_repository_has_a_sample_that_ran":4,"rows_whose_official_repository_has_a_run_with_no_instrument_failure":4,"rows_community_repositories_only":7,"rows_runs_from_other_or_unrecorded_repositories":5,"note":"rows[].official, per row with a graph line: repos (marked official in the archive), state, n_ran and n_unverified (samples from those repositories), and ran_from_kinds (the kinds of repository the paper's samples that ran came from; 'unlocated' is no repository recorded)"},"filters":{"ran_modes":["a run with no instrument failure","any run, instrument failures included"],"default":"a run with no instrument failure","note":"the page's filters hide rows and never renumber them; they read the Ran cell's counts, the same figures rows[].syntology and rows[].official carry"},"row_route":"rows[].syntology carries sample_list (this site's paper page, #ran), syntology_url (syntology.ai's page for the paper, only when that route probed live) and mcp (the agent call that lists the paper's harvested samples; not a verdict on the paper)","developers":"https://syntology.ai/developers","distinct_papers_with_graph_line":17,"distinct_papers_with_any_sample_ran":16,"distinct_papers_with_a_run_with_no_instrument_failure":12,"samples_over_distinct_papers":{"n_ran":84,"n_constructed":12,"n_ran_checked":56,"n_instrument":28,"n_unverified":46,"n_samples":130,"n_pointer_only_licence":51,"note":"each paper (arXiv id) counted once, however many rows it is behind; this is the page-level figure"},"samples_row_weighted":{"n_ran":184,"n_constructed":24,"n_ran_checked":126,"n_instrument":58,"n_unverified":89,"n_samples":273,"n_pointer_only_licence":93,"note":"row-weighted: a paper behind several rows is counted once per row; inflated relative to samples_over_distinct_papers by design, kept for readers summing the per-row syntology blocks"}},"record_sha256":"6c5707cd217fa12dd8eb85a3eb9faacb4c21155eadfd3336ba72cafb408af14e","record_changed_at":"2026-10-05","record_changed_at_basis":"changed"}