{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/task/natural-language-inference","name":"Natural Language Inference","slug":"natural-language-inference","description_markdown":"**Natural language inference (NLI)** is the task of determining whether a \"hypothesis\" is \r\ntrue (entailment), false (contradiction), or undetermined (neutral) given a \"premise\".\r\n\r\nExample:\r\n\r\n| Premise | Label | Hypothesis |\r\n| --- | ---| --- |\r\n| A man inspects the uniform of a figure in some East Asian country. | contradiction | The man is sleeping. |\r\n| An older and younger man smiling. | neutral  | Two men are smiling and laughing at the cats playing on the floor. |\r\n| A soccer game with multiple males playing. | entailment | Some men are playing a sport. |\r\n\r\nApproaches used for NLI include earlier symbolic and statistical approaches to more recent deep learning approaches. Benchmark datasets used for NLI include [SNLI](/dataset/snli), [MultiNLI](/dataset/multinli), [SciTail](/dataset/scitail), among others. You can get hands-on practice on the SNLI task by following this [d2l.ai chapter](https://d2l.ai/chapter_natural-language-processing-applications/natural-language-inference-and-dataset.html).\r\n\r\nFurther readings:\r\n\r\n- [Recent Advances in Natural Language Inference: A Survey of Benchmarks, Resources, and Approaches](https://arxiv.org/abs/1904.01172)","categories":[{"name":"Natural Language Processing","url":"/area/natural-language-processing"},{"name":"Reasoning","url":"/area/reasoning"}],"source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","slug_source":"archive_url"},"counts":{"papers_tagged":1961,"papers_with_code":821,"benchmarks":33,"benchmark_tables_in_archive":37,"benchmark_tables_shown":37,"benchmark_tables_withheld_as_spam":0,"benchmark_definition":"a leaderboard table with at least one row; benchmark_tables_shown also counts the zero-row tables; benchmark_tables_in_archive adds the tables withheld as spam","datasets":81,"subtasks":3,"parent_tasks":0},"benchmarks":[{"leaderboard":"/sota/natural-language-inference-on-snli","slug":"natural-language-inference-on-snli","dataset":"SNLI","dataset_url":"/dataset/snli","rows_in_archive":98,"metrics":["% Test Accuracy","% Train Accuracy","Parameters","Dev Accuracy","% Dev Accuracy","Accuracy"],"first_row_in_archive_order":{"model":"UnitedSynT5 (3B)","paper_title":"First Train to Generate, then Generate to Train: UnitedSynT5 for Few-Shot NLI","paper_url":"/paper/first-train-to-generate-then-generate-to","paper_date":"2024-12-12","arxiv_id":"2412.09263","code_links":[],"syntology":null}},{"leaderboard":"/sota/natural-language-inference-on-rte","slug":"natural-language-inference-on-rte","dataset":"RTE","dataset_url":"/dataset/rte","rows_in_archive":90,"metrics":["Accuracy"],"first_row_in_archive_order":{"model":"Vega v2 6B (KD-based prompt transfer)","paper_title":"Toward Efficient Language Model Pretraining and Downstream Adaptation via Self-Evolution: A Case Study on SuperGLUE","paper_url":"/paper/toward-efficient-language-model-pretraining","paper_date":"2022-12-04","arxiv_id":"2212.01853","code_links":[],"syntology":null}},{"leaderboard":"/sota/natural-language-inference-on-multinli","slug":"natural-language-inference-on-multinli","dataset":"MultiNLI","dataset_url":"/dataset/multinli","rows_in_archive":67,"metrics":["Matched","Mismatched","Accuracy","Dev Matched","Dev Mismatched"],"first_row_in_archive_order":{"model":"Turing NLR v5 XXL 5.4B (fine-tuned)","paper_title":null,"paper_url":null,"paper_date":"","arxiv_id":null,"code_links":[],"syntology":null}},{"leaderboard":"/sota/natural-language-inference-on-qnli","slug":"natural-language-inference-on-qnli","dataset":"QNLI","dataset_url":"/dataset/qnli","rows_in_archive":43,"metrics":["Accuracy"],"first_row_in_archive_order":{"model":"ALBERT","paper_title":"ALBERT: A Lite BERT for Self-supervised Learning of Language Representations","paper_url":"/paper/albert-a-lite-bert-for-self-supervised","paper_date":"2019-09-26","arxiv_id":"1909.11942","code_links":[{"title":"huggingface/transformers","url":"https://github.com/huggingface/transformers"},{"title":"tensorflow/models","url":"https://github.com/tensorflow/models/tree/master/official/nlp/albert"},{"title":"PaddlePaddle/PaddleNLP","url":"https://github.com/PaddlePaddle/PaddleNLP/blob/develop/paddlenlp/transformers/albert/modeling.py"},{"title":"CLUEbenchmark/CLUE","url":"https://github.com/CLUEbenchmark/CLUE"},{"title":"brightmart/albert_zh","url":"https://github.com/brightmart/albert_zh"},{"title":"google-research/ALBERT","url":"https://github.com/google-research/ALBERT"},{"title":"Tencent/TurboTransformers","url":"https://github.com/Tencent/TurboTransformers"},{"title":"kpe/bert-for-tf2","url":"https://github.com/kpe/bert-for-tf2"},{"title":"lonePatient/albert_pytorch","url":"https://github.com/lonePatient/albert_pytorch"},{"title":"plkmo/BERT-Relation-Extraction","url":"https://github.com/plkmo/BERT-Relation-Extraction"},{"title":"facebookresearch/anli","url":"https://github.com/facebookresearch/anli"},{"title":"lucidrains/routing-transformer","url":"https://github.com/lucidrains/routing-transformer"},{"title":"lucidrains/sinkhorn-transformer","url":"https://github.com/lucidrains/sinkhorn-transformer"},{"title":"graykode/ALBERT-Pytorch","url":"https://github.com/graykode/ALBERT-Pytorch"},{"title":"kamalkraj/ALBERT-TF2.0","url":"https://github.com/kamalkraj/ALBERT-TF2.0"},{"title":"lucidrains/compressive-transformer-pytorch","url":"https://github.com/lucidrains/compressive-transformer-pytorch"},{"title":"epfml/collaborative-attention","url":"https://github.com/epfml/collaborative-attention"},{"title":"Sanyuan-Chen/RecAdam","url":"https://github.com/Sanyuan-Chen/RecAdam"},{"title":"mindspore-lab/mindnlp","url":"https://github.com/mindspore-lab/mindnlp/blob/master/mindnlp/models/albert/albert.py"},{"title":"benywon/ReCO","url":"https://github.com/benywon/ReCO"},{"title":"KnightZhang625/BERT_TF","url":"https://github.com/KnightZhang625/BERT_TF"},{"title":"AdamStein97/Semi-Supervised-BERT-NER","url":"https://github.com/AdamStein97/Semi-Supervised-BERT-NER"},{"title":"mandubian/codenets","url":"https://github.com/mandubian/codenets"},{"title":"codegram/calbert","url":"https://github.com/codegram/calbert"},{"title":"yahah100/text_summarization","url":"https://github.com/yahah100/text_summarization"},{"title":"Soikonomou/albert_final_infer12","url":"https://github.com/Soikonomou/albert_final_infer12"},{"title":"xinyooo/ALBERT4Rec","url":"https://github.com/xinyooo/ALBERT4Rec"},{"title":"rajatgermany/qa-nlp","url":"https://github.com/rajatgermany/qa-nlp"},{"title":"Soikonomou/bert_new","url":"https://github.com/Soikonomou/bert_new"},{"title":"Soikonomou/bert_new_new","url":"https://github.com/Soikonomou/bert_new_new"},{"title":"Soikonomou/albert_final","url":"https://github.com/Soikonomou/albert_final"},{"title":"Soikonomou/albert_final_infer8","url":"https://github.com/Soikonomou/albert_final_infer8"},{"title":"benywon/Chinese-GPT-2","url":"https://github.com/benywon/Chinese-GPT-2"},{"title":"jpablou/Matching-The-Blanks-Ths","url":"https://github.com/jpablou/Matching-The-Blanks-Ths"},{"title":"vvvm23/albert","url":"https://github.com/vvvm23/albert"},{"title":"common-english/bert-all","url":"https://github.com/common-english/bert-all"},{"title":"AhmedYounes94/Semi-supervised-BERT-NER","url":"https://github.com/AhmedYounes94/Semi-supervised-BERT-NER"},{"title":"appcoreopc/berty","url":"https://github.com/appcoreopc/berty"},{"title":"lyqcom/albert","url":"https://github.com/lyqcom/albert"},{"title":"mtzcorporations/nlp_teamjodka","url":"https://github.com/mtzcorporations/nlp_teamjodka"},{"title":"cui0523/Code6","url":"https://github.com/cui0523/Code6/tree/main/albert"},{"title":"cypressd1999/FYP_2021","url":"https://github.com/cypressd1999/FYP_2021"},{"title":"pwc-1/Paper-8","url":"https://github.com/pwc-1/Paper-8/tree/main/albert"},{"title":"Mind23-2/MindCode-131","url":"https://github.com/Mind23-2/MindCode-131"},{"title":"2023-MindSpore-1/ms-code-219","url":"https://github.com/2023-MindSpore-1/ms-code-219/tree/main/albert"},{"title":"hieudepchai/BERT_IE","url":"https://github.com/hieudepchai/BERT_IE"},{"title":"DataScienceNigeria/ALBERT-for-Natural-Language-Processing","url":"https://github.com/DataScienceNigeria/ALBERT-for-Natural-Language-Processing"},{"title":"MindSpore-paper-code-3/code9","url":"https://github.com/MindSpore-paper-code-3/code9/tree/main/albert"}],"syntology":{"n":126,"n_ran":81,"n_constructed":17,"n_ran_checked":56,"n_instrument":25,"n_unverified":45,"n_honours":4,"n_violates":0,"n_no_contract":52,"n_pointer_only":28,"phrase":"81 ran (of which 17 constructed an object rather than computing a result; 56 with no instrument failure: 4 honoured, 0 violated, 52 with no contract checked; 25 where Syntology's instrument failed) · 45 unverified","sample_list":"/paper/albert-a-lite-bert-for-self-supervised#ran","syntology_url":"https://syntology.ai/paper/1909.11942","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1909.11942"}},"official":{"repos":["google-research/ALBERT"],"state":"official (archive's flag): 6 ran","n_ran":6,"n_constructed":0,"n_ran_no_instrument_failure":4,"n_unverified":0,"ran_from_kinds":["listed","official"]}}}},{"leaderboard":"/sota/natural-language-inference-on-anli-test","slug":"natural-language-inference-on-anli-test","dataset":"ANLI test","dataset_url":"/dataset/anli","rows_in_archive":25,"metrics":["A1","A2","A3"],"first_row_in_archive_order":{"model":"T5-3B (explanation prompting)","paper_title":"Prompting for explanations improves Adversarial NLI. Is this true? {Yes} it is {true} because {it weakens superficial cues}","paper_url":"/paper/prompting-for-explanations-improves","paper_date":"2023-05-01","arxiv_id":null,"code_links":[],"syntology":null}},{"leaderboard":"/sota/natural-language-inference-on-wnli","slug":"natural-language-inference-on-wnli","dataset":"WNLI","dataset_url":"/dataset/wnli","rows_in_archive":23,"metrics":["Accuracy"],"first_row_in_archive_order":{"model":"Turing NLR v5 XXL 5.4B (fine-tuned)","paper_title":null,"paper_url":null,"paper_date":"","arxiv_id":null,"code_links":[],"syntology":null}},{"leaderboard":"/sota/natural-language-inference-on-lidirus","slug":"natural-language-inference-on-lidirus","dataset":"LiDiRus","dataset_url":"/dataset/lidirus","rows_in_archive":22,"metrics":["MCC"],"first_row_in_archive_order":{"model":"Human Benchmark","paper_title":"RussianSuperGLUE: A Russian Language Understanding Evaluation Benchmark","paper_url":"/paper/russiansuperglue-a-russian-language","paper_date":"2020-10-29","arxiv_id":"2010.15925","code_links":[{"title":"RussianNLP/RussianSuperGLUE","url":"https://github.com/RussianNLP/RussianSuperGLUE"},{"title":"RussianNLP/MOROCCO","url":"https://github.com/RussianNLP/MOROCCO"}],"syntology":{"n":1,"n_ran":1,"n_constructed":0,"n_ran_checked":1,"n_instrument":0,"n_unverified":0,"n_honours":1,"n_violates":0,"n_no_contract":0,"n_pointer_only":0,"phrase":"1 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 1 honoured, 0 violated, 0 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified","sample_list":"/paper/russiansuperglue-a-russian-language#ran","syntology_url":"https://syntology.ai/paper/2010.15925","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2010.15925"}},"official":{"repos":["RussianNLP/RussianSuperGLUE"],"state":"official (archive's flag): 1 ran","n_ran":1,"n_constructed":0,"n_ran_no_instrument_failure":1,"n_unverified":0,"ran_from_kinds":["official"]}}}},{"leaderboard":"/sota/natural-language-inference-on-rcb","slug":"natural-language-inference-on-rcb","dataset":"RCB","dataset_url":"/dataset/rcb","rows_in_archive":22,"metrics":["Average F1","Accuracy"],"first_row_in_archive_order":{"model":"Human Benchmark","paper_title":"RussianSuperGLUE: A Russian Language Understanding Evaluation Benchmark","paper_url":"/paper/russiansuperglue-a-russian-language","paper_date":"2020-10-29","arxiv_id":"2010.15925","code_links":[{"title":"RussianNLP/RussianSuperGLUE","url":"https://github.com/RussianNLP/RussianSuperGLUE"},{"title":"RussianNLP/MOROCCO","url":"https://github.com/RussianNLP/MOROCCO"}],"syntology":{"n":1,"n_ran":1,"n_constructed":0,"n_ran_checked":1,"n_instrument":0,"n_unverified":0,"n_honours":1,"n_violates":0,"n_no_contract":0,"n_pointer_only":0,"phrase":"1 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 1 honoured, 0 violated, 0 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified","sample_list":"/paper/russiansuperglue-a-russian-language#ran","syntology_url":"https://syntology.ai/paper/2010.15925","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2010.15925"}},"official":{"repos":["RussianNLP/RussianSuperGLUE"],"state":"official (archive's flag): 1 ran","n_ran":1,"n_constructed":0,"n_ran_no_instrument_failure":1,"n_unverified":0,"ran_from_kinds":["official"]}}}},{"leaderboard":"/sota/natural-language-inference-on-terra","slug":"natural-language-inference-on-terra","dataset":"TERRa","dataset_url":"/dataset/terra","rows_in_archive":22,"metrics":["Accuracy"],"first_row_in_archive_order":{"model":"Human Benchmark","paper_title":"RussianSuperGLUE: A Russian Language Understanding Evaluation Benchmark","paper_url":"/paper/russiansuperglue-a-russian-language","paper_date":"2020-10-29","arxiv_id":"2010.15925","code_links":[{"title":"RussianNLP/RussianSuperGLUE","url":"https://github.com/RussianNLP/RussianSuperGLUE"},{"title":"RussianNLP/MOROCCO","url":"https://github.com/RussianNLP/MOROCCO"}],"syntology":{"n":1,"n_ran":1,"n_constructed":0,"n_ran_checked":1,"n_instrument":0,"n_unverified":0,"n_honours":1,"n_violates":0,"n_no_contract":0,"n_pointer_only":0,"phrase":"1 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 1 honoured, 0 violated, 0 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified","sample_list":"/paper/russiansuperglue-a-russian-language#ran","syntology_url":"https://syntology.ai/paper/2010.15925","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2010.15925"}},"official":{"repos":["RussianNLP/RussianSuperGLUE"],"state":"official (archive's flag): 1 ran","n_ran":1,"n_constructed":0,"n_ran_no_instrument_failure":1,"n_unverified":0,"ran_from_kinds":["official"]}}}},{"leaderboard":"/sota/natural-language-inference-on-commitmentbank","slug":"natural-language-inference-on-commitmentbank","dataset":"CommitmentBank","dataset_url":"/dataset/commitmentbank","rows_in_archive":20,"metrics":["Accuracy","F1"],"first_row_in_archive_order":{"model":"PaLM 540B (finetuned)","paper_title":"PaLM: Scaling Language Modeling with Pathways","paper_url":"/paper/palm-scaling-language-modeling-with-pathways-1","paper_date":"2022-04-05","arxiv_id":"2204.02311","code_links":[{"title":"lucidrains/CoCa-pytorch","url":"https://github.com/lucidrains/CoCa-pytorch"},{"title":"lucidrains/PaLM-pytorch","url":"https://github.com/lucidrains/PaLM-pytorch"},{"title":"google/paxml","url":"https://github.com/google/paxml"},{"title":"foundation-model-stack/fms-fsdp","url":"https://github.com/foundation-model-stack/fms-fsdp"},{"title":"lucidrains/PaLM-jax","url":"https://github.com/lucidrains/PaLM-jax"},{"title":"chrisociepa/allamo","url":"https://github.com/chrisociepa/allamo"},{"title":"conceptofmind/PaLM-flax","url":"https://github.com/conceptofmind/PaLM-flax"}],"syntology":{"n":37,"n_ran":32,"n_constructed":16,"n_ran_checked":24,"n_instrument":8,"n_unverified":5,"n_honours":2,"n_violates":1,"n_no_contract":21,"n_pointer_only":0,"phrase":"32 ran (of which 16 constructed an object rather than computing a result; 24 with no instrument failure: 2 honoured, 1 violated, 21 with no contract checked; 8 where Syntology's instrument failed) · 5 unverified","sample_list":"/paper/palm-scaling-language-modeling-with-pathways-1#ran","syntology_url":"https://syntology.ai/paper/2204.02311","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2204.02311"}},"official":null}}},{"leaderboard":"/sota/natural-language-inference-on-scitail","slug":"natural-language-inference-on-scitail","dataset":"SciTail","dataset_url":"/dataset/scitail","rows_in_archive":13,"metrics":["Accuracy","Dev Accuracy","% Dev Accuracy","% Test Accuracy"],"first_row_in_archive_order":{"model":"CA-MTL","paper_title":"Conditionally Adaptive Multi-Task Learning: Improving Transfer Learning in NLP Using Fewer Parameters & Less Data","paper_url":"/paper/conditionally-adaptive-multi-task-learning","paper_date":"2020-09-19","arxiv_id":"2009.09139","code_links":[{"title":"CAMTL/CA-MTL","url":"https://github.com/CAMTL/CA-MTL"}],"syntology":null}},{"leaderboard":"/sota/natural-language-inference-on-farstail","slug":"natural-language-inference-on-farstail","dataset":"FarsTail","dataset_url":"/dataset/farstail","rows_in_archive":10,"metrics":["% Test Accuracy"],"first_row_in_archive_order":{"model":"mBERT","paper_title":"FarsTail: A Persian Natural Language Inference Dataset","paper_url":"/paper/farstail-a-persian-natural-language-inference","paper_date":"2020-09-18","arxiv_id":"2009.08820","code_links":[{"title":"dml-qom/FarsTail","url":"https://github.com/dml-qom/FarsTail"}],"syntology":null}},{"leaderboard":"/sota/natural-language-inference-on-multinli-dev","slug":"natural-language-inference-on-multinli-dev","dataset":"MultiNLI Dev","dataset_url":"/dataset/multinli","rows_in_archive":10,"metrics":["Matched","Mismatched"],"first_row_in_archive_order":{"model":"TinyBERT-6 67M","paper_title":"TinyBERT: Distilling BERT for Natural Language Understanding","paper_url":"/paper/190910351","paper_date":"2019-09-23","arxiv_id":"1909.10351","code_links":[{"title":"PaddlePaddle/PaddleNLP","url":"https://github.com/PaddlePaddle/PaddleNLP/tree/develop/paddlenlp/transformers/tinybert"},{"title":"huawei-noah/Pretrained-Language-Model","url":"https://github.com/huawei-noah/Pretrained-Language-Model/tree/master/TinyBERT"},{"title":"mindspore-ai/models","url":"https://github.com/mindspore-ai/models/tree/master/official/nlp/tinybert"},{"title":"mkavim/finetune_bert","url":"https://github.com/mkavim/finetune_bert"},{"title":"MindCode-4/code-5","url":"https://github.com/MindCode-4/code-5/tree/main/tinybert"},{"title":"pwc-1/Paper-9","url":"https://github.com/pwc-1/Paper-9/tree/main/1/tinybert"},{"title":"xiaolilaoli/tiny_bert_ms","url":"https://github.com/xiaolilaoli/tiny_bert_ms"},{"title":"pwc-1/Paper-10","url":"https://github.com/pwc-1/Paper-10/tree/main/tinybert"},{"title":"graison-thomas/TinyFinBERT","url":"https://github.com/graison-thomas/TinyFinBERT"},{"title":"2023-MindSpore-1/ms-code-166","url":"https://github.com/2023-MindSpore-1/ms-code-166"}],"syntology":{"n":4,"n_ran":0,"n_constructed":0,"n_ran_checked":0,"n_instrument":0,"n_unverified":4,"n_honours":0,"n_violates":0,"n_no_contract":0,"n_pointer_only":4,"phrase":"0 ran · 4 unverified","sample_list":"/paper/190910351#ran","syntology_url":"https://syntology.ai/paper/1909.10351","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1909.10351"}},"official":{"repos":["huawei-noah/Pretrained-Language-Model"],"state":"official: harvested, nothing ran","n_ran":0,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":4,"ran_from_kinds":[]}}}},{"leaderboard":"/sota/natural-language-inference-on-mednli","slug":"natural-language-inference-on-mednli","dataset":"MedNLI","dataset_url":"/dataset/mednli","rows_in_archive":7,"metrics":["Accuracy","Params (M)"],"first_row_in_archive_order":{"model":"ClinicalMosaic","paper_title":"Patient Trajectory Prediction: Integrating Clinical Notes with Transformers","paper_url":"/paper/patient-trajectory-prediction-integrating","paper_date":"2025-02-25","arxiv_id":"2502.18009","code_links":[{"title":"MostHumble/PatientTrajectoryForecasting","url":"https://github.com/MostHumble/PatientTrajectoryForecasting"}],"syntology":null}},{"leaderboard":"/sota/natural-language-inference-on-xnli-french","slug":"natural-language-inference-on-xnli-french","dataset":"XNLI French","dataset_url":"/dataset/xnli","rows_in_archive":6,"metrics":["Accuracy"],"first_row_in_archive_order":{"model":"CamemBERT (large)","paper_title":"CamemBERT: a Tasty French Language Model","paper_url":"/paper/camembert-a-tasty-french-language-model","paper_date":"2019-11-10","arxiv_id":"1911.03894","code_links":[{"title":"huggingface/transformers","url":"https://github.com/huggingface/transformers"},{"title":"Karthik-Bhaskar/Context-Based-Question-Answering","url":"https://github.com/Karthik-Bhaskar/Context-Based-Question-Answering"},{"title":"anaishoareau/french_preprocessing","url":"https://github.com/anaishoareau/french_preprocessing"},{"title":"bourrel/French-News-Clustering","url":"https://github.com/bourrel/French-News-Clustering"},{"title":"2024-MindSpore-1/Code2","url":"https://github.com/2024-MindSpore-1/Code2/tree/main/model-1/camembert"},{"title":"hbaflast/bert-sentiment-analysis-pytorch","url":"https://github.com/hbaflast/bert-sentiment-analysis-pytorch"},{"title":"hbaflast/bert-sentiment-analysis-tensorflow","url":"https://github.com/hbaflast/bert-sentiment-analysis-tensorflow"},{"title":"pwc-1/Paper-8","url":"https://github.com/pwc-1/Paper-8/tree/main/camembert"}],"syntology":null}},{"leaderboard":"/sota/natural-language-inference-on-e-snli","slug":"natural-language-inference-on-e-snli","dataset":"e-SNLI","dataset_url":"/dataset/e-snli","rows_in_archive":3,"metrics":["BLEU","Accuracy"],"first_row_in_archive_order":{"model":"ExplainThenPredictAttention (e-InferSent Bi-LSTM + Attention)","paper_title":"e-SNLI: Natural Language Inference with Natural Language Explanations","paper_url":"/paper/e-snli-natural-language-inference-with","paper_date":"2018-12-04","arxiv_id":"1812.01193","code_links":[{"title":"OanaMariaCamburu/e-SNLI","url":"https://github.com/OanaMariaCamburu/e-SNLI"},{"title":"qtli/eib","url":"https://github.com/qtli/eib"}],"syntology":null}},{"leaderboard":"/sota/natural-language-inference-on-v-snli","slug":"natural-language-inference-on-v-snli","dataset":"V-SNLI","dataset_url":null,"rows_in_archive":3,"metrics":["Accuracy"],"first_row_in_archive_order":{"model":"MMBT","paper_title":"Supervised Multimodal Bitransformers for Classifying Images and Text","paper_url":"/paper/supervised-multimodal-bitransformers-for","paper_date":"2019-09-06","arxiv_id":"1909.02950","code_links":[{"title":"huggingface/transformers","url":"https://github.com/huggingface/transformers"},{"title":"ThilinaRajapakse/simpletransformers","url":"https://github.com/ThilinaRajapakse/simpletransformers"},{"title":"facebookresearch/mmbt","url":"https://github.com/facebookresearch/mmbt"},{"title":"IsaacRodgz/multimodal-transformers-movies","url":"https://github.com/IsaacRodgz/multimodal-transformers-movies"},{"title":"IsaacRodgz/mmbt_experiments","url":"https://github.com/IsaacRodgz/mmbt_experiments"},{"title":"adriangrepo/mmbt_lightning","url":"https://github.com/adriangrepo/mmbt_lightning"}],"syntology":null}},{"leaderboard":"/sota/natural-language-inference-on-xnli-chinese","slug":"natural-language-inference-on-xnli-chinese","dataset":"XNLI Chinese Dev","dataset_url":"/dataset/xnli","rows_in_archive":3,"metrics":["Accuracy"],"first_row_in_archive_order":{"model":"ERNIE 2.0 Large","paper_title":"ERNIE 2.0: A Continual Pre-training Framework for Language Understanding","paper_url":"/paper/ernie-20-a-continual-pre-training-framework","paper_date":"2019-07-29","arxiv_id":"1907.12412","code_links":[{"title":"PaddlePaddle/PaddleNLP","url":"https://github.com/PaddlePaddle/PaddleNLP/tree/develop/model_zoo/ernie-1.0"},{"title":"PaddlePaddle/ERNIE","url":"https://github.com/PaddlePaddle/ERNIE"},{"title":"DataScienceNigeria/ERNIE-2.0-from-Baidu-Inc.","url":"https://github.com/DataScienceNigeria/ERNIE-2.0-from-Baidu-Inc."}],"syntology":{"n":1,"n_ran":0,"n_constructed":0,"n_ran_checked":0,"n_instrument":0,"n_unverified":1,"n_honours":0,"n_violates":0,"n_no_contract":0,"n_pointer_only":1,"phrase":"0 ran · 1 unverified","sample_list":"/paper/ernie-20-a-continual-pre-training-framework#ran","syntology_url":"https://syntology.ai/paper/1907.12412","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1907.12412"}},"official":{"repos":["PaddlePaddle/ERNIE"],"state":"official: harvested, nothing ran","n_ran":0,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":1,"ran_from_kinds":[]}}}},{"leaderboard":"/sota/natural-language-inference-on-xnli-chinese-1","slug":"natural-language-inference-on-xnli-chinese-1","dataset":"XNLI Chinese","dataset_url":"/dataset/xnli","rows_in_archive":3,"metrics":["Accuracy"],"first_row_in_archive_order":{"model":"ERNIE 2.0 Large","paper_title":"ERNIE 2.0: A Continual Pre-training Framework for Language Understanding","paper_url":"/paper/ernie-20-a-continual-pre-training-framework","paper_date":"2019-07-29","arxiv_id":"1907.12412","code_links":[{"title":"PaddlePaddle/PaddleNLP","url":"https://github.com/PaddlePaddle/PaddleNLP/tree/develop/model_zoo/ernie-1.0"},{"title":"PaddlePaddle/ERNIE","url":"https://github.com/PaddlePaddle/ERNIE"},{"title":"DataScienceNigeria/ERNIE-2.0-from-Baidu-Inc.","url":"https://github.com/DataScienceNigeria/ERNIE-2.0-from-Baidu-Inc."}],"syntology":{"n":1,"n_ran":0,"n_constructed":0,"n_ran_checked":0,"n_instrument":0,"n_unverified":1,"n_honours":0,"n_violates":0,"n_no_contract":0,"n_pointer_only":1,"phrase":"0 ran · 1 unverified","sample_list":"/paper/ernie-20-a-continual-pre-training-framework#ran","syntology_url":"https://syntology.ai/paper/1907.12412","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1907.12412"}},"official":{"repos":["PaddlePaddle/ERNIE"],"state":"official: harvested, nothing ran","n_ran":0,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":1,"ran_from_kinds":[]}}}},{"leaderboard":"/sota/natural-language-inference-on-jampatoisnli","slug":"natural-language-inference-on-jampatoisnli","dataset":"JamPatoisNLI","dataset_url":"/dataset/jampatoisnli","rows_in_archive":2,"metrics":["Accuracy"],"first_row_in_archive_order":{"model":"roberta-unfrozen","paper_title":"JamPatoisNLI: A Jamaican Patois Natural Language Inference Dataset","paper_url":"/paper/jampatoisnli-a-jamaican-patois-natural","paper_date":"2022-12-07","arxiv_id":"2212.03419","code_links":[],"syntology":null}},{"leaderboard":"/sota/natural-language-inference-on-ax","slug":"natural-language-inference-on-ax","dataset":"AX","dataset_url":null,"rows_in_archive":1,"metrics":["Accuracy"],"first_row_in_archive_order":{"model":"T5","paper_title":"SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimization","paper_url":"/paper/smart-robust-and-efficient-fine-tuning-for","paper_date":"2019-11-08","arxiv_id":"1911.03437","code_links":[{"title":"namisan/mt-dnn","url":"https://github.com/namisan/mt-dnn"},{"title":"microsoft/MT-DNN","url":"https://github.com/microsoft/MT-DNN"},{"title":"archinetai/smart-pytorch","url":"https://github.com/archinetai/smart-pytorch"},{"title":"archinetai/vat-pytorch","url":"https://github.com/archinetai/vat-pytorch"},{"title":"cliang1453/camero","url":"https://github.com/cliang1453/camero"},{"title":"chunhuililili/mt_dnn","url":"https://github.com/chunhuililili/mt_dnn"}],"syntology":{"n":8,"n_ran":8,"n_constructed":0,"n_ran_checked":5,"n_instrument":3,"n_unverified":0,"n_honours":2,"n_violates":1,"n_no_contract":2,"n_pointer_only":1,"phrase":"8 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 2 honoured, 1 violated, 2 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified","sample_list":"/paper/smart-robust-and-efficient-fine-tuning-for#ran","syntology_url":"https://syntology.ai/paper/1911.03437","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1911.03437"}},"official":{"repos":["namisan/mt-dnn"],"state":"official: no sample here; runs from other or unrecorded repositories","n_ran":0,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":0,"ran_from_kinds":["listed","unlocated"]}}}},{"leaderboard":"/sota/natural-language-inference-on-bionli","slug":"natural-language-inference-on-bionli","dataset":"BioNLI","dataset_url":"/dataset/bionli","rows_in_archive":1,"metrics":["Macro F1"],"first_row_in_archive_order":{"model":"BioLinkBert","paper_title":"BioNLI: Generating a Biomedical NLI Dataset Using Lexico-semantic Constraints for Adversarial Examples","paper_url":"/paper/bionli-generating-a-biomedical-nli-dataset","paper_date":"2022-10-26","arxiv_id":"2210.14814","code_links":[{"title":"StonyBrookNLP/BioNLI","url":"https://github.com/StonyBrookNLP/BioNLI"}],"syntology":null}},{"leaderboard":"/sota/natural-language-inference-on-hans","slug":"natural-language-inference-on-hans","dataset":"HANS","dataset_url":"/dataset/hans","rows_in_archive":1,"metrics":["1:1 Accuracy"],"first_row_in_archive_order":{"model":"Roberta-large","paper_title":"Generating Data to Mitigate Spurious Correlations in Natural Language Inference Datasets","paper_url":"/paper/generating-data-to-mitigate-spurious-1","paper_date":"2022-03-24","arxiv_id":"2203.12942","code_links":[{"title":"jimmycode/gen-debiased-nli","url":"https://github.com/jimmycode/gen-debiased-nli"}],"syntology":null}},{"leaderboard":"/sota/natural-language-inference-on-kuake-qqr","slug":"natural-language-inference-on-kuake-qqr","dataset":"KUAKE-QQR","dataset_url":"/dataset/kuake-qqr","rows_in_archive":1,"metrics":["Accuracy"],"first_row_in_archive_order":{"model":"BERT-base","paper_title":"CBLUE: A Chinese Biomedical Language Understanding Evaluation Benchmark","paper_url":"/paper/cblue-a-chinese-biomedical-language","paper_date":"2021-06-15","arxiv_id":"2106.08087","code_links":[{"title":"cbluebenchmark/cblue","url":"https://github.com/cbluebenchmark/cblue"},{"title":"freedomintelligence/sdak","url":"https://github.com/freedomintelligence/sdak"}],"syntology":{"n":16,"n_ran":11,"n_constructed":0,"n_ran_checked":8,"n_instrument":3,"n_unverified":5,"n_honours":1,"n_violates":0,"n_no_contract":7,"n_pointer_only":2,"phrase":"11 ran (of which 0 constructed an object rather than computing a result; 8 with no instrument failure: 1 honoured, 0 violated, 7 with no contract checked; 3 where Syntology's instrument failed) · 5 unverified","sample_list":"/paper/cblue-a-chinese-biomedical-language#ran","syntology_url":"https://syntology.ai/paper/2106.08087","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2106.08087"}},"official":{"repos":["cbluebenchmark/cblue"],"state":"official (archive's flag): 11 ran","n_ran":11,"n_constructed":0,"n_ran_no_instrument_failure":8,"n_unverified":5,"ran_from_kinds":["official"]}}}},{"leaderboard":"/sota/natural-language-inference-on-kuake-qtr","slug":"natural-language-inference-on-kuake-qtr","dataset":"KUAKE-QTR","dataset_url":"/dataset/kuake-qtr","rows_in_archive":1,"metrics":["Accuracy"],"first_row_in_archive_order":{"model":"MacBERT-large","paper_title":"CBLUE: A Chinese Biomedical Language Understanding Evaluation Benchmark","paper_url":"/paper/cblue-a-chinese-biomedical-language","paper_date":"2021-06-15","arxiv_id":"2106.08087","code_links":[{"title":"cbluebenchmark/cblue","url":"https://github.com/cbluebenchmark/cblue"},{"title":"freedomintelligence/sdak","url":"https://github.com/freedomintelligence/sdak"}],"syntology":{"n":16,"n_ran":11,"n_constructed":0,"n_ran_checked":8,"n_instrument":3,"n_unverified":5,"n_honours":1,"n_violates":0,"n_no_contract":7,"n_pointer_only":2,"phrase":"11 ran (of which 0 constructed an object rather than computing a result; 8 with no instrument failure: 1 honoured, 0 violated, 7 with no contract checked; 3 where Syntology's instrument failed) · 5 unverified","sample_list":"/paper/cblue-a-chinese-biomedical-language#ran","syntology_url":"https://syntology.ai/paper/2106.08087","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2106.08087"}},"official":{"repos":["cbluebenchmark/cblue"],"state":"official (archive's flag): 11 ran","n_ran":11,"n_constructed":0,"n_ran_no_instrument_failure":8,"n_unverified":5,"ran_from_kinds":["official"]}}}},{"leaderboard":"/sota/natural-language-inference-on-med","slug":"natural-language-inference-on-med","dataset":"MED","dataset_url":"/dataset/med","rows_in_archive":1,"metrics":["1:1 Accuracy"],"first_row_in_archive_order":{"model":"NeuralLog","paper_title":"NeuralLog: Natural Language Inference with Joint Neural and Logical Reasoning","paper_url":"/paper/neurallog-natural-language-inference-with","paper_date":"2021-05-29","arxiv_id":"2105.14167","code_links":[{"title":"eric11eca/NeuralLog","url":"https://github.com/eric11eca/NeuralLog"}],"syntology":{"n":6,"n_ran":3,"n_constructed":0,"n_ran_checked":3,"n_instrument":0,"n_unverified":3,"n_honours":0,"n_violates":0,"n_no_contract":3,"n_pointer_only":0,"phrase":"3 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 0 violated, 3 with no contract checked; 0 where Syntology's instrument failed) · 3 unverified","sample_list":"/paper/neurallog-natural-language-inference-with#ran","syntology_url":"https://syntology.ai/paper/2105.14167","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2105.14167"}},"official":{"repos":["eric11eca/NeuralLog"],"state":"official (archive's flag): 3 ran","n_ran":3,"n_constructed":0,"n_ran_no_instrument_failure":3,"n_unverified":3,"ran_from_kinds":["official"]}}}},{"leaderboard":"/sota/natural-language-inference-on-mnli-snli-anli","slug":"natural-language-inference-on-mnli-snli-anli","dataset":"MNLI + SNLI + ANLI + FEVER","dataset_url":null,"rows_in_archive":1,"metrics":["% Dev Accuracy","% Test Accuracy"],"first_row_in_archive_order":{"model":"SMARTRoBERTa-LARGE","paper_title":"SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimization","paper_url":"/paper/smart-robust-and-efficient-fine-tuning-for","paper_date":"2019-11-08","arxiv_id":"1911.03437","code_links":[{"title":"namisan/mt-dnn","url":"https://github.com/namisan/mt-dnn"},{"title":"microsoft/MT-DNN","url":"https://github.com/microsoft/MT-DNN"},{"title":"archinetai/smart-pytorch","url":"https://github.com/archinetai/smart-pytorch"},{"title":"archinetai/vat-pytorch","url":"https://github.com/archinetai/vat-pytorch"},{"title":"cliang1453/camero","url":"https://github.com/cliang1453/camero"},{"title":"chunhuililili/mt_dnn","url":"https://github.com/chunhuililili/mt_dnn"}],"syntology":{"n":8,"n_ran":8,"n_constructed":0,"n_ran_checked":5,"n_instrument":3,"n_unverified":0,"n_honours":2,"n_violates":1,"n_no_contract":2,"n_pointer_only":1,"phrase":"8 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 2 honoured, 1 violated, 2 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified","sample_list":"/paper/smart-robust-and-efficient-fine-tuning-for#ran","syntology_url":"https://syntology.ai/paper/1911.03437","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1911.03437"}},"official":{"repos":["namisan/mt-dnn"],"state":"official: no sample here; runs from other or unrecorded repositories","n_ran":0,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":0,"ran_from_kinds":["listed","unlocated"]}}}},{"leaderboard":"/sota/natural-language-inference-on-mrpc","slug":"natural-language-inference-on-mrpc","dataset":"MRPC","dataset_url":"/dataset/mrpc","rows_in_archive":1,"metrics":["Acc"],"first_row_in_archive_order":{"model":"DeBERTaV3large","paper_title":"DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing","paper_url":"/paper/debertav3-improving-deberta-using-electra","paper_date":"2021-11-18","arxiv_id":"2111.09543","code_links":[{"title":"microsoft/DeBERTa","url":"https://github.com/microsoft/DeBERTa"},{"title":"dashenzi721/hra","url":"https://github.com/dashenzi721/hra"},{"title":"stareru/csqa_debertav3","url":"https://github.com/stareru/csqa_debertav3"}],"syntology":{"n":7,"n_ran":6,"n_constructed":0,"n_ran_checked":6,"n_instrument":0,"n_unverified":1,"n_honours":0,"n_violates":0,"n_no_contract":6,"n_pointer_only":0,"phrase":"6 ran (of which 0 constructed an object rather than computing a result; 6 with no instrument failure: 0 honoured, 0 violated, 6 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified","sample_list":"/paper/debertav3-improving-deberta-using-electra#ran","syntology_url":"https://syntology.ai/paper/2111.09543","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2111.09543"}},"official":{"repos":["microsoft/DeBERTa"],"state":"official (archive's flag): 6 ran","n_ran":6,"n_constructed":0,"n_ran_no_instrument_failure":6,"n_unverified":1,"ran_from_kinds":["official"]}}}},{"leaderboard":"/sota/natural-language-inference-on-probability","slug":"natural-language-inference-on-probability","dataset":"Probability words NLI","dataset_url":"/dataset/probability-words-nli","rows_in_archive":1,"metrics":["1:1 Accuracy"],"first_row_in_archive_order":{"model":"roberta-base-mnli","paper_title":"Probing neural language models for understanding of words of estimative probability","paper_url":"/paper/probing-neural-language-models-for","paper_date":"2022-11-07","arxiv_id":"2211.03358","code_links":[],"syntology":null}},{"leaderboard":"/sota/natural-language-inference-on-quora-question","slug":"natural-language-inference-on-quora-question","dataset":"Quora Question Pairs","dataset_url":"/dataset/quora-question-pairs","rows_in_archive":1,"metrics":["Accuracy"],"first_row_in_archive_order":{"model":"aESIM","paper_title":"Attention Boosted Sequential Inference Model","paper_url":"/paper/attention-boosted-sequential-inference-model","paper_date":"2018-12-05","arxiv_id":"1812.01840","code_links":[],"syntology":null}},{"leaderboard":"/sota/natural-language-inference-on-sick","slug":"natural-language-inference-on-sick","dataset":"SICK","dataset_url":"/dataset/sick","rows_in_archive":1,"metrics":["1:1 Accuracy"],"first_row_in_archive_order":{"model":"NeuralLog","paper_title":"NeuralLog: Natural Language Inference with Joint Neural and Logical Reasoning","paper_url":"/paper/neurallog-natural-language-inference-with","paper_date":"2021-05-29","arxiv_id":"2105.14167","code_links":[{"title":"eric11eca/NeuralLog","url":"https://github.com/eric11eca/NeuralLog"}],"syntology":{"n":6,"n_ran":3,"n_constructed":0,"n_ran_checked":3,"n_instrument":0,"n_unverified":3,"n_honours":0,"n_violates":0,"n_no_contract":3,"n_pointer_only":0,"phrase":"3 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 0 violated, 3 with no contract checked; 0 where Syntology's instrument failed) · 3 unverified","sample_list":"/paper/neurallog-natural-language-inference-with#ran","syntology_url":"https://syntology.ai/paper/2105.14167","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2105.14167"}},"official":{"repos":["eric11eca/NeuralLog"],"state":"official (archive's flag): 3 ran","n_ran":3,"n_constructed":0,"n_ran_no_instrument_failure":3,"n_unverified":3,"ran_from_kinds":["official"]}}}},{"leaderboard":"/sota/natural-language-inference-on-tabfact","slug":"natural-language-inference-on-tabfact","dataset":"TabFact","dataset_url":"/dataset/tabfact","rows_in_archive":1,"metrics":["Accuracy"],"first_row_in_archive_order":{"model":"ChatGPT 3.5 SpatialFormat","paper_title":"LAPDoc: Layout-Aware Prompting for Documents","paper_url":"/paper/lapdoc-layout-aware-prompting-for-documents","paper_date":"2024-02-15","arxiv_id":"2402.09841","code_links":[],"syntology":null}},{"leaderboard":"/sota/natural-language-inference-on-xwino","slug":"natural-language-inference-on-xwino","dataset":"XWINO","dataset_url":"/dataset/xwino","rows_in_archive":1,"metrics":["Accuracy"],"first_row_in_archive_order":{"model":"mGPT","paper_title":"mGPT: Few-Shot Learners Go Multilingual","paper_url":"/paper/mgpt-few-shot-learners-go-multilingual","paper_date":"2022-04-15","arxiv_id":"2204.07580","code_links":[{"title":"ai-forever/mgpt","url":"https://github.com/ai-forever/mgpt"}],"syntology":null}},{"leaderboard":null,"slug":"natural-language-inference-on-anli","dataset":"ANLI","dataset_url":"/dataset/anli","rows_in_archive":0,"metrics":["Accuracy","F1 Macro","F1 Micro","F1 Weighted","Precision Macro","Precision Micro","Precision Weighted","Recall Macro","Recall Micro","Recall Weighted","loss"],"first_row_in_archive_order":null},{"leaderboard":null,"slug":"natural-language-inference-on-anli-r3","dataset":"ANLI-r3","dataset_url":"/dataset/anli","rows_in_archive":0,"metrics":["Accuracy"],"first_row_in_archive_order":null},{"leaderboard":null,"slug":"natural-language-inference-on-glue","dataset":"GLUE","dataset_url":"/dataset/glue","rows_in_archive":0,"metrics":["Accuracy","F1 Macro","F1 Micro","F1 Weighted","Precision Macro","Precision Micro","Precision Weighted","Recall Macro","Recall Micro","Recall Weighted","loss"],"first_row_in_archive_order":null},{"leaderboard":null,"slug":"natural-language-inference-on-multi-nli","dataset":"multi_nli","dataset_url":"/dataset/multinli","rows_in_archive":0,"metrics":["Accuracy","F1 Macro","F1 Micro","F1 Weighted","Precision Macro","Precision Micro","Precision Weighted","Recall Macro","Recall Micro","Recall Weighted","loss"],"first_row_in_archive_order":null}],"datasets":[{"url":"/dataset/glue","name":"GLUE","full_name":"General Language Understanding Evaluation benchmark","num_papers_in_archive":3197},{"url":"/dataset/multinli","name":"MultiNLI","full_name":"Multi-Genre Natural Language Inference","num_papers_in_archive":1830},{"url":"/dataset/snli","name":"SNLI","full_name":"Stanford Natural Language Inference","num_papers_in_archive":1311},{"url":"/dataset/qnli","name":"QNLI","full_name":"Question-answering NLI","num_papers_in_archive":1234},{"url":"/dataset/mrpc","name":"MRPC","full_name":"Microsoft Research Paraphrase Corpus","num_papers_in_archive":786},{"url":"/dataset/winogrande","name":"WinoGrande","full_name":"","num_papers_in_archive":703},{"url":"/dataset/xnli","name":"XNLI","full_name":"Cross-lingual Natural Language Inference","num_papers_in_archive":349},{"url":"/dataset/sick","name":"SICK","full_name":"Sentences Involving Compositional Knowledge","num_papers_in_archive":348},{"url":"/dataset/anli","name":"ANLI","full_name":"Adversarial NLI","num_papers_in_archive":287},{"url":"/dataset/paws","name":"PAWS","full_name":"Paraphrase Adversaries from Word Scrambling","num_papers_in_archive":159},{"url":"/dataset/e-snli","name":"e-SNLI","full_name":"","num_papers_in_archive":139},{"url":"/dataset/tabfact","name":"TabFact","full_name":"TabFact","num_papers_in_archive":129},{"url":"/dataset/snli-ve","name":"SNLI-VE","full_name":null,"num_papers_in_archive":117},{"url":"/dataset/gap-coreference-dataset","name":"GAP Coreference Dataset","full_name":"","num_papers_in_archive":104},{"url":"/dataset/listops","name":"ListOps","full_name":"ListOps","num_papers_in_archive":97},{"url":"/dataset/aqua-rat","name":"AQUA-RAT","full_name":"Algebra Question Answering with Rationales","num_papers_in_archive":64},{"url":"/dataset/rte","name":"RTE","full_name":"Recognizing Textual Entailment","num_papers_in_archive":56},{"url":"/dataset/quora-question-pairs","name":"Quora Question Pairs","full_name":"","num_papers_in_archive":55},{"url":"/dataset/ocnli","name":"OCNLI","full_name":"Original Chinese Natural Language Inference","num_papers_in_archive":44},{"url":"/dataset/scrolls","name":"SCROLLS","full_name":"Standardized CompaRison Over Long Language Sequences","num_papers_in_archive":42},{"url":"/dataset/break","name":"BREAK","full_name":"BREAK","num_papers_in_archive":39},{"url":"/dataset/medquad","name":"MedQuAD","full_name":"Medical Question Answering Dataset","num_papers_in_archive":32},{"url":"/dataset/contractnli","name":"ContractNLI","full_name":"","num_papers_in_archive":31},{"url":"/dataset/help","name":"HELP","full_name":"","num_papers_in_archive":30},{"url":"/dataset/chaosnli","name":"ChaosNLI","full_name":"","num_papers_in_archive":27},{"url":"/dataset/headqa","name":"HeadQA","full_name":"","num_papers_in_archive":23},{"url":"/dataset/xglue","name":"XGLUE","full_name":"","num_papers_in_archive":22},{"url":"/dataset/klue","name":"KLUE","full_name":"Korean Language Understanding Evaluation","num_papers_in_archive":21},{"url":"/dataset/quarel","name":"QuaRel","full_name":"QuaRel","num_papers_in_archive":20},{"url":"/dataset/torque","name":"Torque","full_name":"","num_papers_in_archive":19},{"url":"/dataset/infotabs","name":"InfoTabS","full_name":"","num_papers_in_archive":18},{"url":"/dataset/kornli","name":"KorNLI","full_name":null,"num_papers_in_archive":18},{"url":"/dataset/med","name":"MED","full_name":"Monotonicity Entailment Dataset","num_papers_in_archive":18},{"url":"/dataset/indicglue","name":"IndicGLUE","full_name":"Indic General Language Understanding Evaluation Benchmark","num_papers_in_archive":16},{"url":"/dataset/wnli","name":"WNLI","full_name":"Winograd NLI","num_papers_in_archive":14},{"url":"/dataset/korsts","name":"KorSTS","full_name":"","num_papers_in_archive":13},{"url":"/dataset/story-commonsense","name":"Story Commonsense","full_name":"","num_papers_in_archive":13},{"url":"/dataset/flue-french-language-understanding-evaluation","name":"FLUE","full_name":"French Language Understanding Evaluation","num_papers_in_archive":12},{"url":"/dataset/taxinli","name":"TaxiNLI","full_name":"","num_papers_in_archive":11},{"url":"/dataset/commitmentbank","name":"CommitmentBank","full_name":"","num_papers_in_archive":10},{"url":"/dataset/nli4ct","name":"NLI4CT","full_name":"","num_papers_in_archive":10},{"url":"/dataset/art-dataset","name":"ART Dataset","full_name":"Abductive Reasoning in narrative Text","num_papers_in_archive":9},{"url":"/dataset/farstail","name":"FarsTail","full_name":"","num_papers_in_archive":9},{"url":"/dataset/mednli","name":"MedNLI","full_name":"Medical Natural Language Inference","num_papers_in_archive":9},{"url":"/dataset/scitail","name":"SciTail","full_name":"","num_papers_in_archive":9},{"url":"/dataset/selqa","name":"SelQA","full_name":"","num_papers_in_archive":9},{"url":"/dataset/docnli","name":"DocNLI","full_name":"","num_papers_in_archive":8},{"url":"/dataset/danetqa","name":"DaNetQA","full_name":"Yes/no Question Answering Dataset for the Russian","num_papers_in_archive":7},{"url":"/dataset/marc-ja","name":"JGLUE","full_name":"","num_papers_in_archive":7},{"url":"/dataset/kuake-qqr","name":"KUAKE-QQR","full_name":"Query-Query Relevance Dataset","num_papers_in_archive":7},{"url":"/dataset/kuake-qtr","name":"KUAKE-QTR","full_name":"Query-Title Relevance Dataset","num_papers_in_archive":7},{"url":"/dataset/terra","name":"TERRa","full_name":"Textual Entailment Recognition for Russian","num_papers_in_archive":7},{"url":"/dataset/imppres","name":"IMPPRES","full_name":"","num_papers_in_archive":6},{"url":"/dataset/rcb","name":"RCB","full_name":"Russian Commitment Bank","num_papers_in_archive":6},{"url":"/dataset/bionli","name":"BioNLI","full_name":"Biomedical Natural Language Inference","num_papers_in_archive":5},{"url":"/dataset/lidirus","name":"LiDiRus","full_name":"Linguistic Diagnostic for Russian","num_papers_in_archive":5},{"url":"/dataset/rusentrel","name":"RuSentRel","full_name":"","num_papers_in_archive":5},{"url":"/dataset/indonli","name":"IndoNLI","full_name":"","num_papers_in_archive":4},{"url":"/dataset/mindgames","name":"Mindgames","full_name":"","num_papers_in_archive":4},{"url":"/dataset/xwino","name":"XWINO","full_name":"","num_papers_in_archive":4},{"url":"/dataset/esxnli","name":"esXNLI","full_name":null,"num_papers_in_archive":3},{"url":"/dataset/janli","name":"JaNLI","full_name":"Japanese Adversarial Natural Language Inference","num_papers_in_archive":3},{"url":"/dataset/justice","name":"JUSTICE","full_name":"JUSTICE: A Dataset for Supreme Court’s Judgment Prediction","num_papers_in_archive":3},{"url":"/dataset/pars-absa","name":"Pars-ABSA","full_name":"","num_papers_in_archive":3},{"url":"/dataset/tasksource","name":"Tasksource","full_name":"","num_papers_in_archive":3},{"url":"/dataset/wili-2018","name":"WiLI-2018","full_name":"","num_papers_in_archive":3},{"url":"/dataset/arnli","name":"ArNLI","full_name":"","num_papers_in_archive":2},{"url":"/dataset/newsph-nli","name":"NewsPH-NLI","full_name":"","num_papers_in_archive":2},{"url":"/dataset/nli-tr","name":"NLI-TR","full_name":"Natural Language Inference in Turkish","num_papers_in_archive":2},{"url":"/dataset/propsegment","name":"PropSegmEnt","full_name":"","num_papers_in_archive":2},{"url":"/dataset/candor-corpus-1","name":"CANDOR Corpus","full_name":"CANDOR = Conversation: A Naturalistic Dataset of Online Recordings","num_papers_in_archive":1},{"url":"/dataset/distnli","name":"DistNLI","full_name":"","num_papers_in_archive":1},{"url":"/dataset/gd-nli","name":"GD-NLI","full_name":"Generated Debiased NLI Datasets","num_papers_in_archive":1},{"url":"/dataset/gigaword-entailment","name":"Gigaword Entailment","full_name":null,"num_papers_in_archive":1},{"url":"/dataset/gqnli-fr","name":"GQNLI-FR","full_name":"","num_papers_in_archive":1},{"url":"/dataset/hans","name":"HANS","full_name":"Heuristic Analysis for NLI Systems","num_papers_in_archive":1},{"url":"/dataset/jampatoisnli","name":"JamPatoisNLI","full_name":"","num_papers_in_archive":1},{"url":"/dataset/nli4wills-corpus","name":"NLI4Wills Corpus","full_name":"","num_papers_in_archive":1},{"url":"/dataset/probability-words-nli","name":"Probability words NLI","full_name":"Natural language inference with words estimative of probability (WEP)","num_papers_in_archive":1},{"url":"/dataset/probability-words-nli-1","name":"probability_words_nli","full_name":"","num_papers_in_archive":1},{"url":"/dataset/rte3-fr","name":"RTE3-FR","full_name":"","num_papers_in_archive":1}],"subtasks":[{"url":"/task/answer-generation","name":"Answer Generation"},{"url":"/task/cross-lingual-natural-language-inference","name":"Cross-Lingual Natural Language Inference"},{"url":"/task/visual-entailment","name":"Visual Entailment"}],"parent_tasks":[],"papers":{"order":"repositories listed in the archive (most first), then date (newest first), then slug; the archive holds no stars","population":"every paper the archive tags with this task (distinct papers; spam titles not listed). 'of' counts the papers listed; until 2026-09-25 it counted only the papers with a code link, and that number is of_with_code","page":1,"pages_in_order":20,"rows_per_page":100,"shown":100,"of":1961,"of_with_code":821,"tagged_in_all":1961,"counts":{"archive_papers_tagged":1961,"with_a_code_link":821,"where_syntology_ran_a_sample":209,"not_listed_spam_title":0,"listed":1961,"listed_where_code_ran":209,"where_syntology_ran_a_sample_split":{"with_a_run_with_no_instrument_failure":167,"every_run_a_failure_of_syntologys_instrument":42,"listed_with_a_run_with_no_instrument_failure":167,"listed_every_run_a_failure_of_syntologys_instrument":42,"filter":{"states":["a run with no instrument failure","any run, instrument failures included"],"default":"a run with no instrument failure","note":"on the 'only where code ran' pages the default hides, in the browser, the rows where every run was a failure of Syntology's instrument; the second state shows them again. Rows are hidden, never re-ordered; these twins list every row"}},"definition":"distinct papers the archive tags; 'where Syntology ran a sample' counts papers with at least one harvested sample that ran, which is not a correctness claim"},"items":[{"url":"/paper/bert-pre-training-of-deep-bidirectional","slug":"bert-pre-training-of-deep-bidirectional","title":"BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding","date":"2018-10-11","arxiv_id":"1810.04805","repositories_listed":534,"syntology":{"n":659,"n_ran":300,"n_constructed":75,"n_ran_checked":234,"n_instrument":66,"n_unverified":359,"n_honours":18,"n_violates":5,"n_no_contract":211,"n_pointer_only":164,"phrase":"300 ran (of which 75 constructed an object rather than computing a result; 234 with no instrument failure: 18 honoured, 5 violated, 211 with no contract checked; 66 where Syntology's instrument failed) · 359 unverified","sample_list":"/paper/bert-pre-training-of-deep-bidirectional#ran","syntology_url":"https://syntology.ai/paper/1810.04805","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1810.04805"}},"official":{"repos":["google-research/bert"],"state":"official: no sample here; runs from other or unrecorded repositories","n_ran":0,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":0,"ran_from_kinds":["listed","unlocated"]}}},{"url":"/paper/language-models-are-few-shot-learners","slug":"language-models-are-few-shot-learners","title":"Language Models are Few-Shot Learners","date":"2020-05-28","arxiv_id":"2005.14165","repositories_listed":67,"syntology":{"n":65,"n_ran":45,"n_constructed":0,"n_ran_checked":37,"n_instrument":8,"n_unverified":20,"n_honours":3,"n_violates":2,"n_no_contract":32,"n_pointer_only":7,"phrase":"45 ran (of which 0 constructed an object rather than computing a result; 37 with no instrument failure: 3 honoured, 2 violated, 32 with no contract checked; 8 where Syntology's instrument failed) · 20 unverified","sample_list":"/paper/language-models-are-few-shot-learners#ran","syntology_url":"https://syntology.ai/paper/2005.14165","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2005.14165"}},"official":{"repos":["openai/gpt-3"],"state":"community repositories only","n_ran":0,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":0,"ran_from_kinds":["listed"]}}},{"url":"/paper/roberta-a-robustly-optimized-bert-pretraining","slug":"roberta-a-robustly-optimized-bert-pretraining","title":"RoBERTa: A Robustly Optimized BERT Pretraining Approach","date":"2019-07-26","arxiv_id":"1907.11692","repositories_listed":67,"syntology":{"n":48,"n_ran":37,"n_constructed":11,"n_ran_checked":36,"n_instrument":1,"n_unverified":11,"n_honours":2,"n_violates":0,"n_no_contract":34,"n_pointer_only":24,"phrase":"37 ran (of which 11 constructed an object rather than computing a result; 36 with no instrument failure: 2 honoured, 0 violated, 34 with no contract checked; 1 where Syntology's instrument failed) · 11 unverified","sample_list":"/paper/roberta-a-robustly-optimized-bert-pretraining#ran","syntology_url":"https://syntology.ai/paper/1907.11692","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1907.11692"}},"official":{"repos":["pytorch/fairseq"],"state":"community repositories only","n_ran":0,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":0,"ran_from_kinds":["listed"]}}},{"url":"/paper/exploring-the-limits-of-transfer-learning","slug":"exploring-the-limits-of-transfer-learning","title":"Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer","date":"2019-10-23","arxiv_id":"1910.10683","repositories_listed":57,"syntology":{"n":31,"n_ran":21,"n_constructed":0,"n_ran_checked":20,"n_instrument":1,"n_unverified":10,"n_honours":1,"n_violates":0,"n_no_contract":19,"n_pointer_only":0,"phrase":"21 ran (of which 0 constructed an object rather than computing a result; 20 with no instrument failure: 1 honoured, 0 violated, 19 with no contract checked; 1 where Syntology's instrument failed) · 10 unverified","sample_list":"/paper/exploring-the-limits-of-transfer-learning#ran","syntology_url":"https://syntology.ai/paper/1910.10683","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1910.10683"}},"official":null}},{"url":"/paper/a-structured-self-attentive-sentence","slug":"a-structured-self-attentive-sentence","title":"A Structured Self-attentive Sentence Embedding","date":"2017-03-09","arxiv_id":"1703.03130","repositories_listed":52,"syntology":{"n":10,"n_ran":10,"n_constructed":0,"n_ran_checked":6,"n_instrument":4,"n_unverified":0,"n_honours":0,"n_violates":0,"n_no_contract":6,"n_pointer_only":3,"phrase":"10 ran (of which 0 constructed an object rather than computing a result; 6 with no instrument failure: 0 honoured, 0 violated, 6 with no contract checked; 4 where Syntology's instrument failed) · 0 unverified","sample_list":"/paper/a-structured-self-attentive-sentence#ran","syntology_url":"https://syntology.ai/paper/1703.03130","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1703.03130"}},"official":null}},{"url":"/paper/albert-a-lite-bert-for-self-supervised","slug":"albert-a-lite-bert-for-self-supervised","title":"ALBERT: A Lite BERT for Self-supervised Learning of Language Representations","date":"2019-09-26","arxiv_id":"1909.11942","repositories_listed":48,"syntology":{"n":126,"n_ran":81,"n_constructed":17,"n_ran_checked":56,"n_instrument":25,"n_unverified":45,"n_honours":4,"n_violates":0,"n_no_contract":52,"n_pointer_only":28,"phrase":"81 ran (of which 17 constructed an object rather than computing a result; 56 with no instrument failure: 4 honoured, 0 violated, 52 with no contract checked; 25 where Syntology's instrument failed) · 45 unverified","sample_list":"/paper/albert-a-lite-bert-for-self-supervised#ran","syntology_url":"https://syntology.ai/paper/1909.11942","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1909.11942"}},"official":{"repos":["google-research/ALBERT"],"state":"official (archive's flag): 6 ran","n_ran":6,"n_constructed":0,"n_ran_no_instrument_failure":4,"n_unverified":0,"ran_from_kinds":["listed","official"]}}},{"url":"/paper/bart-denoising-sequence-to-sequence-pre","slug":"bart-denoising-sequence-to-sequence-pre","title":"BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension","date":"2019-10-29","arxiv_id":"1910.13461","repositories_listed":47,"syntology":{"n":53,"n_ran":40,"n_constructed":7,"n_ran_checked":30,"n_instrument":10,"n_unverified":13,"n_honours":1,"n_violates":2,"n_no_contract":27,"n_pointer_only":13,"phrase":"40 ran (of which 7 constructed an object rather than computing a result; 30 with no instrument failure: 1 honoured, 2 violated, 27 with no contract checked; 10 where Syntology's instrument failed) · 13 unverified","sample_list":"/paper/bart-denoising-sequence-to-sequence-pre#ran","syntology_url":"https://syntology.ai/paper/1910.13461","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1910.13461"}},"official":null}},{"url":"/paper/deep-contextualized-word-representations","slug":"deep-contextualized-word-representations","title":"Deep contextualized word representations","date":"2018-02-15","arxiv_id":"1802.05365","repositories_listed":46,"syntology":{"n":58,"n_ran":30,"n_constructed":12,"n_ran_checked":24,"n_instrument":6,"n_unverified":28,"n_honours":2,"n_violates":0,"n_no_contract":22,"n_pointer_only":25,"phrase":"30 ran (of which 12 constructed an object rather than computing a result; 24 with no instrument failure: 2 honoured, 0 violated, 22 with no contract checked; 6 where Syntology's instrument failed) · 28 unverified","sample_list":"/paper/deep-contextualized-word-representations#ran","syntology_url":"https://syntology.ai/paper/1802.05365","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1802.05365"}},"official":null}},{"url":"/paper/distilbert-a-distilled-version-of-bert","slug":"distilbert-a-distilled-version-of-bert","title":"DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter","date":"2019-10-02","arxiv_id":"1910.01108","repositories_listed":37,"syntology":{"n":27,"n_ran":21,"n_constructed":5,"n_ran_checked":12,"n_instrument":9,"n_unverified":6,"n_honours":3,"n_violates":1,"n_no_contract":8,"n_pointer_only":2,"phrase":"21 ran (of which 5 constructed an object rather than computing a result; 12 with no instrument failure: 3 honoured, 1 violated, 8 with no contract checked; 9 where Syntology's instrument failed) · 6 unverified","sample_list":"/paper/distilbert-a-distilled-version-of-bert#ran","syntology_url":"https://syntology.ai/paper/1910.01108","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1910.01108"}},"official":{"repos":["huggingface/swift-coreml-transformers","huggingface/transformers"],"state":"official (archive's flag): 1 ran","n_ran":1,"n_constructed":0,"n_ran_no_instrument_failure":1,"n_unverified":0,"ran_from_kinds":["listed","official"]}}},{"url":"/paper/xlnet-generalized-autoregressive-pretraining","slug":"xlnet-generalized-autoregressive-pretraining","title":"XLNet: Generalized Autoregressive Pretraining for Language Understanding","date":"2019-06-19","arxiv_id":"1906.08237","repositories_listed":27,"syntology":{"n":24,"n_ran":15,"n_constructed":2,"n_ran_checked":10,"n_instrument":5,"n_unverified":9,"n_honours":2,"n_violates":0,"n_no_contract":8,"n_pointer_only":4,"phrase":"15 ran (of which 2 constructed an object rather than computing a result; 10 with no instrument failure: 2 honoured, 0 violated, 8 with no contract checked; 5 where Syntology's instrument failed) · 9 unverified","sample_list":"/paper/xlnet-generalized-autoregressive-pretraining#ran","syntology_url":"https://syntology.ai/paper/1906.08237","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1906.08237"}},"official":{"repos":["zihangdai/xlnet"],"state":"official (archive's flag): 1 ran","n_ran":1,"n_constructed":0,"n_ran_no_instrument_failure":1,"n_unverified":1,"ran_from_kinds":["listed","official"]}}},{"url":"/paper/simcse-simple-contrastive-learning-of","slug":"simcse-simple-contrastive-learning-of","title":"SimCSE: Simple Contrastive Learning of Sentence Embeddings","date":"2021-04-18","arxiv_id":"2104.08821","repositories_listed":23,"syntology":{"n":30,"n_ran":17,"n_constructed":9,"n_ran_checked":12,"n_instrument":5,"n_unverified":13,"n_honours":1,"n_violates":0,"n_no_contract":11,"n_pointer_only":19,"phrase":"17 ran (of which 9 constructed an object rather than computing a result; 12 with no instrument failure: 1 honoured, 0 violated, 11 with no contract checked; 5 where Syntology's instrument failed) · 13 unverified","sample_list":"/paper/simcse-simple-contrastive-learning-of#ran","syntology_url":"https://syntology.ai/paper/2104.08821","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2104.08821"}},"official":{"repos":["princeton-nlp/SimCSE"],"state":"community repositories only","n_ran":0,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":0,"ran_from_kinds":["listed"]}}},{"url":"/paper/supervised-learning-of-universal-sentence","slug":"supervised-learning-of-universal-sentence","title":"Supervised Learning of Universal Sentence Representations from Natural Language Inference Data","date":"2017-05-05","arxiv_id":"1705.02364","repositories_listed":23,"syntology":{"n":7,"n_ran":6,"n_constructed":0,"n_ran_checked":2,"n_instrument":4,"n_unverified":1,"n_honours":2,"n_violates":0,"n_no_contract":0,"n_pointer_only":7,"phrase":"6 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 2 honoured, 0 violated, 0 with no contract checked; 4 where Syntology's instrument failed) · 1 unverified","sample_list":"/paper/supervised-learning-of-universal-sentence#ran","syntology_url":"https://syntology.ai/paper/1705.02364","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1705.02364"}},"official":{"repos":["facebookresearch/InferSent","facebookresearch/SentEval"],"state":"official (archive's flag): 3 ran","n_ran":3,"n_constructed":0,"n_ran_no_instrument_failure":1,"n_unverified":0,"ran_from_kinds":["listed","official"]}}},{"url":"/paper/pay-attention-to-mlps","slug":"pay-attention-to-mlps","title":"Pay Attention to MLPs","date":"2021-05-17","arxiv_id":"2105.08050","repositories_listed":20,"syntology":{"n":44,"n_ran":34,"n_constructed":15,"n_ran_checked":31,"n_instrument":3,"n_unverified":10,"n_honours":1,"n_violates":4,"n_no_contract":26,"n_pointer_only":11,"phrase":"34 ran (of which 15 constructed an object rather than computing a result; 31 with no instrument failure: 1 honoured, 4 violated, 26 with no contract checked; 3 where Syntology's instrument failed) · 10 unverified","sample_list":"/paper/pay-attention-to-mlps#ran","syntology_url":"https://syntology.ai/paper/2105.08050","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2105.08050"}},"official":null}},{"url":"/paper/ernie-enhanced-representation-through","slug":"ernie-enhanced-representation-through","title":"ERNIE: Enhanced Representation through Knowledge Integration","date":"2019-04-19","arxiv_id":"1904.09223","repositories_listed":19,"syntology":{"n":7,"n_ran":6,"n_constructed":0,"n_ran_checked":6,"n_instrument":0,"n_unverified":1,"n_honours":0,"n_violates":0,"n_no_contract":6,"n_pointer_only":0,"phrase":"6 ran (of which 0 constructed an object rather than computing a result; 6 with no instrument failure: 0 honoured, 0 violated, 6 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified","sample_list":"/paper/ernie-enhanced-representation-through#ran","syntology_url":"https://syntology.ai/paper/1904.09223","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1904.09223"}},"official":{"repos":["PaddlePaddle/PaddleNLP"],"state":"community repositories only","n_ran":0,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":0,"ran_from_kinds":["listed"]}}},{"url":"/paper/rwkv-reinventing-rnns-for-the-transformer-era","slug":"rwkv-reinventing-rnns-for-the-transformer-era","title":"RWKV: Reinventing RNNs for the Transformer Era","date":"2023-05-22","arxiv_id":"2305.13048","repositories_listed":14,"syntology":{"n":11,"n_ran":6,"n_constructed":0,"n_ran_checked":4,"n_instrument":2,"n_unverified":5,"n_honours":1,"n_violates":0,"n_no_contract":3,"n_pointer_only":0,"phrase":"6 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 1 honoured, 0 violated, 3 with no contract checked; 2 where Syntology's instrument failed) · 5 unverified","sample_list":"/paper/rwkv-reinventing-rnns-for-the-transformer-era#ran","syntology_url":"https://syntology.ai/paper/2305.13048","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2305.13048"}},"official":{"repos":["BlinkDL/RWKV-LM","blinkdl/chatrwkv"],"state":"official (archive's flag): 3 ran","n_ran":3,"n_constructed":0,"n_ran_no_instrument_failure":1,"n_unverified":4,"ran_from_kinds":["listed","official"]}}},{"url":"/paper/big-bird-transformers-for-longer-sequences","slug":"big-bird-transformers-for-longer-sequences","title":"Big Bird: Transformers for Longer Sequences","date":"2020-07-28","arxiv_id":"2007.14062","repositories_listed":14,"syntology":{"n":15,"n_ran":10,"n_constructed":1,"n_ran_checked":2,"n_instrument":8,"n_unverified":5,"n_honours":1,"n_violates":0,"n_no_contract":1,"n_pointer_only":11,"phrase":"10 ran (of which 1 constructed an object rather than computing a result; 2 with no instrument failure: 1 honoured, 0 violated, 1 with no contract checked; 8 where Syntology's instrument failed) · 5 unverified","sample_list":"/paper/big-bird-transformers-for-longer-sequences#ran","syntology_url":"https://syntology.ai/paper/2007.14062","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2007.14062"}},"official":{"repos":["google-research/bigbird"],"state":"official (archive's flag): 1 ran","n_ran":1,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":1,"ran_from_kinds":["listed","official"]}}},{"url":"/paper/deberta-decoding-enhanced-bert-with","slug":"deberta-decoding-enhanced-bert-with","title":"DeBERTa: Decoding-enhanced BERT with Disentangled Attention","date":"2020-06-05","arxiv_id":"2006.03654","repositories_listed":14,"syntology":{"n":13,"n_ran":4,"n_constructed":0,"n_ran_checked":3,"n_instrument":1,"n_unverified":9,"n_honours":2,"n_violates":1,"n_no_contract":0,"n_pointer_only":3,"phrase":"4 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 2 honoured, 1 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 9 unverified","sample_list":"/paper/deberta-decoding-enhanced-bert-with#ran","syntology_url":"https://syntology.ai/paper/2006.03654","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2006.03654"}},"official":{"repos":["microsoft/DeBERTa"],"state":"official: no sample here; runs from other or unrecorded repositories","n_ran":0,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":0,"ran_from_kinds":["listed","named_in_paper","unlocated"]}}},{"url":"/paper/massively-multilingual-sentence-embeddings","slug":"massively-multilingual-sentence-embeddings","title":"Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond","date":"2018-12-26","arxiv_id":"1812.10464","repositories_listed":13,"syntology":{"n":10,"n_ran":10,"n_constructed":0,"n_ran_checked":8,"n_instrument":2,"n_unverified":0,"n_honours":2,"n_violates":0,"n_no_contract":6,"n_pointer_only":4,"phrase":"10 ran (of which 0 constructed an object rather than computing a result; 8 with no instrument failure: 2 honoured, 0 violated, 6 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified","sample_list":"/paper/massively-multilingual-sentence-embeddings#ran","syntology_url":"https://syntology.ai/paper/1812.10464","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1812.10464"}},"official":{"repos":["facebookresearch/LASER"],"state":"community repositories only","n_ran":0,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":0,"ran_from_kinds":["listed"]}}},{"url":"/paper/improving-language-understanding-by","slug":"improving-language-understanding-by","title":"Improving Language Understanding by Generative Pre-Training","date":"2018-06-11","arxiv_id":null,"repositories_listed":13,"syntology":null},{"url":"/paper/data2vec-a-general-framework-for-self-1","slug":"data2vec-a-general-framework-for-self-1","title":"data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language","date":"2022-02-07","arxiv_id":"2202.03555","repositories_listed":12,"syntology":{"n":6,"n_ran":4,"n_constructed":0,"n_ran_checked":4,"n_instrument":0,"n_unverified":2,"n_honours":0,"n_violates":0,"n_no_contract":4,"n_pointer_only":0,"phrase":"4 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 0 honoured, 0 violated, 4 with no contract checked; 0 where Syntology's instrument failed) · 2 unverified","sample_list":"/paper/data2vec-a-general-framework-for-self-1#ran","syntology_url":"https://syntology.ai/paper/2202.03555","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2202.03555"}},"official":{"repos":["pytorch/fairseq"],"state":"community repositories only","n_ran":0,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":0,"ran_from_kinds":["listed"]}}},{"url":"/paper/fnet-mixing-tokens-with-fourier-transforms","slug":"fnet-mixing-tokens-with-fourier-transforms","title":"FNet: Mixing Tokens with Fourier Transforms","date":"2021-05-09","arxiv_id":"2105.03824","repositories_listed":12,"syntology":{"n":2,"n_ran":2,"n_constructed":0,"n_ran_checked":1,"n_instrument":1,"n_unverified":0,"n_honours":1,"n_violates":0,"n_no_contract":0,"n_pointer_only":1,"phrase":"2 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 1 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified","sample_list":"/paper/fnet-mixing-tokens-with-fourier-transforms#ran","syntology_url":"https://syntology.ai/paper/2105.03824","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2105.03824"}},"official":{"repos":["google-research/google-research"],"state":"community repositories only","n_ran":0,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":0,"ran_from_kinds":["listed"]}}},{"url":"/paper/enhanced-lstm-for-natural-language-inference","slug":"enhanced-lstm-for-natural-language-inference","title":"Enhanced LSTM for Natural Language Inference","date":"2016-09-20","arxiv_id":"1609.06038","repositories_listed":12,"syntology":null},{"url":"/paper/glue-a-multi-task-benchmark-and-analysis","slug":"glue-a-multi-task-benchmark-and-analysis","title":"GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding","date":"2018-04-20","arxiv_id":"1804.07461","repositories_listed":11,"syntology":{"n":14,"n_ran":14,"n_constructed":0,"n_ran_checked":12,"n_instrument":2,"n_unverified":0,"n_honours":1,"n_violates":1,"n_no_contract":10,"n_pointer_only":1,"phrase":"14 ran (of which 0 constructed an object rather than computing a result; 12 with no instrument failure: 1 honoured, 1 violated, 10 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified","sample_list":"/paper/glue-a-multi-task-benchmark-and-analysis#ran","syntology_url":"https://syntology.ai/paper/1804.07461","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1804.07461"}},"official":null}},{"url":"/paper/senteval-an-evaluation-toolkit-for-universal","slug":"senteval-an-evaluation-toolkit-for-universal","title":"SentEval: An Evaluation Toolkit for Universal Sentence Representations","date":"2018-03-14","arxiv_id":"1803.05449","repositories_listed":11,"syntology":{"n":8,"n_ran":8,"n_constructed":0,"n_ran_checked":5,"n_instrument":3,"n_unverified":0,"n_honours":2,"n_violates":1,"n_no_contract":2,"n_pointer_only":6,"phrase":"8 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 2 honoured, 1 violated, 2 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified","sample_list":"/paper/senteval-an-evaluation-toolkit-for-universal#ran","syntology_url":"https://syntology.ai/paper/1803.05449","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1803.05449"}},"official":{"repos":["facebookresearch/SentEval"],"state":"official (archive's flag): 3 ran","n_ran":3,"n_constructed":0,"n_ran_no_instrument_failure":1,"n_unverified":0,"ran_from_kinds":["listed","official"]}}},{"url":"/paper/from-softmax-to-sparsemax-a-sparse-model-of","slug":"from-softmax-to-sparsemax-a-sparse-model-of","title":"From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classification","date":"2016-02-05","arxiv_id":"1602.02068","repositories_listed":11,"syntology":{"n":8,"n_ran":8,"n_constructed":0,"n_ran_checked":6,"n_instrument":2,"n_unverified":0,"n_honours":4,"n_violates":2,"n_no_contract":0,"n_pointer_only":2,"phrase":"8 ran (of which 0 constructed an object rather than computing a result; 6 with no instrument failure: 4 honoured, 2 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified","sample_list":"/paper/from-softmax-to-sparsemax-a-sparse-model-of#ran","syntology_url":"https://syntology.ai/paper/1602.02068","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1602.02068"}},"official":null}},{"url":"/paper/new-datasets-for-automatic-detection-of","slug":"new-datasets-for-automatic-detection-of","title":"New Datasets for Automatic Detection of Textual Entailment and of Contradictions between Sentences in French","date":"2024-05-23","arxiv_id":null,"repositories_listed":10,"syntology":null},{"url":"/paper/nystromformer-a-nystrom-based-algorithm-for","slug":"nystromformer-a-nystrom-based-algorithm-for","title":"Nyströmformer: A Nyström-Based Algorithm for Approximating Self-Attention","date":"2021-02-07","arxiv_id":"2102.03902","repositories_listed":10,"syntology":{"n":2,"n_ran":1,"n_constructed":0,"n_ran_checked":1,"n_instrument":0,"n_unverified":1,"n_honours":0,"n_violates":1,"n_no_contract":0,"n_pointer_only":1,"phrase":"1 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 1 violated, 0 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified","sample_list":"/paper/nystromformer-a-nystrom-based-algorithm-for#ran","syntology_url":"https://syntology.ai/paper/2102.03902","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2102.03902"}},"official":{"repos":["mlpen/Nystromformer"],"state":"official: no sample here; runs from other or unrecorded repositories","n_ran":0,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":0,"ran_from_kinds":["unlocated"]}}},{"url":"/paper/190910351","slug":"190910351","title":"TinyBERT: Distilling BERT for Natural Language Understanding","date":"2019-09-23","arxiv_id":"1909.10351","repositories_listed":10,"syntology":{"n":4,"n_ran":0,"n_constructed":0,"n_ran_checked":0,"n_instrument":0,"n_unverified":4,"n_honours":0,"n_violates":0,"n_no_contract":0,"n_pointer_only":4,"phrase":"0 ran · 4 unverified","sample_list":"/paper/190910351#ran","syntology_url":"https://syntology.ai/paper/1909.10351","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1909.10351"}},"official":{"repos":["huawei-noah/Pretrained-Language-Model"],"state":"official: harvested, nothing ran","n_ran":0,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":4,"ran_from_kinds":[]}}},{"url":"/paper/nezha-neural-contextualized-representation","slug":"nezha-neural-contextualized-representation","title":"NEZHA: Neural Contextualized Representation for Chinese Language Understanding","date":"2019-08-31","arxiv_id":"1909.00204","repositories_listed":10,"syntology":null},{"url":"/paper/bilateral-multi-perspective-matching-for","slug":"bilateral-multi-perspective-matching-for","title":"Bilateral Multi-Perspective Matching for Natural Language Sentences","date":"2017-02-13","arxiv_id":"1702.03814","repositories_listed":10,"syntology":{"n":4,"n_ran":1,"n_constructed":0,"n_ran_checked":0,"n_instrument":1,"n_unverified":3,"n_honours":0,"n_violates":0,"n_no_contract":0,"n_pointer_only":1,"phrase":"1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 3 unverified","sample_list":"/paper/bilateral-multi-perspective-matching-for#ran","syntology_url":"https://syntology.ai/paper/1702.03814","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1702.03814"}},"official":null}},{"url":"/paper/a-decomposable-attention-model-for-natural","slug":"a-decomposable-attention-model-for-natural","title":"A Decomposable Attention Model for Natural Language Inference","date":"2016-06-06","arxiv_id":"1606.01933","repositories_listed":10,"syntology":{"n":6,"n_ran":4,"n_constructed":0,"n_ran_checked":4,"n_instrument":0,"n_unverified":2,"n_honours":0,"n_violates":0,"n_no_contract":4,"n_pointer_only":0,"phrase":"4 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 0 honoured, 0 violated, 4 with no contract checked; 0 where Syntology's instrument failed) · 2 unverified","sample_list":"/paper/a-decomposable-attention-model-for-natural#ran","syntology_url":"https://syntology.ai/paper/1606.01933","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1606.01933"}},"official":null}},{"url":"/paper/xnli-evaluating-cross-lingual-sentence","slug":"xnli-evaluating-cross-lingual-sentence","title":"XNLI: Evaluating Cross-lingual Sentence Representations","date":"2018-09-13","arxiv_id":"1809.05053","repositories_listed":9,"syntology":null},{"url":"/paper/finetuned-language-models-are-zero-shot","slug":"finetuned-language-models-are-zero-shot","title":"Finetuned Language Models Are Zero-Shot Learners","date":"2021-09-03","arxiv_id":"2109.01652","repositories_listed":8,"syntology":{"n":1,"n_ran":0,"n_constructed":0,"n_ran_checked":0,"n_instrument":0,"n_unverified":1,"n_honours":0,"n_violates":0,"n_no_contract":0,"n_pointer_only":0,"phrase":"0 ran · 1 unverified","sample_list":"/paper/finetuned-language-models-are-zero-shot#ran","syntology_url":"https://syntology.ai/paper/2109.01652","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2109.01652"}},"official":{"repos":["google-research/flan"],"state":"official: harvested for another paper","n_ran":0,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":0,"ran_from_kinds":[]}}},{"url":"/paper/mt5-a-massively-multilingual-pre-trained-text","slug":"mt5-a-massively-multilingual-pre-trained-text","title":"mT5: A massively multilingual pre-trained text-to-text transformer","date":"2020-10-22","arxiv_id":"2010.11934","repositories_listed":8,"syntology":{"n":13,"n_ran":10,"n_constructed":0,"n_ran_checked":10,"n_instrument":0,"n_unverified":3,"n_honours":0,"n_violates":0,"n_no_contract":10,"n_pointer_only":0,"phrase":"10 ran (of which 0 constructed an object rather than computing a result; 10 with no instrument failure: 0 honoured, 0 violated, 10 with no contract checked; 0 where Syntology's instrument failed) · 3 unverified","sample_list":"/paper/mt5-a-massively-multilingual-pre-trained-text#ran","syntology_url":"https://syntology.ai/paper/2010.11934","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2010.11934"}},"official":{"repos":["google-research/multilingual-t5"],"state":"official (archive's flag): 5 ran","n_ran":5,"n_constructed":0,"n_ran_no_instrument_failure":5,"n_unverified":0,"ran_from_kinds":["listed","official"]}}},{"url":"/paper/distributionally-robust-neural-networks-for","slug":"distributionally-robust-neural-networks-for","title":"Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization","date":"2019-11-20","arxiv_id":"1911.08731","repositories_listed":8,"syntology":{"n":10,"n_ran":9,"n_constructed":3,"n_ran_checked":5,"n_instrument":4,"n_unverified":1,"n_honours":2,"n_violates":0,"n_no_contract":3,"n_pointer_only":2,"phrase":"9 ran (of which 3 constructed an object rather than computing a result; 5 with no instrument failure: 2 honoured, 0 violated, 3 with no contract checked; 4 where Syntology's instrument failed) · 1 unverified","sample_list":"/paper/distributionally-robust-neural-networks-for#ran","syntology_url":"https://syntology.ai/paper/1911.08731","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1911.08731"}},"official":{"repos":["kohpangwei/group_DRO","worksheets.codalab.org/worksheets/0x621811fe446b49bb818293bae2ef88c0"],"state":"community repositories only","n_ran":0,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":0,"ran_from_kinds":["listed"]}}},{"url":"/paper/camembert-a-tasty-french-language-model","slug":"camembert-a-tasty-french-language-model","title":"CamemBERT: a Tasty French Language Model","date":"2019-11-10","arxiv_id":"1911.03894","repositories_listed":8,"syntology":null},{"url":"/paper/utilizing-bert-for-aspect-based-sentiment","slug":"utilizing-bert-for-aspect-based-sentiment","title":"Utilizing BERT for Aspect-Based Sentiment Analysis via Constructing Auxiliary Sentence","date":"2019-03-22","arxiv_id":"1903.09588","repositories_listed":8,"syntology":{"n":27,"n_ran":19,"n_constructed":0,"n_ran_checked":14,"n_instrument":5,"n_unverified":8,"n_honours":3,"n_violates":0,"n_no_contract":11,"n_pointer_only":9,"phrase":"19 ran (of which 0 constructed an object rather than computing a result; 14 with no instrument failure: 3 honoured, 0 violated, 11 with no contract checked; 5 where Syntology's instrument failed) · 8 unverified","sample_list":"/paper/utilizing-bert-for-aspect-based-sentiment#ran","syntology_url":"https://syntology.ai/paper/1903.09588","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1903.09588"}},"official":{"repos":["HSLCY/ABSA-BERT-pair"],"state":"official (archive's flag): 8 ran","n_ran":8,"n_constructed":0,"n_ran_no_instrument_failure":6,"n_unverified":2,"ran_from_kinds":["listed","official"]}}},{"url":"/paper/abcnn-attention-based-convolutional-neural","slug":"abcnn-attention-based-convolutional-neural","title":"ABCNN: Attention-Based Convolutional Neural Network for Modeling Sentence Pairs","date":"2015-12-16","arxiv_id":"1512.05193","repositories_listed":8,"syntology":null},{"url":"/paper/palm-scaling-language-modeling-with-pathways-1","slug":"palm-scaling-language-modeling-with-pathways-1","title":"PaLM: Scaling Language Modeling with Pathways","date":"2022-04-05","arxiv_id":"2204.02311","repositories_listed":7,"syntology":{"n":37,"n_ran":32,"n_constructed":16,"n_ran_checked":24,"n_instrument":8,"n_unverified":5,"n_honours":2,"n_violates":1,"n_no_contract":21,"n_pointer_only":0,"phrase":"32 ran (of which 16 constructed an object rather than computing a result; 24 with no instrument failure: 2 honoured, 1 violated, 21 with no contract checked; 8 where Syntology's instrument failed) · 5 unverified","sample_list":"/paper/palm-scaling-language-modeling-with-pathways-1#ran","syntology_url":"https://syntology.ai/paper/2204.02311","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2204.02311"}},"official":null}},{"url":"/paper/i-bert-integer-only-bert-quantization","slug":"i-bert-integer-only-bert-quantization","title":"I-BERT: Integer-only BERT Quantization","date":"2021-01-05","arxiv_id":"2101.01321","repositories_listed":7,"syntology":null},{"url":"/paper/mobilebert-a-compact-task-agnostic-bert-for","slug":"mobilebert-a-compact-task-agnostic-bert-for","title":"MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices","date":"2020-04-06","arxiv_id":"2004.02984","repositories_listed":7,"syntology":null},{"url":"/paper/flaubert-unsupervised-language-model-pre","slug":"flaubert-unsupervised-language-model-pre","title":"FlauBERT: Unsupervised Language Model Pre-training for French","date":"2019-12-11","arxiv_id":"1912.05372","repositories_listed":7,"syntology":{"n":9,"n_ran":7,"n_constructed":0,"n_ran_checked":7,"n_instrument":0,"n_unverified":2,"n_honours":0,"n_violates":1,"n_no_contract":6,"n_pointer_only":1,"phrase":"7 ran (of which 0 constructed an object rather than computing a result; 7 with no instrument failure: 0 honoured, 1 violated, 6 with no contract checked; 0 where Syntology's instrument failed) · 2 unverified","sample_list":"/paper/flaubert-unsupervised-language-model-pre#ran","syntology_url":"https://syntology.ai/paper/1912.05372","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1912.05372"}},"official":{"repos":["getalp/Flaubert"],"state":"community repositories only","n_ran":0,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":0,"ran_from_kinds":["listed"]}}},{"url":"/paper/zen-pre-training-chinese-text-encoder","slug":"zen-pre-training-chinese-text-encoder","title":"ZEN: Pre-training Chinese Text Encoder Enhanced by N-gram Representations","date":"2019-11-02","arxiv_id":"1911.00720","repositories_listed":7,"syntology":{"n":35,"n_ran":28,"n_constructed":0,"n_ran_checked":22,"n_instrument":6,"n_unverified":7,"n_honours":1,"n_violates":0,"n_no_contract":21,"n_pointer_only":2,"phrase":"28 ran (of which 0 constructed an object rather than computing a result; 22 with no instrument failure: 1 honoured, 0 violated, 21 with no contract checked; 6 where Syntology's instrument failed) · 7 unverified","sample_list":"/paper/zen-pre-training-chinese-text-encoder#ran","syntology_url":"https://syntology.ai/paper/1911.00720","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1911.00720"}},"official":{"repos":["sinovation/ZEN"],"state":"official (archive's flag): 1 ran","n_ran":1,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":1,"ran_from_kinds":["listed","official"]}}},{"url":"/paper/is-bert-really-robust-natural-language-attack","slug":"is-bert-really-robust-natural-language-attack","title":"Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailment","date":"2019-07-27","arxiv_id":"1907.11932","repositories_listed":7,"syntology":{"n":2,"n_ran":2,"n_constructed":0,"n_ran_checked":0,"n_instrument":2,"n_unverified":0,"n_honours":0,"n_violates":0,"n_no_contract":0,"n_pointer_only":2,"phrase":"2 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified","sample_list":"/paper/is-bert-really-robust-natural-language-attack#ran","syntology_url":"https://syntology.ai/paper/1907.11932","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1907.11932"}},"official":{"repos":["jind11/TextFooler"],"state":"official: no sample here; runs from other or unrecorded repositories","n_ran":0,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":0,"ran_from_kinds":["unlocated"]}}},{"url":"/paper/multi-task-deep-neural-networks-for-natural","slug":"multi-task-deep-neural-networks-for-natural","title":"Multi-Task Deep Neural Networks for Natural Language Understanding","date":"2019-01-31","arxiv_id":"1901.11504","repositories_listed":7,"syntology":{"n":13,"n_ran":9,"n_constructed":0,"n_ran_checked":4,"n_instrument":5,"n_unverified":4,"n_honours":1,"n_violates":1,"n_no_contract":2,"n_pointer_only":4,"phrase":"9 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 1 honoured, 1 violated, 2 with no contract checked; 5 where Syntology's instrument failed) · 4 unverified","sample_list":"/paper/multi-task-deep-neural-networks-for-natural#ran","syntology_url":"https://syntology.ai/paper/1901.11504","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1901.11504"}},"official":{"repos":["namisan/mt-dnn"],"state":"community repositories only","n_ran":0,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":0,"ran_from_kinds":["listed"]}}},{"url":"/paper/reasoning-about-entailment-with-neural","slug":"reasoning-about-entailment-with-neural","title":"Reasoning about Entailment with Neural Attention","date":"2015-09-22","arxiv_id":"1509.06664","repositories_listed":7,"syntology":{"n":1,"n_ran":1,"n_constructed":0,"n_ran_checked":0,"n_instrument":1,"n_unverified":0,"n_honours":0,"n_violates":0,"n_no_contract":0,"n_pointer_only":1,"phrase":"1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified","sample_list":"/paper/reasoning-about-entailment-with-neural#ran","syntology_url":"https://syntology.ai/paper/1509.06664","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1509.06664"}},"official":null}},{"url":"/paper/squeezebert-what-can-computer-vision-teach","slug":"squeezebert-what-can-computer-vision-teach","title":"SqueezeBERT: What can computer vision teach NLP about efficient neural networks?","date":"2020-06-19","arxiv_id":"2006.11316","repositories_listed":6,"syntology":{"n":1,"n_ran":1,"n_constructed":0,"n_ran_checked":1,"n_instrument":0,"n_unverified":0,"n_honours":0,"n_violates":0,"n_no_contract":1,"n_pointer_only":0,"phrase":"1 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified","sample_list":"/paper/squeezebert-what-can-computer-vision-teach#ran","syntology_url":"https://syntology.ai/paper/2006.11316","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2006.11316"}},"official":{"repos":["huggingface/transformers"],"state":"community repositories only","n_ran":0,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":0,"ran_from_kinds":["listed"]}}},{"url":"/paper/exploiting-cloze-questions-for-few-shot-text","slug":"exploiting-cloze-questions-for-few-shot-text","title":"Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference","date":"2020-01-21","arxiv_id":"2001.07676","repositories_listed":6,"syntology":null},{"url":"/paper/smart-robust-and-efficient-fine-tuning-for","slug":"smart-robust-and-efficient-fine-tuning-for","title":"SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimization","date":"2019-11-08","arxiv_id":"1911.03437","repositories_listed":6,"syntology":{"n":8,"n_ran":8,"n_constructed":0,"n_ran_checked":5,"n_instrument":3,"n_unverified":0,"n_honours":2,"n_violates":1,"n_no_contract":2,"n_pointer_only":1,"phrase":"8 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 2 honoured, 1 violated, 2 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified","sample_list":"/paper/smart-robust-and-efficient-fine-tuning-for#ran","syntology_url":"https://syntology.ai/paper/1911.03437","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1911.03437"}},"official":{"repos":["namisan/mt-dnn"],"state":"official: no sample here; runs from other or unrecorded repositories","n_ran":0,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":0,"ran_from_kinds":["listed","unlocated"]}}},{"url":"/paper/supervised-multimodal-bitransformers-for","slug":"supervised-multimodal-bitransformers-for","title":"Supervised Multimodal Bitransformers for Classifying Images and Text","date":"2019-09-06","arxiv_id":"1909.02950","repositories_listed":6,"syntology":null},{"url":"/paper/spanbert-improving-pre-training-by","slug":"spanbert-improving-pre-training-by","title":"SpanBERT: Improving Pre-training by Representing and Predicting Spans","date":"2019-07-24","arxiv_id":"1907.10529","repositories_listed":6,"syntology":{"n":15,"n_ran":9,"n_constructed":0,"n_ran_checked":5,"n_instrument":4,"n_unverified":6,"n_honours":0,"n_violates":0,"n_no_contract":5,"n_pointer_only":6,"phrase":"9 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 0 honoured, 0 violated, 5 with no contract checked; 4 where Syntology's instrument failed) · 6 unverified","sample_list":"/paper/spanbert-improving-pre-training-by#ran","syntology_url":"https://syntology.ai/paper/1907.10529","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1907.10529"}},"official":{"repos":["facebookresearch/SpanBERT"],"state":"official (archive's flag): 2 ran","n_ran":2,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":1,"ran_from_kinds":["listed","official"]}}},{"url":"/paper/right-for-the-wrong-reasons-diagnosing","slug":"right-for-the-wrong-reasons-diagnosing","title":"Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference","date":"2019-02-04","arxiv_id":"1902.01007","repositories_listed":6,"syntology":{"n":1,"n_ran":1,"n_constructed":0,"n_ran_checked":0,"n_instrument":1,"n_unverified":0,"n_honours":0,"n_violates":0,"n_no_contract":0,"n_pointer_only":0,"phrase":"1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified","sample_list":"/paper/right-for-the-wrong-reasons-diagnosing#ran","syntology_url":"https://syntology.ai/paper/1902.01007","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1902.01007"}},"official":{"repos":["tommccoy1/hans"],"state":"official (archive's flag): 1 ran","n_ran":1,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":0,"ran_from_kinds":["official"]}}},{"url":"/paper/the-natural-language-decathlon-multitask","slug":"the-natural-language-decathlon-multitask","title":"The Natural Language Decathlon: Multitask Learning as Question Answering","date":"2018-06-20","arxiv_id":"1806.08730","repositories_listed":6,"syntology":{"n":13,"n_ran":13,"n_constructed":0,"n_ran_checked":12,"n_instrument":1,"n_unverified":0,"n_honours":0,"n_violates":0,"n_no_contract":12,"n_pointer_only":4,"phrase":"13 ran (of which 0 constructed an object rather than computing a result; 12 with no instrument failure: 0 honoured, 0 violated, 12 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified","sample_list":"/paper/the-natural-language-decathlon-multitask#ran","syntology_url":"https://syntology.ai/paper/1806.08730","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1806.08730"}},"official":{"repos":["salesforce/decaNLP"],"state":"official (archive's flag): 9 ran","n_ran":9,"n_constructed":0,"n_ran_no_instrument_failure":9,"n_unverified":0,"ran_from_kinds":["listed","official"]}}},{"url":"/paper/pangu-a-large-scale-autoregressive-pretrained","slug":"pangu-a-large-scale-autoregressive-pretrained","title":"PanGu-$α$: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computation","date":"2021-04-26","arxiv_id":"2104.12369","repositories_listed":5,"syntology":null},{"url":"/paper/informer-transformer-likes-informed-attention","slug":"informer-transformer-likes-informed-attention","title":"RealFormer: Transformer Likes Residual Attention","date":"2020-12-21","arxiv_id":"2012.11747","repositories_listed":5,"syntology":null},{"url":"/paper/autoprompt-eliciting-knowledge-from-language","slug":"autoprompt-eliciting-knowledge-from-language","title":"AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts","date":"2020-10-29","arxiv_id":"2010.15980","repositories_listed":5,"syntology":{"n":3,"n_ran":3,"n_constructed":0,"n_ran_checked":1,"n_instrument":2,"n_unverified":0,"n_honours":1,"n_violates":0,"n_no_contract":0,"n_pointer_only":1,"phrase":"3 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 1 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified","sample_list":"/paper/autoprompt-eliciting-knowledge-from-language#ran","syntology_url":"https://syntology.ai/paper/2010.15980","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2010.15980"}},"official":{"repos":["ucinlp/autoprompt"],"state":"official (archive's flag): 2 ran","n_ran":2,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":0,"ran_from_kinds":["listed","official"]}}},{"url":"/paper/q8bert-quantized-8bit-bert","slug":"q8bert-quantized-8bit-bert","title":"Q8BERT: Quantized 8Bit BERT","date":"2019-10-14","arxiv_id":"1910.06188","repositories_listed":5,"syntology":{"n":11,"n_ran":10,"n_constructed":0,"n_ran_checked":9,"n_instrument":1,"n_unverified":1,"n_honours":2,"n_violates":0,"n_no_contract":7,"n_pointer_only":3,"phrase":"10 ran (of which 0 constructed an object rather than computing a result; 9 with no instrument failure: 2 honoured, 0 violated, 7 with no contract checked; 1 where Syntology's instrument failed) · 1 unverified","sample_list":"/paper/q8bert-quantized-8bit-bert#ran","syntology_url":"https://syntology.ai/paper/1910.06188","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1910.06188"}},"official":{"repos":["intellabs/model-compression-research-package","NervanaSystems/nlp-architect"],"state":"official (archive's flag): 5 ran","n_ran":5,"n_constructed":0,"n_ran_no_instrument_failure":5,"n_unverified":0,"ran_from_kinds":["listed","official","unlocated"]}}},{"url":"/paper/temporal-and-aspectual-entailment","slug":"temporal-and-aspectual-entailment","title":"Temporal and Aspectual Entailment","date":"2019-04-02","arxiv_id":"1904.01297","repositories_listed":5,"syntology":null},{"url":"/paper/generating-natural-language-adversarial","slug":"generating-natural-language-adversarial","title":"Generating Natural Language Adversarial Examples","date":"2018-04-21","arxiv_id":"1804.07998","repositories_listed":5,"syntology":{"n":5,"n_ran":4,"n_constructed":0,"n_ran_checked":4,"n_instrument":0,"n_unverified":1,"n_honours":0,"n_violates":0,"n_no_contract":4,"n_pointer_only":0,"phrase":"4 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 0 honoured, 0 violated, 4 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified","sample_list":"/paper/generating-natural-language-adversarial#ran","syntology_url":"https://syntology.ai/paper/1804.07998","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1804.07998"}},"official":{"repos":["nesl/nlp_adversarial_examples"],"state":"official (archive's flag): 4 ran","n_ran":4,"n_constructed":0,"n_ran_no_instrument_failure":4,"n_unverified":1,"ran_from_kinds":["official"]}}},{"url":"/paper/machine-comprehension-using-match-lstm-and","slug":"machine-comprehension-using-match-lstm-and","title":"Machine Comprehension Using Match-LSTM and Answer Pointer","date":"2016-08-29","arxiv_id":"1608.07905","repositories_listed":5,"syntology":{"n":3,"n_ran":3,"n_constructed":0,"n_ran_checked":2,"n_instrument":1,"n_unverified":0,"n_honours":1,"n_violates":1,"n_no_contract":0,"n_pointer_only":3,"phrase":"3 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 1 honoured, 1 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified","sample_list":"/paper/machine-comprehension-using-match-lstm-and#ran","syntology_url":"https://syntology.ai/paper/1608.07905","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1608.07905"}},"official":{"repos":["shuohangwang/SeqMatchSeq"],"state":"official: no sample here; runs from other or unrecorded repositories","n_ran":0,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":0,"ran_from_kinds":["unlocated"]}}},{"url":"/paper/llm-int8-8-bit-matrix-multiplication-for","slug":"llm-int8-8-bit-matrix-multiplication-for","title":"LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale","date":"2022-08-15","arxiv_id":"2208.07339","repositories_listed":4,"syntology":{"n":5,"n_ran":4,"n_constructed":0,"n_ran_checked":4,"n_instrument":0,"n_unverified":1,"n_honours":0,"n_violates":0,"n_no_contract":4,"n_pointer_only":2,"phrase":"4 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 0 honoured, 0 violated, 4 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified","sample_list":"/paper/llm-int8-8-bit-matrix-multiplication-for#ran","syntology_url":"https://syntology.ai/paper/2208.07339","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2208.07339"}},"official":{"repos":["timdettmers/bitsandbytes"],"state":"official: harvested, nothing ran","n_ran":0,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":1,"ran_from_kinds":["listed"]}}},{"url":"/paper/klue-korean-language-understanding-evaluation","slug":"klue-korean-language-understanding-evaluation","title":"KLUE: Korean Language Understanding Evaluation","date":"2021-05-20","arxiv_id":"2105.09680","repositories_listed":4,"syntology":null},{"url":"/paper/how-to-train-bert-with-an-academic-budget","slug":"how-to-train-bert-with-an-academic-budget","title":"How to Train BERT with an Academic Budget","date":"2021-04-15","arxiv_id":"2104.07705","repositories_listed":4,"syntology":null},{"url":"/paper/evaluating-the-factual-consistency-of","slug":"evaluating-the-factual-consistency-of","title":"Evaluating the Factual Consistency of Abstractive Text Summarization","date":"2019-10-28","arxiv_id":"1910.12840","repositories_listed":4,"syntology":{"n":3,"n_ran":3,"n_constructed":0,"n_ran_checked":1,"n_instrument":2,"n_unverified":0,"n_honours":1,"n_violates":0,"n_no_contract":0,"n_pointer_only":0,"phrase":"3 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 1 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified","sample_list":"/paper/evaluating-the-factual-consistency-of#ran","syntology_url":"https://syntology.ai/paper/1910.12840","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1910.12840"}},"official":{"repos":["yuhui-zh15/FactCCX"],"state":"official (archive's flag): 2 ran","n_ran":2,"n_constructed":0,"n_ran_no_instrument_failure":1,"n_unverified":0,"ran_from_kinds":["listed","official"]}}},{"url":"/paper/benchmarking-zero-shot-text-classification","slug":"benchmarking-zero-shot-text-classification","title":"Benchmarking Zero-shot Text Classification: Datasets, Evaluation and Entailment Approach","date":"2019-08-31","arxiv_id":"1909.00161","repositories_listed":4,"syntology":{"n":6,"n_ran":3,"n_constructed":0,"n_ran_checked":1,"n_instrument":2,"n_unverified":3,"n_honours":0,"n_violates":0,"n_no_contract":1,"n_pointer_only":3,"phrase":"3 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 2 where Syntology's instrument failed) · 3 unverified","sample_list":"/paper/benchmarking-zero-shot-text-classification#ran","syntology_url":"https://syntology.ai/paper/1909.00161","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1909.00161"}},"official":{"repos":["yinwenpeng/BenchmarkingZeroShot"],"state":"official (archive's flag): 1 ran","n_ran":1,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":0,"ran_from_kinds":["listed","official","unlocated"]}}},{"url":"/paper/transfer-learning-in-biomedical-natural","slug":"transfer-learning-in-biomedical-natural","title":"Transfer Learning in Biomedical Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasets","date":"2019-06-13","arxiv_id":"1906.05474","repositories_listed":4,"syntology":{"n":2,"n_ran":2,"n_constructed":0,"n_ran_checked":2,"n_instrument":0,"n_unverified":0,"n_honours":0,"n_violates":0,"n_no_contract":2,"n_pointer_only":0,"phrase":"2 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified","sample_list":"/paper/transfer-learning-in-biomedical-natural#ran","syntology_url":"https://syntology.ai/paper/1906.05474","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1906.05474"}},"official":{"repos":["ncbi-nlp/BLUE_Benchmark","ncbi-nlp/NCBI_BERT"],"state":"community repositories only","n_ran":0,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":0,"ran_from_kinds":["listed"]}}},{"url":"/paper/distilling-task-specific-knowledge-from-bert","slug":"distilling-task-specific-knowledge-from-bert","title":"Distilling Task-Specific Knowledge from BERT into Simple Neural Networks","date":"2019-03-28","arxiv_id":"1903.12136","repositories_listed":4,"syntology":{"n":9,"n_ran":9,"n_constructed":0,"n_ran_checked":3,"n_instrument":6,"n_unverified":0,"n_honours":3,"n_violates":0,"n_no_contract":0,"n_pointer_only":0,"phrase":"9 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 3 honoured, 0 violated, 0 with no contract checked; 6 where Syntology's instrument failed) · 0 unverified","sample_list":"/paper/distilling-task-specific-knowledge-from-bert#ran","syntology_url":"https://syntology.ai/paper/1903.12136","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1903.12136"}},"official":null}},{"url":"/paper/learning-general-purpose-distributed-sentence","slug":"learning-general-purpose-distributed-sentence","title":"Learning General Purpose Distributed Sentence Representations via Large Scale Multi-task Learning","date":"2018-03-30","arxiv_id":"1804.00079","repositories_listed":4,"syntology":{"n":3,"n_ran":3,"n_constructed":0,"n_ran_checked":1,"n_instrument":2,"n_unverified":0,"n_honours":1,"n_violates":0,"n_no_contract":0,"n_pointer_only":3,"phrase":"3 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 1 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified","sample_list":"/paper/learning-general-purpose-distributed-sentence#ran","syntology_url":"https://syntology.ai/paper/1804.00079","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1804.00079"}},"official":{"repos":["facebookresearch/SentEval","Maluuba/gensen"],"state":"official (archive's flag): 3 ran","n_ran":3,"n_constructed":0,"n_ran_no_instrument_failure":1,"n_unverified":0,"ran_from_kinds":["official"]}}},{"url":"/paper/a-broad-coverage-challenge-corpus-for","slug":"a-broad-coverage-challenge-corpus-for","title":"A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference","date":"2017-04-18","arxiv_id":"1704.05426","repositories_listed":4,"syntology":null},{"url":"/paper/learning-natural-language-inference-with-lstm","slug":"learning-natural-language-inference-with-lstm","title":"Learning Natural Language Inference with LSTM","date":"2015-12-30","arxiv_id":"1512.08849","repositories_listed":4,"syntology":{"n":1,"n_ran":1,"n_constructed":0,"n_ran_checked":0,"n_instrument":1,"n_unverified":0,"n_honours":0,"n_violates":0,"n_no_contract":0,"n_pointer_only":1,"phrase":"1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified","sample_list":"/paper/learning-natural-language-inference-with-lstm#ran","syntology_url":"https://syntology.ai/paper/1512.08849","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1512.08849"}},"official":{"repos":["shuohangwang/SeqMatchSeq"],"state":"community repositories only","n_ran":0,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":0,"ran_from_kinds":["listed"]}}},{"url":"/paper/modulora-finetuning-3-bit-llms-on-consumer","slug":"modulora-finetuning-3-bit-llms-on-consumer","title":"ModuLoRA: Finetuning 2-Bit LLMs on Consumer GPUs by Integrating with Modular Quantizers","date":"2023-09-28","arxiv_id":"2309.16119","repositories_listed":3,"syntology":{"n":9,"n_ran":5,"n_constructed":0,"n_ran_checked":3,"n_instrument":2,"n_unverified":4,"n_honours":0,"n_violates":0,"n_no_contract":3,"n_pointer_only":8,"phrase":"5 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 0 violated, 3 with no contract checked; 2 where Syntology's instrument failed) · 4 unverified","sample_list":"/paper/modulora-finetuning-3-bit-llms-on-consumer#ran","syntology_url":"https://syntology.ai/paper/2309.16119","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2309.16119"}},"official":{"repos":["kuleshov-group/llmtools","kuleshov-group/modulora-experiment"],"state":"official (archive's flag): 5 ran","n_ran":5,"n_constructed":0,"n_ran_no_instrument_failure":3,"n_unverified":4,"ran_from_kinds":["official"]}}},{"url":"/paper/xlm-v-overcoming-the-vocabulary-bottleneck-in","slug":"xlm-v-overcoming-the-vocabulary-bottleneck-in","title":"XLM-V: Overcoming the Vocabulary Bottleneck in Multilingual Masked Language Models","date":"2023-01-25","arxiv_id":"2301.10472","repositories_listed":3,"syntology":null},{"url":"/paper/hungry-hungry-hippos-towards-language","slug":"hungry-hungry-hippos-towards-language","title":"Hungry Hungry Hippos: Towards Language Modeling with State Space Models","date":"2022-12-28","arxiv_id":"2212.14052","repositories_listed":3,"syntology":{"n":15,"n_ran":7,"n_constructed":0,"n_ran_checked":3,"n_instrument":4,"n_unverified":8,"n_honours":1,"n_violates":0,"n_no_contract":2,"n_pointer_only":0,"phrase":"7 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 1 honoured, 0 violated, 2 with no contract checked; 4 where Syntology's instrument failed) · 8 unverified","sample_list":"/paper/hungry-hungry-hippos-towards-language#ran","syntology_url":"https://syntology.ai/paper/2212.14052","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2212.14052"}},"official":{"repos":["hazyresearch/h3"],"state":"official (archive's flag): 1 ran","n_ran":1,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":0,"ran_from_kinds":["listed","official"]}}},{"url":"/paper/ask-me-anything-a-simple-strategy-for","slug":"ask-me-anything-a-simple-strategy-for","title":"Ask Me Anything: A simple strategy for prompting language models","date":"2022-10-05","arxiv_id":"2210.02441","repositories_listed":3,"syntology":{"n":2,"n_ran":2,"n_constructed":0,"n_ran_checked":0,"n_instrument":2,"n_unverified":0,"n_honours":0,"n_violates":0,"n_no_contract":0,"n_pointer_only":0,"phrase":"2 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified","sample_list":"/paper/ask-me-anything-a-simple-strategy-for#ran","syntology_url":"https://syntology.ai/paper/2210.02441","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2210.02441"}},"official":{"repos":["hazyresearch/ama_prompting"],"state":"official (archive's flag): 2 ran","n_ran":2,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":0,"ran_from_kinds":["official"]}}},{"url":"/paper/designing-effective-sparse-expert-models","slug":"designing-effective-sparse-expert-models","title":"ST-MoE: Designing Stable and Transferable Sparse Expert Models","date":"2022-02-17","arxiv_id":"2202.08906","repositories_listed":3,"syntology":{"n":5,"n_ran":5,"n_constructed":0,"n_ran_checked":3,"n_instrument":2,"n_unverified":0,"n_honours":3,"n_violates":0,"n_no_contract":0,"n_pointer_only":5,"phrase":"5 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 3 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified","sample_list":"/paper/designing-effective-sparse-expert-models#ran","syntology_url":"https://syntology.ai/paper/2202.08906","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2202.08906"}},"official":{"repos":["tensorflow/mesh"],"state":"community repositories only","n_ran":0,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":0,"ran_from_kinds":["listed"]}}},{"url":"/paper/zerogen-efficient-zero-shot-learning-via","slug":"zerogen-efficient-zero-shot-learning-via","title":"ZeroGen: Efficient Zero-shot Learning via Dataset Generation","date":"2022-02-16","arxiv_id":"2202.07922","repositories_listed":3,"syntology":{"n":7,"n_ran":3,"n_constructed":0,"n_ran_checked":0,"n_instrument":3,"n_unverified":4,"n_honours":0,"n_violates":0,"n_no_contract":0,"n_pointer_only":7,"phrase":"3 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 4 unverified","sample_list":"/paper/zerogen-efficient-zero-shot-learning-via#ran","syntology_url":"https://syntology.ai/paper/2202.07922","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2202.07922"}},"official":{"repos":["HKUNLP/zerogen"],"state":"official (archive's flag): 3 ran","n_ran":3,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":4,"ran_from_kinds":["official"]}}},{"url":"/paper/debertav3-improving-deberta-using-electra","slug":"debertav3-improving-deberta-using-electra","title":"DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing","date":"2021-11-18","arxiv_id":"2111.09543","repositories_listed":3,"syntology":{"n":7,"n_ran":6,"n_constructed":0,"n_ran_checked":6,"n_instrument":0,"n_unverified":1,"n_honours":0,"n_violates":0,"n_no_contract":6,"n_pointer_only":0,"phrase":"6 ran (of which 0 constructed an object rather than computing a result; 6 with no instrument failure: 0 honoured, 0 violated, 6 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified","sample_list":"/paper/debertav3-improving-deberta-using-electra#ran","syntology_url":"https://syntology.ai/paper/2111.09543","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2111.09543"}},"official":{"repos":["microsoft/DeBERTa"],"state":"official (archive's flag): 6 ran","n_ran":6,"n_constructed":0,"n_ran_no_instrument_failure":6,"n_unverified":1,"ran_from_kinds":["official"]}}},{"url":"/paper/summac-re-visiting-nli-based-models-for","slug":"summac-re-visiting-nli-based-models-for","title":"SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization","date":"2021-11-18","arxiv_id":"2111.09525","repositories_listed":3,"syntology":{"n":23,"n_ran":12,"n_constructed":0,"n_ran_checked":10,"n_instrument":2,"n_unverified":11,"n_honours":1,"n_violates":0,"n_no_contract":9,"n_pointer_only":0,"phrase":"12 ran (of which 0 constructed an object rather than computing a result; 10 with no instrument failure: 1 honoured, 0 violated, 9 with no contract checked; 2 where Syntology's instrument failed) · 11 unverified","sample_list":"/paper/summac-re-visiting-nli-based-models-for#ran","syntology_url":"https://syntology.ai/paper/2111.09525","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2111.09525"}},"official":{"repos":["tingofurro/summac"],"state":"official (archive's flag): 3 ran","n_ran":3,"n_constructed":0,"n_ran_no_instrument_failure":1,"n_unverified":0,"ran_from_kinds":["listed","official"]}}},{"url":"/paper/chinesebert-chinese-pretraining-enhanced-by","slug":"chinesebert-chinese-pretraining-enhanced-by","title":"ChineseBERT: Chinese Pretraining Enhanced by Glyph and Pinyin Information","date":"2021-06-30","arxiv_id":"2106.16038","repositories_listed":3,"syntology":null},{"url":"/paper/entailment-as-few-shot-learner","slug":"entailment-as-few-shot-learner","title":"Entailment as Few-Shot Learner","date":"2021-04-29","arxiv_id":"2104.14690","repositories_listed":3,"syntology":{"n":3,"n_ran":1,"n_constructed":0,"n_ran_checked":1,"n_instrument":0,"n_unverified":2,"n_honours":0,"n_violates":1,"n_no_contract":0,"n_pointer_only":0,"phrase":"1 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 1 violated, 0 with no contract checked; 0 where Syntology's instrument failed) · 2 unverified","sample_list":"/paper/entailment-as-few-shot-learner#ran","syntology_url":"https://syntology.ai/paper/2104.14690","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2104.14690"}},"official":null}},{"url":"/paper/beyond-fully-connected-layers-with","slug":"beyond-fully-connected-layers-with","title":"Beyond Fully-Connected Layers with Quaternions: Parameterization of Hypercomplex Multiplications with $1/n$ Parameters","date":"2021-02-17","arxiv_id":"2102.08597","repositories_listed":3,"syntology":{"n":1,"n_ran":0,"n_constructed":0,"n_ran_checked":0,"n_instrument":0,"n_unverified":1,"n_honours":0,"n_violates":0,"n_no_contract":0,"n_pointer_only":0,"phrase":"0 ran · 1 unverified","sample_list":"/paper/beyond-fully-connected-layers-with#ran","syntology_url":"https://syntology.ai/paper/2102.08597","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2102.08597"}},"official":{"repos":["astonzhang/Parameterization-of-Hypercomplex-Multiplications"],"state":"official: harvested, nothing ran","n_ran":0,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":1,"ran_from_kinds":[]}}},{"url":"/paper/improving-factual-completeness-and","slug":"improving-factual-completeness-and","title":"Improving Factual Completeness and Consistency of Image-to-Text Radiology Report Generation","date":"2020-10-20","arxiv_id":"2010.10042","repositories_listed":3,"syntology":{"n":3,"n_ran":3,"n_constructed":0,"n_ran_checked":0,"n_instrument":3,"n_unverified":0,"n_honours":0,"n_violates":0,"n_no_contract":0,"n_pointer_only":0,"phrase":"3 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified","sample_list":"/paper/improving-factual-completeness-and#ran","syntology_url":"https://syntology.ai/paper/2010.10042","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2010.10042"}},"official":{"repos":["ysmiura/ifcc"],"state":"official (archive's flag): 2 ran","n_ran":2,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":0,"ran_from_kinds":["listed","official"]}}},{"url":"/paper/adversarial-training-for-large-neural","slug":"adversarial-training-for-large-neural","title":"Adversarial Training for Large Neural Language Models","date":"2020-04-20","arxiv_id":"2004.08994","repositories_listed":3,"syntology":null},{"url":"/paper/e-snli-ve-2-0-corrected-visual-textual","slug":"e-snli-ve-2-0-corrected-visual-textual","title":"e-SNLI-VE: Corrected Visual-Textual Entailment with Natural Language Explanations","date":"2020-04-07","arxiv_id":"2004.03744","repositories_listed":3,"syntology":{"n":3,"n_ran":3,"n_constructed":0,"n_ran_checked":1,"n_instrument":2,"n_unverified":0,"n_honours":0,"n_violates":1,"n_no_contract":0,"n_pointer_only":2,"phrase":"3 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 1 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified","sample_list":"/paper/e-snli-ve-2-0-corrected-visual-textual#ran","syntology_url":"https://syntology.ai/paper/2004.03744","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2004.03744"}},"official":{"repos":["virginie-do/e-SNLI-VE"],"state":"official (archive's flag): 2 ran","n_ran":2,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":0,"ran_from_kinds":["listed","official"]}}},{"url":"/paper/kornli-and-korsts-new-benchmark-datasets-for","slug":"kornli-and-korsts-new-benchmark-datasets-for","title":"KorNLI and KorSTS: New Benchmark Datasets for Korean Natural Language Understanding","date":"2020-04-07","arxiv_id":"2004.03289","repositories_listed":3,"syntology":null},{"url":"/paper/semantic-relatedness-based-re-ranker-for-text","slug":"semantic-relatedness-based-re-ranker-for-text","title":"Semantic Relatedness Based Re-ranker for Text Spotting","date":"2019-09-17","arxiv_id":"1909.07950","repositories_listed":3,"syntology":null},{"url":"/paper/probing-natural-language-inference-models","slug":"probing-natural-language-inference-models","title":"Probing Natural Language Inference Models through Semantic Fragments","date":"2019-09-16","arxiv_id":"1909.07521","repositories_listed":3,"syntology":null},{"url":"/paper/dont-take-the-easy-way-out-ensemble-based","slug":"dont-take-the-easy-way-out-ensemble-based","title":"Don't Take the Easy Way Out: Ensemble Based Methods for Avoiding Known Dataset Biases","date":"2019-09-09","arxiv_id":"1909.03683","repositories_listed":3,"syntology":{"n":6,"n_ran":4,"n_constructed":0,"n_ran_checked":1,"n_instrument":3,"n_unverified":2,"n_honours":0,"n_violates":0,"n_no_contract":1,"n_pointer_only":3,"phrase":"4 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 3 where Syntology's instrument failed) · 2 unverified","sample_list":"/paper/dont-take-the-easy-way-out-ensemble-based#ran","syntology_url":"https://syntology.ai/paper/1909.03683","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1909.03683"}},"official":{"repos":["chrisc36/debias"],"state":"official (archive's flag): 1 ran","n_ran":1,"n_constructed":0,"n_ran_no_instrument_failure":1,"n_unverified":2,"ran_from_kinds":["listed","official"]}}},{"url":"/paper/simple-and-effective-text-matching-with-1","slug":"simple-and-effective-text-matching-with-1","title":"Simple and Effective Text Matching with Richer Alignment Features","date":"2019-08-01","arxiv_id":"1908.00300","repositories_listed":3,"syntology":{"n":1,"n_ran":1,"n_constructed":0,"n_ran_checked":1,"n_instrument":0,"n_unverified":0,"n_honours":1,"n_violates":0,"n_no_contract":0,"n_pointer_only":1,"phrase":"1 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 1 honoured, 0 violated, 0 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified","sample_list":"/paper/simple-and-effective-text-matching-with-1#ran","syntology_url":"https://syntology.ai/paper/1908.00300","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1908.00300"}},"official":{"repos":["hitvoice/RE2"],"state":"community repositories only","n_ran":0,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":0,"ran_from_kinds":["listed"]}}},{"url":"/paper/ernie-20-a-continual-pre-training-framework","slug":"ernie-20-a-continual-pre-training-framework","title":"ERNIE 2.0: A Continual Pre-training Framework for Language Understanding","date":"2019-07-29","arxiv_id":"1907.12412","repositories_listed":3,"syntology":{"n":1,"n_ran":0,"n_constructed":0,"n_ran_checked":0,"n_instrument":0,"n_unverified":1,"n_honours":0,"n_violates":0,"n_no_contract":0,"n_pointer_only":1,"phrase":"0 ran · 1 unverified","sample_list":"/paper/ernie-20-a-continual-pre-training-framework#ran","syntology_url":"https://syntology.ai/paper/1907.12412","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1907.12412"}},"official":{"repos":["PaddlePaddle/ERNIE"],"state":"official: harvested, nothing ran","n_ran":0,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":1,"ran_from_kinds":[]}}},{"url":"/paper/a-hybrid-neural-network-model-for-commonsense","slug":"a-hybrid-neural-network-model-for-commonsense","title":"A Hybrid Neural Network Model for Commonsense Reasoning","date":"2019-07-27","arxiv_id":"1907.11983","repositories_listed":3,"syntology":{"n":1,"n_ran":0,"n_constructed":0,"n_ran_checked":0,"n_instrument":0,"n_unverified":1,"n_honours":0,"n_violates":0,"n_no_contract":0,"n_pointer_only":0,"phrase":"0 ran · 1 unverified","sample_list":"/paper/a-hybrid-neural-network-model-for-commonsense#ran","syntology_url":"https://syntology.ai/paper/1907.11983","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1907.11983"}},"official":{"repos":["namisan/mt-dnn"],"state":"official: harvested for another paper","n_ran":0,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":0,"ran_from_kinds":[]}}},{"url":"/paper/improving-multi-task-deep-neural-networks-via","slug":"improving-multi-task-deep-neural-networks-via","title":"Improving Multi-Task Deep Neural Networks via Knowledge Distillation for Natural Language Understanding","date":"2019-04-20","arxiv_id":"1904.09482","repositories_listed":3,"syntology":null},{"url":"/paper/linspector-multilingual-probing-tasks-for","slug":"linspector-multilingual-probing-tasks-for","title":"LINSPECTOR: Multilingual Probing Tasks for Word Representations","date":"2019-03-22","arxiv_id":"1903.09442","repositories_listed":3,"syntology":null},{"url":"/paper/combining-similarity-features-and-deep","slug":"combining-similarity-features-and-deep","title":"Combining Similarity Features and Deep Representation Learning for Stance Detection in the Context of Checking Fake News","date":"2018-11-02","arxiv_id":"1811.00706","repositories_listed":3,"syntology":null},{"url":"/paper/simplifying-neural-machine-translation-with","slug":"simplifying-neural-machine-translation-with","title":"Simplifying Neural Machine Translation with Addition-Subtraction Twin-Gated Recurrent Networks","date":"2018-10-30","arxiv_id":"1810.12546","repositories_listed":3,"syntology":null},{"url":"/paper/lessons-from-natural-language-inference-in","slug":"lessons-from-natural-language-inference-in","title":"Lessons from Natural Language Inference in the Clinical Domain","date":"2018-08-21","arxiv_id":"1808.06752","repositories_listed":3,"syntology":null},{"url":"/paper/hyperbolic-neural-networks","slug":"hyperbolic-neural-networks","title":"Hyperbolic Neural Networks","date":"2018-05-23","arxiv_id":"1805.09112","repositories_listed":3,"syntology":{"n":5,"n_ran":4,"n_constructed":0,"n_ran_checked":3,"n_instrument":1,"n_unverified":1,"n_honours":0,"n_violates":3,"n_no_contract":0,"n_pointer_only":3,"phrase":"4 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 3 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 1 unverified","sample_list":"/paper/hyperbolic-neural-networks#ran","syntology_url":"https://syntology.ai/paper/1805.09112","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1805.09112"}},"official":{"repos":["dalab/hyperbolic_nn"],"state":"official (archive's flag): 2 ran","n_ran":2,"n_constructed":0,"n_ran_no_instrument_failure":2,"n_unverified":0,"ran_from_kinds":["listed","official"]}}},{"url":"/paper/stochastic-answer-networks-for-natural","slug":"stochastic-answer-networks-for-natural","title":"Stochastic Answer Networks for Natural Language Inference","date":"2018-04-21","arxiv_id":"1804.07888","repositories_listed":3,"syntology":null},{"url":"/paper/sparsemap-differentiable-sparse-structured","slug":"sparsemap-differentiable-sparse-structured","title":"SparseMAP: Differentiable Sparse Structured Inference","date":"2018-02-12","arxiv_id":"1802.04223","repositories_listed":3,"syntology":{"n":2,"n_ran":2,"n_constructed":0,"n_ran_checked":0,"n_instrument":2,"n_unverified":0,"n_honours":0,"n_violates":0,"n_no_contract":0,"n_pointer_only":2,"phrase":"2 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified","sample_list":"/paper/sparsemap-differentiable-sparse-structured#ran","syntology_url":"https://syntology.ai/paper/1802.04223","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1802.04223"}},"official":{"repos":["vene/sparsemap"],"state":"community repositories only","n_ran":0,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":0,"ran_from_kinds":["listed"]}}},{"url":"/paper/disan-directional-self-attention-network-for","slug":"disan-directional-self-attention-network-for","title":"DiSAN: Directional Self-Attention Network for RNN/CNN-Free Language Understanding","date":"2017-09-14","arxiv_id":"1709.04696","repositories_listed":3,"syntology":null}],"syntology_records":70,"syntology_note":"a paper without a record is not a recorded non-run: it may lack an arXiv id or simply be absent from the graph layer"},"key_notes":{"n_ran_checked":"legacy name, kept unchanged so existing readers do not break: it counts the samples that ran with no instrument failure (honoured, violated, and ran with no contract checked); it does not mean a contract was checked, and the pages print it as 'K with no instrument failure', not 'K checked'","n_constructed":"a sub-count of the samples that ran, never subtracted from them and never a failure: an executed sample whose run returned an instance of its own class (fixture_out_type equals the entry name): the run built an object and did not compute a result (Syntology's RAN record, counts.constructed)"},"since_archive_judged":{"papers":41,"papers_placed":1,"judged_by":{"model":"Claude Sonnet 4.5","models_as_recorded":["global.anthropic.claude-sonnet-4-5-20250929-v1:0"],"from":"lb_rows_run_verified.jsonl: each record's `model` field (the extractor's call that wrote decisions[])"},"note":"a model's judgement from the paper's own tables that they report on this leaderboard, not a result: no value from a paper that was not placed is printed or carried here; papers counts each paper once over these leaderboards","backfill":"8,886 of 18,637 newer papers checked so far (733 more had nothing to place; 41 could not be read)","backfill_complete":false,"leaderboards":[{"leaderboard":"/sota/natural-language-inference-on-snli","since_archive":"/sota/natural-language-inference-on-snli#since-judged","papers_judged":4,"placed":0},{"leaderboard":"/sota/natural-language-inference-on-rte","since_archive":"/sota/natural-language-inference-on-rte#since-judged","papers_judged":28,"placed":1},{"leaderboard":"/sota/natural-language-inference-on-multinli","since_archive":"/sota/natural-language-inference-on-multinli#since-judged","papers_judged":18,"placed":0},{"leaderboard":"/sota/natural-language-inference-on-qnli","since_archive":"/sota/natural-language-inference-on-qnli#since-judged","papers_judged":21,"placed":1},{"leaderboard":"/sota/natural-language-inference-on-wnli","since_archive":"/sota/natural-language-inference-on-wnli#since-judged","papers_judged":1,"placed":0},{"leaderboard":"/sota/natural-language-inference-on-scitail","since_archive":"/sota/natural-language-inference-on-scitail#since-judged","papers_judged":1,"placed":0},{"leaderboard":"/sota/natural-language-inference-on-multinli-dev","since_archive":"/sota/natural-language-inference-on-multinli-dev#since-judged","papers_judged":3,"placed":0},{"leaderboard":"/sota/natural-language-inference-on-v-snli","since_archive":"/sota/natural-language-inference-on-v-snli#since-judged","papers_judged":1,"placed":0},{"leaderboard":"/sota/natural-language-inference-on-mrpc","since_archive":"/sota/natural-language-inference-on-mrpc#since-judged","papers_judged":2,"placed":0},{"leaderboard":"/sota/natural-language-inference-on-tabfact","since_archive":"/sota/natural-language-inference-on-tabfact#since-judged","papers_judged":1,"placed":0}]},"papers_where_code_ran":209,"papers_where_code_ran_first_page":"/task/natural-language-inference/papers/ran/1","papers_where_code_ran_caption":"We ran code from the paper's repository; we did not run it on this task or check it against the task's benchmarks.","pages":[{"order":"archive","page":2,"of":20,"url":"/task/natural-language-inference/papers/2","json":"/task/natural-language-inference/papers/2.json","rows":100},{"order":"archive","page":3,"of":20,"url":"/task/natural-language-inference/papers/3","json":"/task/natural-language-inference/papers/3.json","rows":100},{"order":"archive","page":4,"of":20,"url":"/task/natural-language-inference/papers/4","json":"/task/natural-language-inference/papers/4.json","rows":100},{"order":"archive","page":5,"of":20,"url":"/task/natural-language-inference/papers/5","json":"/task/natural-language-inference/papers/5.json","rows":100},{"order":"archive","page":6,"of":20,"url":"/task/natural-language-inference/papers/6","json":"/task/natural-language-inference/papers/6.json","rows":100},{"order":"archive","page":7,"of":20,"url":"/task/natural-language-inference/papers/7","json":"/task/natural-language-inference/papers/7.json","rows":100},{"order":"archive","page":8,"of":20,"url":"/task/natural-language-inference/papers/8","json":"/task/natural-language-inference/papers/8.json","rows":100},{"order":"archive","page":9,"of":20,"url":"/task/natural-language-inference/papers/9","json":"/task/natural-language-inference/papers/9.json","rows":100},{"order":"archive","page":10,"of":20,"url":"/task/natural-language-inference/papers/10","json":"/task/natural-language-inference/papers/10.json","rows":100},{"order":"archive","page":11,"of":20,"url":"/task/natural-language-inference/papers/11","json":"/task/natural-language-inference/papers/11.json","rows":100},{"order":"archive","page":12,"of":20,"url":"/task/natural-language-inference/papers/12","json":"/task/natural-language-inference/papers/12.json","rows":100},{"order":"archive","page":13,"of":20,"url":"/task/natural-language-inference/papers/13","json":"/task/natural-language-inference/papers/13.json","rows":100},{"order":"archive","page":14,"of":20,"url":"/task/natural-language-inference/papers/14","json":"/task/natural-language-inference/papers/14.json","rows":100},{"order":"archive","page":15,"of":20,"url":"/task/natural-language-inference/papers/15","json":"/task/natural-language-inference/papers/15.json","rows":100},{"order":"archive","page":16,"of":20,"url":"/task/natural-language-inference/papers/16","json":"/task/natural-language-inference/papers/16.json","rows":100},{"order":"archive","page":17,"of":20,"url":"/task/natural-language-inference/papers/17","json":"/task/natural-language-inference/papers/17.json","rows":100},{"order":"archive","page":18,"of":20,"url":"/task/natural-language-inference/papers/18","json":"/task/natural-language-inference/papers/18.json","rows":100},{"order":"archive","page":19,"of":20,"url":"/task/natural-language-inference/papers/19","json":"/task/natural-language-inference/papers/19.json","rows":100},{"order":"archive","page":20,"of":20,"url":"/task/natural-language-inference/papers/20","json":"/task/natural-language-inference/papers/20.json","rows":61},{"order":"ran","page":1,"of":3,"url":"/task/natural-language-inference/papers/ran/1","json":"/task/natural-language-inference/papers/ran/1.json","rows":100},{"order":"ran","page":2,"of":3,"url":"/task/natural-language-inference/papers/ran/2","json":"/task/natural-language-inference/papers/ran/2.json","rows":100},{"order":"ran","page":3,"of":3,"url":"/task/natural-language-inference/papers/ran/3","json":"/task/natural-language-inference/papers/ran/3.json","rows":9}],"description_links":{"kept":3,"unwrapped_to_text":0,"bare_urls_linked":0,"relative_images_dropped":0,"rule":"internal links are kept only when the target slug exists in the catalog"},"syntology":{"read_at":"2026-10-05T00:36:11+00:00","claim":"Per-sample execution status on synthesized fixtures ('ran N of M samples'); not a correctness claim and not a ranking signal.","status_vocabulary":{"ran_honours":"ran, honoured the contract we drafted","ran_violates":"ran, violated the contract we drafted","ran_draft_wrong":"ran; our contract draft was wrong, not the code","ran_fixture":"ran; the contract check could not run (RAISES, DEP_MISSING or TIMEOUT)","ran":"ran on a synthesized input","unverified":"unverified (harvested, no recorded run)"},"row_route":"each syntology object on a benchmark's first row and on a paper carries sample_list (this site's paper page, #ran), syntology_url (syntology.ai's page for the paper, only when that route probed live) and mcp (the agent call that lists the paper's harvested samples; not a verdict on the paper)","developers":"https://syntology.ai/developers"},"not_shown":{"libraries":"the archive has no per-task library table","trend_sparklines":"the Trend column of the benchmarks table was a rendered image; it is not in the archive","social_and_latest_sorts":"stars and social signals are not in the archive"},"record_sha256":"47201cc3e51f3b01516a629712540b20a909cfafd5f8f2a64e2a569dab9a49af","record_changed_at":"2026-10-05","record_changed_at_basis":"changed"}