{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/language-models-are-few-shot-learners","title":"Language Models are Few-Shot Learners","arxiv_id":"2005.14165","date":"2020-05-28","proceeding":"NeurIPS 2020 12","authors":["Tom B. Brown","Benjamin Mann","Nick Ryder","Melanie Subbiah","Jared Kaplan","Prafulla Dhariwal","Arvind Neelakantan","Pranav Shyam","Girish Sastry","Amanda Askell","Sandhini Agarwal","Ariel Herbert-Voss","Gretchen Krueger","Tom Henighan","Rewon Child","Aditya Ramesh","Daniel M. Ziegler","Jeffrey Wu","Clemens Winter","Christopher Hesse","Mark Chen","Eric Sigler","Mateusz Litwin","Scott Gray","Benjamin Chess","Jack Clark","Christopher Berner","Sam McCandlish","Alec Radford","Ilya Sutskever","Dario Amodei"],"abstract":"Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task. While typically task-agnostic in architecture, this method still requires task-specific fine-tuning datasets of thousands or tens of thousands of examples. By contrast, humans can generally perform a new language task from only a few examples or from simple instructions - something which current NLP systems still largely struggle to do. Here we show that scaling up language models greatly improves task-agnostic, few-shot performance, sometimes even reaching competitiveness with prior state-of-the-art fine-tuning approaches. Specifically, we train GPT-3, an autoregressive language model with 175 billion parameters, 10x more than any previous non-sparse language model, and test its performance in the few-shot setting. For all tasks, GPT-3 is applied without any gradient updates or fine-tuning, with tasks and few-shot demonstrations specified purely via text interaction with the model. GPT-3 achieves strong performance on many NLP datasets, including translation, question-answering, and cloze tasks, as well as several tasks that require on-the-fly reasoning or domain adaptation, such as unscrambling words, using a novel word in a sentence, or performing 3-digit arithmetic. At the same time, we also identify some datasets where GPT-3's few-shot learning still struggles, as well as some datasets where GPT-3 faces methodological issues related to training on large web corpora. Finally, we find that GPT-3 can generate samples of news articles which human evaluators have difficulty distinguishing from articles written by humans. We discuss broader societal impacts of this finding and of GPT-3 in general.","url_abs":"https://arxiv.org/abs/2005.14165v4","url_pdf":"https://arxiv.org/pdf/2005.14165v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/openai/gpt-3","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/EightRice/atn_GPT-3","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok"}},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/EleutherAI/gpt-neo","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"unanswered"}},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/EleutherAI/lm-evaluation-harness","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"jax","reach":null},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/EleutherAI/lm_evaluation_harness","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/NVIDIA/NeMo-Curator","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/RUCAIBox/LLMBox","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/Samyu0304/thought-propagation","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"NOASSERTION"}},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/Sypherd/lm-evaluation-harness","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/VachanVY/gpt.jax","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/ai21labs/lm-evaluation","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"unanswered"}},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/allenai/macaw","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/asahi417/lmppl","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/asahi417/relbert","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/bigscience-workshop/Megatron-DeepSpeed","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/bigscience-workshop/promptsource","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":null},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/codedotal/gpt-code-clippy","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/contextlab/abstract2paper","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/crazydigger/Callibration-of-GPT","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"unanswered"}},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/ethanjperez/true_few_shot","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/facebookresearch/anli","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/fywalter/label-bias","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/ggerganov/llama.cpp","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/ggml-org/llama.cpp","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/gmum/dl-mo-2021","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/grantslatton/llama.cpp","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/haiyang-w/git","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/hazyresearch/ama_prompting","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":null},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/hilberthit/gpt-3","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/hojjat-mokhtarabadi/promptsource","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/insait-institute/lm-evaluation-harness-bg","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/juletx/lm-evaluation-harness","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/karpathy/build-nanogpt","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/karpathy/llm.c","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/lambert-x/prolab","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/longhao-chen/aicas2024","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/ltruncel/Microsoft_Azure_50daysofudacity","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok"}},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/mbzuai-paris/lm-evaluation-harness-atlas-chat","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/national-center-for-ai-saudi-arabia/lm-evaluation-harness","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"jax","reach":null},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/ncoop57/gpt-code-clippy","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/neuralmagic/lm-evaluation-harness","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/nlx-group/overlapy","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/openbiolink/promptsource","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/opengptx/lm-evaluation-harness","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/postech-ami/smile-dataset","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/ramanakshay/nanogpt","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/roberttwomey/machine-imagination-isea","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/roberttwomey/machine-imagination-workshop","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/sambanova/lm-evaluation-harness","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/scrayish/ML_NLP","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/shreyashankar/gpt3-sandbox","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/smarton-empower/smarton-ai","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":null},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/smile-data/smile","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/tonyzhaozh/few-shot-learning","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/turkunlp/megatron-deepspeed","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/um-arm-lab/efficient-eng-2-ltl","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/vilm-ai/viet-llm-eval","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/volcengine/vegiantmodel","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/x-lance/neusym-rag","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/zphang/lm_evaluation_harness","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/Mind23-2/MindCode-138","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":{"status":"ok"}},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/PaddlePaddle/PaddleNLP/tree/develop/examples/language_model/gpt-3","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"paddle","reach":null},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/abhaskumarsinha/Corpus2GPT","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":null},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/abhaskumarsinha/MinimalGPT","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"tf","reach":null},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/kyegomez/GPT3","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/milmor/GPT","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"tf","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/mindspore-ai/models/tree/master/official/nlp/gpt","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":null}],"tasks":[{"task_slug":"articles","task_name":"Articles"},{"task_slug":"common-sense-reasoning","task_name":"Common Sense Reasoning"},{"task_slug":"coreference-resolution","task_name":"Coreference Resolution"},{"task_slug":"domain-adaptation","task_name":"Domain Adaptation"},{"task_slug":"few-shot-learning","task_name":"Few-Shot Learning"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"multi-task-learning","task_name":"Multi-Task Learning"},{"task_slug":"multi-task-language-understanding","task_name":"Multi-task Language Understanding"},{"task_slug":"natural-language-inference","task_name":"Natural Language Inference"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"reading-comprehension","task_name":"Reading Comprehension"},{"task_slug":"sentence","task_name":"Sentence"},{"task_slug":"sentence-completion","task_name":"Sentence Completion"},{"task_slug":"unsupervised-machine-translation","task_name":"Unsupervised Machine Translation"},{"task_slug":"word-sense-disambiguation","task_name":"Word Sense Disambiguation"},{"task_slug":"zero-shot-learning","task_name":"Zero-Shot Learning"},{"task_slug":"answerability-prediction","task_name":"answerability prediction"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"cosine-annealing","method_name":"Cosine Annealing"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"gpt-3","method_name":"GPT-3"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-cosine-annealing","method_name":"Linear Warmup With Cosine Annealing"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"weight-decay","method_name":"Weight Decay"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/common-sense-reasoning-on-arc-challenge","task":"Common Sense Reasoning","dataset":"ARC (Challenge)","model":"GPT-3 175B (1 shot)","rank_in_archive_order":31,"of":54,"metrics":{"Accuracy":"53.2"},"uses_additional_data":true},{"leaderboard":"/sota/common-sense-reasoning-on-arc-challenge","task":"Common Sense Reasoning","dataset":"ARC (Challenge)","model":"GPT-3 175B (0-shot)","rank_in_archive_order":34,"of":54,"metrics":{"Accuracy":"51.4"},"uses_additional_data":false},{"leaderboard":"/sota/common-sense-reasoning-on-arc-easy","task":"Common Sense Reasoning","dataset":"ARC (Easy)","model":"GPT-3 175B (1 shot)","rank_in_archive_order":30,"of":47,"metrics":{"Accuracy":"71.2"},"uses_additional_data":false},{"leaderboard":"/sota/common-sense-reasoning-on-arc-easy","task":"Common Sense Reasoning","dataset":"ARC (Easy)","model":"GPT-3 175B (0-shot)","rank_in_archive_order":38,"of":47,"metrics":{"Accuracy":"68.8"},"uses_additional_data":false},{"leaderboard":"/sota/common-sense-reasoning-on-record","task":"Common Sense Reasoning","dataset":"ReCoRD","model":"GPT-3 Large 760M (0-shot)","rank_in_archive_order":15,"of":45,"metrics":{"EM":"82.1"},"uses_additional_data":false},{"leaderboard":"/sota/common-sense-reasoning-on-winogrande","task":"Common Sense Reasoning","dataset":"WinoGrande","model":"GPT-3 175B (0-shot)","rank_in_archive_order":39,"of":77,"metrics":{"Accuracy":"70.2"},"uses_additional_data":false},{"leaderboard":"/sota/common-sense-reasoning-on-winogrande","task":"Common Sense Reasoning","dataset":"WinoGrande","model":"GPT-3 Large 760M (0-shot)","rank_in_archive_order":58,"of":77,"metrics":{"Accuracy":"57.4"},"uses_additional_data":false},{"leaderboard":"/sota/coreference-resolution-on-winograd-schema","task":"Coreference Resolution","dataset":"Winograd Schema Challenge","model":"GPT-3 175B (few-shot)","rank_in_archive_order":21,"of":82,"metrics":{"Accuracy":"80.1"},"uses_additional_data":false},{"leaderboard":"/sota/few-shot-learning-on-medconceptsqa","task":"Few-Shot Learning","dataset":"MedConceptsQA","model":"gpt-3.5-turbo","rank_in_archive_order":2,"of":12,"metrics":{"Accuracy":"41.476"},"uses_additional_data":false},{"leaderboard":"/sota/language-modelling-on-lambada","task":"Language Modelling","dataset":"LAMBADA","model":"GPT-3 175B (Few-Shot)","rank_in_archive_order":3,"of":37,"metrics":{"Accuracy":"86.4","Perplexity":"1.92"},"uses_additional_data":false},{"leaderboard":"/sota/language-modelling-on-lambada","task":"Language Modelling","dataset":"LAMBADA","model":"GPT-3 175B (Zero-Shot)","rank_in_archive_order":18,"of":37,"metrics":{"Accuracy":"76.2","Perplexity":"3.00"},"uses_additional_data":false},{"leaderboard":"/sota/language-modelling-on-lambada","task":"Language Modelling","dataset":"LAMBADA","model":"GPT-3 13B (Zero-Shot)","rank_in_archive_order":20,"of":37,"metrics":{"Accuracy":"72.5","Perplexity":"3.56"},"uses_additional_data":false},{"leaderboard":"/sota/language-modelling-on-lambada","task":"Language Modelling","dataset":"LAMBADA","model":"GPT-3 6.7B (Zero-Shot)","rank_in_archive_order":23,"of":37,"metrics":{"Accuracy":"70.3","Perplexity":"4.00"},"uses_additional_data":false},{"leaderboard":"/sota/language-modelling-on-lambada","task":"Language Modelling","dataset":"LAMBADA","model":"GPT-3 2.7B (Zero-Shot)","rank_in_archive_order":28,"of":37,"metrics":{"Accuracy":"67.1","Perplexity":"4.60"},"uses_additional_data":false},{"leaderboard":"/sota/language-modelling-on-penn-treebank-word","task":"Language Modelling","dataset":"Penn Treebank (Word Level)","model":"GPT-3 (Zero-Shot)","rank_in_archive_order":1,"of":43,"metrics":{"Params":"175000M","Test perplexity":"20.5"},"uses_additional_data":true},{"leaderboard":"/sota/natural-language-inference-on-anli-test","task":"Natural Language Inference","dataset":"ANLI test","model":"GPT-3","rank_in_archive_order":13,"of":25,"metrics":{"A1":"36.8","A2":"34","A3":"40.2"},"uses_additional_data":true},{"leaderboard":"/sota/natural-language-inference-on-commitmentbank","task":"Natural Language Inference","dataset":"CommitmentBank","model":"GPT-3 175B (Few-Shot)","rank_in_archive_order":13,"of":20,"metrics":{"Accuracy":"75.6"},"uses_additional_data":false},{"leaderboard":"/sota/natural-language-inference-on-commitmentbank","task":"Natural Language Inference","dataset":"CommitmentBank","model":"GPT-3 175B (few-shot, k=32)","rank_in_archive_order":20,"of":20,"metrics":{"F1":"52"},"uses_additional_data":false},{"leaderboard":"/sota/natural-language-inference-on-rte","task":"Natural Language Inference","dataset":"RTE","model":"GPT-3 175B (few-shot, k=32)","rank_in_archive_order":58,"of":90,"metrics":{"Accuracy":"69%"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-boolq","task":"Question Answering","dataset":"BoolQ","model":"GPT-3 175B (few-shot, k=32)","rank_in_archive_order":34,"of":65,"metrics":{"Accuracy":"76.4"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-boolq","task":"Question Answering","dataset":"BoolQ","model":"GPT-3 75B (0-shot)","rank_in_archive_order":57,"of":65,"metrics":{"Accuracy":"60.5"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-copa","task":"Question Answering","dataset":"COPA","model":"GPT-3 175B (few-shot, k=32)","rank_in_archive_order":11,"of":60,"metrics":{"Accuracy":"92"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-copa","task":"Question Answering","dataset":"COPA","model":"GPT-3 175B (0-shot)","rank_in_archive_order":15,"of":60,"metrics":{"Accuracy":"91"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-copa","task":"Question Answering","dataset":"COPA","model":"GPT-3 175B (1-shot)","rank_in_archive_order":23,"of":60,"metrics":{"Accuracy":"87"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-copa","task":"Question Answering","dataset":"COPA","model":"GPT-3 13B (few-shot, k=32)","rank_in_archive_order":27,"of":60,"metrics":{"Accuracy":"86"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-copa","task":"Question Answering","dataset":"COPA","model":"GPT-3 Large 760M (0-shot)","rank_in_archive_order":46,"of":60,"metrics":{"Accuracy":"73.0"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-coqa","task":"Question Answering","dataset":"CoQA","model":"GPT-3 175B (few-shot, k=32)","rank_in_archive_order":7,"of":9,"metrics":{"Overall":"85"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-drop-test","task":"Question Answering","dataset":"DROP Test","model":"GPT-3 175B (few-shot, k=32)","rank_in_archive_order":15,"of":16,"metrics":{"F1":"36.5"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-multirc","task":"Question Answering","dataset":"MultiRC","model":"GPT-3 175B (Few-Shot)","rank_in_archive_order":13,"of":30,"metrics":{"F1":"75.4"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-natural-questions","task":"Question Answering","dataset":"Natural Questions","model":"GPT-3 175B (Few-Shot, k=64)","rank_in_archive_order":35,"of":47,"metrics":{"EM":"29.9"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-obqa","task":"Question Answering","dataset":"OBQA","model":"GPT-3 175B (zero-shot)","rank_in_archive_order":5,"of":9,"metrics":{"Accuracy":"57.6"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-openbookqa","task":"Question Answering","dataset":"OpenBookQA","model":"GPT-3 175B (few-shot, k=32)","rank_in_archive_order":29,"of":45,"metrics":{"Accuracy":"65.4"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-piqa","task":"Question Answering","dataset":"PIQA","model":"GPT-3 175B (0-shot)","rank_in_archive_order":29,"of":67,"metrics":{"Accuracy":"81.0"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-piqa","task":"Question Answering","dataset":"PIQA","model":"GPT-3 Large 760M (0-shot)","rank_in_archive_order":52,"of":67,"metrics":{"Accuracy":"72.9"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-peerqa","task":"Question Answering","dataset":"PeerQA","model":"GPT-3.5-Turbo-0613-16k","rank_in_archive_order":6,"of":6,"metrics":{"AlignScore":"0.1378","Prometheus-2 Answer Correctness":"3.0408","Rouge-L":"0.2414"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-quac","task":"Question Answering","dataset":"QuAC","model":"GPT-3 175B (few-shot, k=32)","rank_in_archive_order":2,"of":2,"metrics":{"F1":"44.3"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-race","task":"Question Answering","dataset":"RACE","model":"GPT-3 175B (few-shot, k=32)","rank_in_archive_order":6,"of":7,"metrics":{"RACE-m":"58.1"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-race","task":"Question Answering","dataset":"RACE","model":"GPT-3 175B (Few-Shot)","rank_in_archive_order":7,"of":7,"metrics":{"RACE-h":"46.8"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-story-cloze","task":"Question Answering","dataset":"Story Cloze","model":"GPT-3 175B (Few-Shot)","rank_in_archive_order":2,"of":7,"metrics":{"Accuracy":"87.7"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-storycloze","task":"Question Answering","dataset":"StoryCloze","model":"GPT-3 Large 760M (zero-shot)","rank_in_archive_order":19,"of":23,"metrics":{"Accuracy":"72.4"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-triviaqa","task":"Question Answering","dataset":"TriviaQA","model":"GPT-3 175B (Few-Shot)","rank_in_archive_order":31,"of":56,"metrics":{"EM":"71.2"},"uses_additional_data":true},{"leaderboard":"/sota/question-answering-on-webquestions","task":"Question Answering","dataset":"WebQuestions","model":"Few-shot","rank_in_archive_order":9,"of":37,"metrics":{"EM":"44.7"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-webquestions","task":"Question Answering","dataset":"WebQuestions","model":"GPT-3-175B (Few-Shot)","rank_in_archive_order":18,"of":37,"metrics":{"EM":"41.5"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-webquestions","task":"Question Answering","dataset":"WebQuestions","model":"GPT-3-175B (One-Shot)","rank_in_archive_order":29,"of":37,"metrics":{"EM":"25.3"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-webquestions","task":"Question Answering","dataset":"WebQuestions","model":"GPT-3-175B (Zero-Shot)","rank_in_archive_order":33,"of":37,"metrics":{"EM":"14.4"},"uses_additional_data":false},{"leaderboard":"/sota/reading-comprehension-on-race","task":"Reading Comprehension","dataset":"RACE","model":"GPT-3 175B (0-shot)","rank_in_archive_order":18,"of":24,"metrics":{"Accuracy (Middle)":"58.4"},"uses_additional_data":false},{"leaderboard":"/sota/reading-comprehension-on-race","task":"Reading Comprehension","dataset":"RACE","model":"GPT-3 175B (zero-shot)","rank_in_archive_order":24,"of":24,"metrics":{"Accuracy (High)":"45.5"},"uses_additional_data":false},{"leaderboard":"/sota/sentence-completion-on-hellaswag","task":"Sentence Completion","dataset":"HellaSwag","model":"GPT-3 175B (few-shot, k=32)","rank_in_archive_order":45,"of":89,"metrics":{"Accuracy":"79.3"},"uses_additional_data":false},{"leaderboard":"/sota/sentence-completion-on-hellaswag","task":"Sentence Completion","dataset":"HellaSwag","model":"GPT-3 (0-shot)","rank_in_archive_order":48,"of":89,"metrics":{"Accuracy":"78.9"},"uses_additional_data":false},{"leaderboard":"/sota/sentence-completion-on-hellaswag","task":"Sentence Completion","dataset":"HellaSwag","model":"GPT-3 Large 760M (0-shot)","rank_in_archive_order":65,"of":89,"metrics":{"Accuracy":"51.0"},"uses_additional_data":false},{"leaderboard":"/sota/unsupervised-machine-translation-on-wmt2014-2","task":"Unsupervised Machine Translation","dataset":"WMT2014 English-French","model":"GPT-3 175B (Few-Shot)","rank_in_archive_order":5,"of":7,"metrics":{"BLEU":"32.6"},"uses_additional_data":false},{"leaderboard":"/sota/unsupervised-machine-translation-on-wmt2014-1","task":"Unsupervised Machine Translation","dataset":"WMT2014 French-English","model":"GPT-3 175B (Few-Shot)","rank_in_archive_order":1,"of":7,"metrics":{"BLEU":"39.2"},"uses_additional_data":false},{"leaderboard":"/sota/unsupervised-machine-translation-on-wmt2016","task":"Unsupervised Machine Translation","dataset":"WMT2016 English-German","model":"GPT-3 175B (Few-Shot)","rank_in_archive_order":1,"of":7,"metrics":{"BLEU":"29.7"},"uses_additional_data":false},{"leaderboard":"/sota/unsupervised-machine-translation-on-wmt2016-2","task":"Unsupervised Machine Translation","dataset":"WMT2016 English-Romanian","model":"GPT-3 175B (Few-Shot)","rank_in_archive_order":1,"of":3,"metrics":{"BLEU":"21"},"uses_additional_data":false},{"leaderboard":"/sota/unsupervised-machine-translation-on-wmt2016-1","task":"Unsupervised Machine Translation","dataset":"WMT2016 German-English","model":"GPT-3 175B (Few-Shot)","rank_in_archive_order":1,"of":7,"metrics":{"BLEU":"40.6"},"uses_additional_data":false},{"leaderboard":"/sota/unsupervised-machine-translation-on-wmt2016-3","task":"Unsupervised Machine Translation","dataset":"WMT2016 Romanian-English","model":"GPT-3 175B (Few-Shot)","rank_in_archive_order":1,"of":3,"metrics":{"BLEU":"39.5"},"uses_additional_data":false},{"leaderboard":"/sota/word-sense-disambiguation-on-words-in-context","task":"Word Sense Disambiguation","dataset":"Words in Context","model":"GPT-3 175B (few-shot, k=32)","rank_in_archive_order":36,"of":37,"metrics":{"Accuracy":"49.4"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-learning-on-medconceptsqa","task":"Zero-Shot Learning","dataset":"MedConceptsQA","model":"gpt-3.5-turbo","rank_in_archive_order":2,"of":13,"metrics":{"Accuracy":"37.058"},"uses_additional_data":false},{"leaderboard":"/sota/answerability-prediction-on-peerqa","task":"answerability prediction","dataset":"PeerQA","model":"GPT-3.5-Turbo-0613-16k","rank_in_archive_order":3,"of":6,"metrics":{"Macro F1":"0.3304"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2005.14165","atlas_url":"https://app.syntology.ai/?focus=2005.14165","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2005.14165"},"calls":[{"tool":"get_harvested_code_for_paper","arguments":{"arxiv_id":"2005.14165"},"returns":"the samples harvested from this paper, each with its run record; it does not verify the paper"},{"tool":"get_code","arguments_in":"samples[].code_sha256","returns":"one stored sample by its code_sha256; not shown on this site"}]},"developers":"https://syntology.ai/developers","held":true,"reason":"in_graph_layer","checked_against":null,"agent_calls":[{"tool":"get_harvested_code_for_paper","arguments":{"arxiv_id":"2005.14165"},"call":"get_harvested_code_for_paper(arxiv_id=\"2005.14165\")"},{"tool":"get_code","arguments_in":{"code_sha256":"syntology.samples[].code_sha256"},"call":"get_code(code_sha256=\"…\")"},{"tool":"get_citation_path","arguments":{"paper_1":"2005.14165"},"arguments_in":{"paper_2":"another paper's arXiv id or title"},"call":"get_citation_path(paper_1=\"2005.14165\", paper_2=\"…\")"},{"tool":"get_concepts_for_paper","arguments":{"arxiv_id":"2005.14165"},"call":"get_concepts_for_paper(arxiv_id=\"2005.14165\")"}],"by_repo":[{"repo":"openai/gpt-3","kind":"official","samples":0,"ran":0,"instrument":0,"unverified":0},{"repo":"EightRice/atn_GPT-3","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0},{"repo":"EleutherAI/gpt-neo","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0},{"repo":"EleutherAI/lm-evaluation-harness","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0,"harvested_for_other_papers":2,"harvested_for":["2503.00137","2505.18383"],"harvested_for_calls":[{"arxiv_id":"2503.00137","tool":"get_harvested_code_for_paper","arguments":{"arxiv_id":"2503.00137"},"call":"get_harvested_code_for_paper(arxiv_id=\"2503.00137\")"},{"arxiv_id":"2505.18383","tool":"get_harvested_code_for_paper","arguments":{"arxiv_id":"2505.18383"},"call":"get_harvested_code_for_paper(arxiv_id=\"2505.18383\")"}]},{"repo":"EleutherAI/lm_evaluation_harness","kind":"listed","samples":2,"ran":2,"instrument":0,"unverified":0},{"repo":"NVIDIA/NeMo-Curator","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0},{"repo":"RUCAIBox/LLMBox","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0,"harvested_for_other_papers":1,"harvested_for":["2407.05563"],"harvested_for_calls":[{"arxiv_id":"2407.05563","tool":"get_harvested_code_for_paper","arguments":{"arxiv_id":"2407.05563"},"call":"get_harvested_code_for_paper(arxiv_id=\"2407.05563\")"}]},{"repo":"Samyu0304/thought-propagation","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0,"harvested_for_other_papers":1,"harvested_for":["2310.03965"],"harvested_for_calls":[{"arxiv_id":"2310.03965","tool":"get_harvested_code_for_paper","arguments":{"arxiv_id":"2310.03965"},"call":"get_harvested_code_for_paper(arxiv_id=\"2310.03965\")"}]},{"repo":"Sypherd/lm-evaluation-harness","kind":"listed","samples":4,"ran":4,"instrument":3,"unverified":0},{"repo":"VachanVY/gpt.jax","kind":"listed","samples":2,"ran":1,"instrument":0,"unverified":1},{"repo":"ai21labs/lm-evaluation","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0},{"repo":"allenai/macaw","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0,"harvested_for_other_papers":1,"harvested_for":["2203.07540"],"harvested_for_calls":[{"arxiv_id":"2203.07540","tool":"get_harvested_code_for_paper","arguments":{"arxiv_id":"2203.07540"},"call":"get_harvested_code_for_paper(arxiv_id=\"2203.07540\")"}]},{"repo":"asahi417/lmppl","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0,"harvested_for_other_papers":1,"harvested_for":["2506.15629"],"harvested_for_calls":[{"arxiv_id":"2506.15629","tool":"get_harvested_code_for_paper","arguments":{"arxiv_id":"2506.15629"},"call":"get_harvested_code_for_paper(arxiv_id=\"2506.15629\")"}]},{"repo":"asahi417/relbert","kind":"listed","samples":15,"ran":10,"instrument":0,"unverified":5},{"repo":"bigscience-workshop/Megatron-DeepSpeed","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0},{"repo":"bigscience-workshop/promptsource","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0,"harvested_for_other_papers":2,"harvested_for":["2110.08207","2202.01279"],"harvested_for_calls":[{"arxiv_id":"2110.08207","tool":"get_harvested_code_for_paper","arguments":{"arxiv_id":"2110.08207"},"call":"get_harvested_code_for_paper(arxiv_id=\"2110.08207\")"},{"arxiv_id":"2202.01279","tool":"get_harvested_code_for_paper","arguments":{"arxiv_id":"2202.01279"},"call":"get_harvested_code_for_paper(arxiv_id=\"2202.01279\")"}]},{"repo":"codedotal/gpt-code-clippy","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0,"harvested_for_other_papers":1,"harvested_for":["2107.03374"],"harvested_for_calls":[{"arxiv_id":"2107.03374","tool":"get_harvested_code_for_paper","arguments":{"arxiv_id":"2107.03374"},"call":"get_harvested_code_for_paper(arxiv_id=\"2107.03374\")"}]},{"repo":"contextlab/abstract2paper","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0},{"repo":"crazydigger/Callibration-of-GPT","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0},{"repo":"ethanjperez/true_few_shot","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0,"harvested_for_other_papers":1,"harvested_for":["2103.11955"],"harvested_for_calls":[{"arxiv_id":"2103.11955","tool":"get_harvested_code_for_paper","arguments":{"arxiv_id":"2103.11955"},"call":"get_harvested_code_for_paper(arxiv_id=\"2103.11955\")"}]},{"repo":"facebookresearch/anli","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0},{"repo":"fywalter/label-bias","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0,"harvested_for_other_papers":1,"harvested_for":["2305.19148"],"harvested_for_calls":[{"arxiv_id":"2305.19148","tool":"get_harvested_code_for_paper","arguments":{"arxiv_id":"2305.19148"},"call":"get_harvested_code_for_paper(arxiv_id=\"2305.19148\")"}]},{"repo":"ggerganov/llama.cpp","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0,"harvested_for_other_papers":1,"harvested_for":["2306.00978"],"harvested_for_calls":[{"arxiv_id":"2306.00978","tool":"get_harvested_code_for_paper","arguments":{"arxiv_id":"2306.00978"},"call":"get_harvested_code_for_paper(arxiv_id=\"2306.00978\")"}]},{"repo":"ggml-org/llama.cpp","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0,"harvested_for_other_papers":1,"harvested_for":["2609.01315"],"harvested_for_calls":[{"arxiv_id":"2609.01315","tool":"get_harvested_code_for_paper","arguments":{"arxiv_id":"2609.01315"},"call":"get_harvested_code_for_paper(arxiv_id=\"2609.01315\")"}]},{"repo":"gmum/dl-mo-2021","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0},{"repo":"grantslatton/llama.cpp","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0,"harvested_for_other_papers":1,"harvested_for":["2302.13971"],"harvested_for_calls":[{"arxiv_id":"2302.13971","tool":"get_harvested_code_for_paper","arguments":{"arxiv_id":"2302.13971"},"call":"get_harvested_code_for_paper(arxiv_id=\"2302.13971\")"}]},{"repo":"haiyang-w/git","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0},{"repo":"hazyresearch/ama_prompting","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0,"harvested_for_other_papers":1,"harvested_for":["2210.02441"],"harvested_for_calls":[{"arxiv_id":"2210.02441","tool":"get_harvested_code_for_paper","arguments":{"arxiv_id":"2210.02441"},"call":"get_harvested_code_for_paper(arxiv_id=\"2210.02441\")"}]},{"repo":"hilberthit/gpt-3","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0},{"repo":"hojjat-mokhtarabadi/promptsource","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0,"harvested_for_other_papers":1,"harvested_for":["2110.08207"],"harvested_for_calls":[{"arxiv_id":"2110.08207","tool":"get_harvested_code_for_paper","arguments":{"arxiv_id":"2110.08207"},"call":"get_harvested_code_for_paper(arxiv_id=\"2110.08207\")"}]},{"repo":"insait-institute/lm-evaluation-harness-bg","kind":"listed","samples":2,"ran":1,"instrument":0,"unverified":1},{"repo":"juletx/lm-evaluation-harness","kind":"listed","samples":4,"ran":4,"instrument":0,"unverified":0},{"repo":"karpathy/build-nanogpt","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0},{"repo":"karpathy/llm.c","kind":"listed","samples":1,"ran":1,"instrument":1,"unverified":0},{"repo":"lambert-x/prolab","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0,"harvested_for_other_papers":1,"harvested_for":["05527"],"harvested_for_calls":[]},{"repo":"longhao-chen/aicas2024","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0,"harvested_for_other_papers":1,"harvested_for":["2302.13971"],"harvested_for_calls":[{"arxiv_id":"2302.13971","tool":"get_harvested_code_for_paper","arguments":{"arxiv_id":"2302.13971"},"call":"get_harvested_code_for_paper(arxiv_id=\"2302.13971\")"}]},{"repo":"ltruncel/Microsoft_Azure_50daysofudacity","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0},{"repo":"mbzuai-paris/lm-evaluation-harness-atlas-chat","kind":"listed","samples":2,"ran":0,"instrument":0,"unverified":2},{"repo":"national-center-for-ai-saudi-arabia/lm-evaluation-harness","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0,"harvested_for_other_papers":1,"harvested_for":["2402.01781"],"harvested_for_calls":[{"arxiv_id":"2402.01781","tool":"get_harvested_code_for_paper","arguments":{"arxiv_id":"2402.01781"},"call":"get_harvested_code_for_paper(arxiv_id=\"2402.01781\")"}]},{"repo":"ncoop57/gpt-code-clippy","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0},{"repo":"neuralmagic/lm-evaluation-harness","kind":"listed","samples":5,"ran":5,"instrument":0,"unverified":0},{"repo":"nlx-group/overlapy","kind":"listed","samples":1,"ran":1,"instrument":0,"unverified":0},{"repo":"openbiolink/promptsource","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0},{"repo":"opengptx/lm-evaluation-harness","kind":"listed","samples":2,"ran":2,"instrument":0,"unverified":0},{"repo":"postech-ami/smile-dataset","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0,"harvested_for_other_papers":1,"harvested_for":["2312.09818"],"harvested_for_calls":[{"arxiv_id":"2312.09818","tool":"get_harvested_code_for_paper","arguments":{"arxiv_id":"2312.09818"},"call":"get_harvested_code_for_paper(arxiv_id=\"2312.09818\")"}]},{"repo":"ramanakshay/nanogpt","kind":"listed","samples":2,"ran":2,"instrument":0,"unverified":0},{"repo":"roberttwomey/machine-imagination-isea","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0},{"repo":"roberttwomey/machine-imagination-workshop","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0},{"repo":"sambanova/lm-evaluation-harness","kind":"listed","samples":2,"ran":1,"instrument":1,"unverified":1},{"repo":"scrayish/ML_NLP","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0,"harvested_for_other_papers":1,"harvested_for":["1706.03762"],"harvested_for_calls":[{"arxiv_id":"1706.03762","tool":"get_harvested_code_for_paper","arguments":{"arxiv_id":"1706.03762"},"call":"get_harvested_code_for_paper(arxiv_id=\"1706.03762\")"}]},{"repo":"shreyashankar/gpt3-sandbox","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0},{"repo":"smarton-empower/smarton-ai","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0,"harvested_for_other_papers":1,"harvested_for":["2307.14740"],"harvested_for_calls":[{"arxiv_id":"2307.14740","tool":"get_harvested_code_for_paper","arguments":{"arxiv_id":"2307.14740"},"call":"get_harvested_code_for_paper(arxiv_id=\"2307.14740\")"}]},{"repo":"smile-data/smile","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0,"harvested_for_other_papers":1,"harvested_for":["2312.09818"],"harvested_for_calls":[{"arxiv_id":"2312.09818","tool":"get_harvested_code_for_paper","arguments":{"arxiv_id":"2312.09818"},"call":"get_harvested_code_for_paper(arxiv_id=\"2312.09818\")"}]},{"repo":"tonyzhaozh/few-shot-learning","kind":"listed","samples":4,"ran":0,"instrument":0,"unverified":4,"placed_by_identical_code":4},{"repo":"turkunlp/megatron-deepspeed","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0},{"repo":"um-arm-lab/efficient-eng-2-ltl","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0,"harvested_for_other_papers":1,"harvested_for":["2303.08006"],"harvested_for_calls":[{"arxiv_id":"2303.08006","tool":"get_harvested_code_for_paper","arguments":{"arxiv_id":"2303.08006"},"call":"get_harvested_code_for_paper(arxiv_id=\"2303.08006\")"}]},{"repo":"vilm-ai/viet-llm-eval","kind":"listed","samples":7,"ran":6,"instrument":3,"unverified":1},{"repo":"volcengine/vegiantmodel","kind":"listed","samples":7,"ran":3,"instrument":0,"unverified":4},{"repo":"x-lance/neusym-rag","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0},{"repo":"zphang/lm_evaluation_harness","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0},{"repo":"Mind23-2/MindCode-138","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0},{"repo":"PaddlePaddle/PaddleNLP","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0,"harvested_for_other_papers":1,"harvested_for":["2301.03416"],"harvested_for_calls":[{"arxiv_id":"2301.03416","tool":"get_harvested_code_for_paper","arguments":{"arxiv_id":"2301.03416"},"call":"get_harvested_code_for_paper(arxiv_id=\"2301.03416\")"}]},{"repo":"abhaskumarsinha/Corpus2GPT","kind":"listed","samples":2,"ran":1,"instrument":0,"unverified":1},{"repo":"abhaskumarsinha/MinimalGPT","kind":"listed","samples":1,"ran":1,"instrument":0,"unverified":0},{"repo":"kyegomez/GPT3","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0},{"repo":"milmor/GPT","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0},{"repo":"mindspore-ai/models","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0,"harvested_for_other_papers":1,"harvested_for":["2208.08084"],"harvested_for_calls":[{"arxiv_id":"2208.08084","tool":"get_harvested_code_for_paper","arguments":{"arxiv_id":"2208.08084"},"call":"get_harvested_code_for_paper(arxiv_id=\"2208.08084\")"}]}],"by_repo_harvested_for_is":"harvested_for: the arXiv ids (as Syntology's graph holds them) of every paper whose samples Syntology's graph links to this repository, in arXiv-id order, complete; the page names the first 3 and says how many more. A link only: it does not say the papers are the same work, that one copied the other, or that the code implements this paper's method; harvested_for_calls gives the get_harvested_code_for_paper call for each arXiv-shaped id among them","by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code","read_at":"2026-10-05T00:36:11+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","ran_record":{"url":"https://syntology.ai/api/ran/2005.14165.json","schema":"syntology.ran_record/1","counts_crosswalk":{"n_samples":"syntology.counts.lifted","n_ran":"syntology.counts.executed","n_ran_checked":"syntology.counts.checked","n_instrument":"syntology.counts.instrument_failures","n_unverified":"syntology.counts.no_recorded_run"},"note":"live record, measured at request time; may be newer than this page's graph read","badge":"https://syntology.ai/api/ran/2005.14165.svg"},"claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/tonyzhaozh/few-shot-learning","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/bigscience-workshop/promptsource","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Samyu0304/thought-propagation","reach":{"status":"ok","spdx":"NOASSERTION"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ggml-org/llama.cpp","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/EleutherAI/lm-evaluation-harness","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mindspore-ai/models/tree/master/official/nlp/gpt","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ai21labs/lm-evaluation","reach":{"status":"unanswered"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/juletx/lm-evaluation-harness","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/shreyashankar/gpt3-sandbox","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/um-arm-lab/efficient-eng-2-ltl","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/abhaskumarsinha/Corpus2GPT","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/haiyang-w/git","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/neuralmagic/lm-evaluation-harness","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ltruncel/Microsoft_Azure_50daysofudacity","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/EightRice/atn_GPT-3","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/EleutherAI/gpt-neo","reach":{"status":"unanswered"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/fywalter/label-bias","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/abhaskumarsinha/MinimalGPT","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/hazyresearch/ama_prompting","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/hojjat-mokhtarabadi/promptsource","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/RUCAIBox/LLMBox","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/openai/gpt-3","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/allenai/macaw","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/PaddlePaddle/PaddleNLP/tree/develop/examples/language_model/gpt-3","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/crazydigger/Callibration-of-GPT","reach":{"status":"unanswered"}}],"summary":{"ran_violates":2,"ran":32,"ran_draft_wrong":7,"ran_honours":3,"ran_fixture":1,"unverified":20},"n_samples":65,"n_ran":45,"n_constructed":0,"n_ran_checked":37,"n_instrument":8,"n_unverified":20,"n_instrument_is":"failures of Syntology's instrument, not of the code","every_run_is_an_instrument_failure":false,"key_notes":{"n_ran_checked":"legacy name, kept unchanged so existing readers do not break: it counts the samples that ran with no instrument failure (honoured, violated, and ran with no contract checked); it does not mean a contract was checked, and the pages print it as 'K with no instrument failure', not 'K checked'","n_constructed":"a sub-count of the samples that ran, never subtracted from them and never a failure: an executed sample whose run returned an instance of its own class (fixture_out_type equals the entry name): the run built an object and did not compute a result (Syntology's RAN record, counts.constructed)"},"by_repo_kind":{"listed":{"samples":65,"ran":45,"n_ran_checked":37,"n_instrument":8,"repositories":18}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"official_not_harvested":{"repositories":["openai/gpt-3"],"harvested_for_other_papers":[0],"samples_from_other_repositories":65,"other_repositories":18,"samples_unlocated":0},"n_pointer_only_for_licence":7,"inline_ok_is":"inline_ok is true only when both licence gates allow Syntology to serve this code's text: this paper's link (inline_ok_link, from the licence of this paper's copy) and the stored copy of the same code that Syntology serves, which may have been first harvested from another repository. withheld_clause is Syntology's own wording of which gate keeps the text a pointer and why, and is null when inline_ok is true. This site shows no code text either way; n_pointer_only_for_licence counts samples with inline_ok false (both gates), n_pointer_only_link_allows those of them whose inline_ok_link is true","n_pointer_only_link_allows":3,"fixture_fields_is":{"signature_and_fixture_out_type":"the arguments Syntology's fixture called the sample with, each as name:type/rank/element kind (? where none was recorded), and the type of what the fixture's run returned, exactly as the harness recorded them; the arguments and the returned type were recorded by two separate runs of the same fixture. No output value is shown, and a returned type says nothing about whether the output was right","level_recorded_at":"when Syntology's graph wrote the sample's verification level, not when the code executed; absent where the level was written before the date was kept, and never filled in with the build date","present_on":"samples that ran only; a key is absent, never empty, where the graph records nothing","signature_not_shown":"the graph records a signature for this sample and it is not shown; signature_not_shown_because names why, as the export's field-exclusion reason","signature_not_shown_because":{"signature_is_harness_object_repr":"an argument's recorded type or element kind is a printout of an object in Syntology's harness (such as a mock object, a class or a pandas printout), not the type of an input, so it does not describe the call's shape"}},"n_ran_with_level_date":43,"n_ran_without_level_date":2,"n_ran_signature_not_shown":1,"levels_written_between":{"first":"2026-09-22","last":"2026-09-25","is":"the first and last UTC dates on which Syntology's graph wrote a verification level for a sample on this page that ran, not when the code executed"},"code_page_min_papers":5,"code_page_is":"a page for the entry name across every paper's harvested code; the name is a string, not an identity","census_is":"Syntology's function-name census ran implementations with this entry name on one shared input per bucket (the positional rank, kind and dtype of each array argument) and counted the distinct output digests (sha256 of the output rounded to 6 decimals). implementations and papers count the compared samples; distinct_outputs counts digests across all buckets; n_buckets is how many shared inputs were used. It is not a statement that any two samples compute one thing, and the name is a string, not an identity.","samples":[{"code_sha256":"f0fc0ac7590a2d39","code_sha256_prefix":"f0fc0ac7590a2d39","entry":"FLOP","repo":"abhaskumarsinha/Corpus2GPT","repo_kind":"listed","path":"models/GPT.py","file_url":"https://github.com/abhaskumarsinha/Corpus2GPT/blob/HEAD/models/GPT.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"signature":"input_len:int/0/int, vocab_size:int/0/int, embed_dim:int/0/int, num_heads:int/0/int, num_decoders:int/0/int, fc_dim_factor:int/0/int","fixture_out_type":"int","level_recorded_at":"2026-09-22T14:31:46.068493+00:00","retrieve_with":{"tool":"get_code","code_sha256":"f0fc0ac7590a2d39"},"code_page":null,"census":null,"licence":"Apache-2.0","inline_ok":true,"inline_ok_link":true,"withheld_clause":null,"mcp_get_code":{"code_sha256":"f0fc0ac7590a2d39"}},{"code_sha256":"904fac6af5735d5a","code_sha256_prefix":"904fac6af5735d5a","entry":"column_parallel_load_hook","repo":"volcengine/vegiantmodel","repo_kind":"listed","path":"src/veGiantModel/module/dense.py","file_url":"https://github.com/volcengine/vegiantmodel/blob/HEAD/src/veGiantModel/module/dense.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"signature":"module:callable/0/?, log_fn:callable/0/?","fixture_out_type":"function","level_recorded_at":"2026-09-24T22:24:42.659479+00:00","code_page":null,"census":null,"licence":"Apache-2.0","inline_ok":true,"inline_ok_link":true,"withheld_clause":null,"mcp_get_code":{"code_sha256":"904fac6af5735d5a"}},{"code_sha256":"64087979238e1d41","code_sha256_prefix":"64087979238e1d41","entry":"column_serial_load_hook","repo":"volcengine/vegiantmodel","repo_kind":"listed","path":"src/veGiantModel/module/dense.py","file_url":"https://github.com/volcengine/vegiantmodel/blob/HEAD/src/veGiantModel/module/dense.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"signature":"module:callable/0/?, log_fn:callable/0/?","fixture_out_type":"function","level_recorded_at":"2026-09-24T22:24:42.659479+00:00","code_page":null,"census":null,"licence":"Apache-2.0","inline_ok":true,"inline_ok_link":true,"withheld_clause":null,"mcp_get_code":{"code_sha256":"64087979238e1d41"}},{"code_sha256":"451bc0edacfc317d","code_sha256_prefix":"451bc0edacfc317d","entry":"configure_optimizers","repo":"ramanakshay/nanogpt","repo_kind":"listed","path":"src/algorithm/optimizer.py","file_url":"https://github.com/ramanakshay/nanogpt/blob/HEAD/src/algorithm/optimizer.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"signature":"model:callable/0/?, weight_decay:float/0/float, learning_rate:float/0/float, betas:tuple/1/float, eps:float/0/float, state:State/0/?","fixture_out_type":"AdamW","level_recorded_at":"2026-09-24T22:24:42.659479+00:00","code_page":"/code/configure-optimizers","census":null,"licence":"MIT","inline_ok":true,"inline_ok_link":true,"withheld_clause":null,"mcp_get_code":{"code_sha256":"451bc0edacfc317d"}},{"code_sha256":"3055a2ccdae837c7","code_sha256_prefix":"3055a2ccdae837c7","entry":"cosine_similarity","repo":"asahi417/relbert","repo_kind":"listed","path":"relbert/evaluation/analogy_questions.py","file_url":"https://github.com/asahi417/relbert/blob/HEAD/relbert/evaluation/analogy_questions.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"signature":"a:list/1/float, b:list/1/float, zero_vector_mask:int/0/int","fixture_out_type":"float","level_recorded_at":"2026-09-24T22:24:42.659479+00:00","code_page":"/code/cosine-similarity","census":{"url":"/census/cosine-similarity","implementations":26,"papers":30,"distinct_outputs":11,"n_buckets":5,"bucket_key":null,"bucket_index":null,"bucket_label":null},"census_bucket_fragment":null,"licence":"MIT","inline_ok":true,"inline_ok_link":true,"withheld_clause":null,"mcp_get_code":{"code_sha256":"3055a2ccdae837c7"}},{"code_sha256":"3e6ff0c8581ee623","code_sha256_prefix":"3e6ff0c8581ee623","entry":"cosine_similarity","repo":"asahi417/relbert","repo_kind":"listed","path":"relbert/evaluation/relation_mapping.py","file_url":"https://github.com/asahi417/relbert/blob/HEAD/relbert/evaluation/relation_mapping.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"signature":"a:ndarray/1/float, b:ndarray/1/float","fixture_out_type":"float64","level_recorded_at":"2026-09-24T22:24:42.659479+00:00","retrieve_with":{"tool":"get_code","code_sha256":"3e6ff0c8581ee623"},"code_page":"/code/cosine-similarity","census":{"url":"/census/cosine-similarity","implementations":26,"papers":30,"distinct_outputs":11,"n_buckets":5,"bucket_key":null,"bucket_index":null,"bucket_label":null},"census_bucket_fragment":null,"licence":"MIT","inline_ok":true,"inline_ok_link":true,"withheld_clause":null,"mcp_get_code":{"code_sha256":"3e6ff0c8581ee623"}},{"code_sha256":"ffcac685bcb61388","code_sha256_prefix":"ffcac685bcb61388","entry":"create_image_prompt","repo":"neuralmagic/lm-evaluation-harness","repo_kind":"listed","path":"lm_eval/models/api_models.py","file_url":"https://github.com/neuralmagic/lm-evaluation-harness/blob/HEAD/lm_eval/models/api_models.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"signature":"imgs:list/1/Image, chat:list/1/dict, fmt:str/0/str","fixture_out_type":"list","level_recorded_at":"2026-09-25T09:36:09.529386+00:00","code_page":null,"census":null,"licence":"MIT","inline_ok":true,"inline_ok_link":true,"withheld_clause":null,"mcp_get_code":{"code_sha256":"ffcac685bcb61388"}},{"code_sha256":"a20ac4af0aec2beb","code_sha256_prefix":"a20ac4af0aec2beb","entry":"create_image_prompt","repo":"EleutherAI/lm_evaluation_harness","repo_kind":"listed","path":"lm_eval/models/api_models.py","file_url":"https://github.com/EleutherAI/lm_evaluation_harness/blob/HEAD/lm_eval/models/api_models.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"signature":"imgs:list/1/Image, chat:list/1/dict, fmt:str/0/str","fixture_out_type":"list","level_recorded_at":"2026-09-24T22:24:42.659479+00:00","code_page":null,"census":null,"licence":"MIT","inline_ok":true,"inline_ok_link":true,"withheld_clause":null,"mcp_get_code":{"code_sha256":"a20ac4af0aec2beb"}},{"code_sha256":"ecbbe3133a0f68b3","code_sha256_prefix":"ecbbe3133a0f68b3","entry":"create_image_prompt","repo":"juletx/lm-evaluation-harness","repo_kind":"listed","path":"lm_eval/models/api_models.py","file_url":"https://github.com/juletx/lm-evaluation-harness/blob/HEAD/lm_eval/models/api_models.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"signature":"imgs:list/1/Image, chat:list/1/dict, fmt:str/0/str","fixture_out_type":"list","level_recorded_at":"2026-09-24T22:24:42.659479+00:00","code_page":null,"census":null,"licence":"MIT","inline_ok":true,"inline_ok_link":true,"withheld_clause":null,"mcp_get_code":{"code_sha256":"ecbbe3133a0f68b3"}},{"code_sha256":"5307f83d52fe15d1","code_sha256_prefix":"5307f83d52fe15d1","entry":"custom_prompter","repo":"asahi417/relbert","repo_kind":"listed","path":"relbert/lm.py","file_url":"https://github.com/asahi417/relbert/blob/HEAD/relbert/lm.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"signature":"word_pair:tuple/1/str, template:str/0/str, mask_token:str/0/str","fixture_out_type":"str","level_recorded_at":"2026-09-24T22:24:42.659479+00:00","code_page":null,"census":null,"licence":"MIT","inline_ok":true,"inline_ok_link":true,"withheld_clause":null,"mcp_get_code":{"code_sha256":"5307f83d52fe15d1"}},{"code_sha256":"03edfbaef45e2c28","code_sha256_prefix":"03edfbaef45e2c28","entry":"default_gen_kwargs","repo":"neuralmagic/lm-evaluation-harness","repo_kind":"listed","path":"lm_eval/defaults.py","file_url":"https://github.com/neuralmagic/lm-evaluation-harness/blob/HEAD/lm_eval/defaults.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"signature":"until:NoneType/0/?, max_gen_toks:int/0/int","fixture_out_type":"dict","level_recorded_at":"2026-09-25T09:36:09.529386+00:00","code_page":null,"census":null,"licence":"MIT","inline_ok":true,"inline_ok_link":true,"withheld_clause":null,"mcp_get_code":{"code_sha256":"03edfbaef45e2c28"}},{"code_sha256":"57e732507bad40db","code_sha256_prefix":"57e732507bad40db","entry":"escaped_split","repo":"vilm-ai/viet-llm-eval","repo_kind":"listed","path":"lm_eval/utils.py","file_url":"https://github.com/vilm-ai/viet-llm-eval/blob/HEAD/lm_eval/utils.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"signature":"text:str/0/str, sep_char:str/0/str, maxsplit:int/0/int","fixture_out_type":"list","level_recorded_at":"2026-09-24T15:16:18.137819+00:00","code_page":"/code/escaped-split","census":null,"licence":"MIT","inline_ok":true,"inline_ok_link":true,"withheld_clause":null,"mcp_get_code":{"code_sha256":"57e732507bad40db"}},{"code_sha256":"3866cbd2c9f5fd58","code_sha256_prefix":"3866cbd2c9f5fd58","entry":"euclidean_distance","repo":"asahi417/relbert","repo_kind":"listed","path":"relbert/evaluation/analogy_questions.py","file_url":"https://github.com/asahi417/relbert/blob/HEAD/relbert/evaluation/analogy_questions.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"signature":"a:list/1/float, b:list/1/float","fixture_out_type":"float","level_recorded_at":"2026-09-24T22:24:42.659479+00:00","retrieve_with":{"tool":"get_code","code_sha256":"3866cbd2c9f5fd58"},"code_page":"/code/euclidean-distance","census":null,"licence":"MIT","inline_ok":true,"inline_ok_link":true,"withheld_clause":null,"mcp_get_code":{"code_sha256":"3866cbd2c9f5fd58"}},{"code_sha256":"b1e0531f8c17d5d6","code_sha256_prefix":"b1e0531f8c17d5d6","entry":"get_lr","repo":"ramanakshay/nanogpt","repo_kind":"listed","path":"src/algorithm/optimizer.py","file_url":"https://github.com/ramanakshay/nanogpt/blob/HEAD/src/algorithm/optimizer.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"signature":"it:int/0/int, warmup_iters:int/0/int, lr_decay_iters:int/0/int, min_lr:float/0/float, lr:float/0/float","fixture_out_type":"float","level_recorded_at":"2026-09-24T22:24:42.659479+00:00","code_page":"/code/get-lr","census":null,"licence":"MIT","inline_ok":true,"inline_ok_link":true,"withheld_clause":null,"mcp_get_code":{"code_sha256":"b1e0531f8c17d5d6"}},{"code_sha256":"53354f082d9addc6","code_sha256_prefix":"53354f082d9addc6","entry":"get_result","repo":"neuralmagic/lm-evaluation-harness","repo_kind":"listed","path":"lm_eval/models/gguf.py","file_url":"https://github.com/neuralmagic/lm-evaluation-harness/blob/HEAD/lm_eval/models/gguf.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"signature":"logprobs:dict/0/?, context_length:int/0/int","fixture_out_type":"tuple","level_recorded_at":"2026-09-24T15:16:18.137819+00:00","code_page":"/code/get-result","census":null,"licence":"MIT","inline_ok":false,"inline_ok_link":true,"withheld_clause":"this paper's link allows the body inline (MIT), but get_code serves the stored copy, first harvested from MrGGLS/BlockPruner, and GitHub detected no licence in that repository when it was checked","mcp_get_code":{"code_sha256":"53354f082d9addc6"}},{"code_sha256":"5decf8cec7692d54","code_sha256_prefix":"5decf8cec7692d54","entry":"get_result","repo":"insait-institute/lm-evaluation-harness-bg","repo_kind":"listed","path":"lm_eval/models/openai_completions.py","file_url":"https://github.com/insait-institute/lm-evaluation-harness-bg/blob/HEAD/lm_eval/models/openai_completions.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"signature":"response:MockResponse/0/?, ctxlen:int/0/int","fixture_out_type":"tuple","level_recorded_at":"2026-09-24T22:24:42.659479+00:00","code_page":"/code/get-result","census":null,"licence":"MIT","inline_ok":true,"inline_ok_link":true,"withheld_clause":null,"mcp_get_code":{"code_sha256":"5decf8cec7692d54"}},{"code_sha256":"89b22f3da55a855c","code_sha256_prefix":"89b22f3da55a855c","entry":"get_result","repo":"vilm-ai/viet-llm-eval","repo_kind":"listed","path":"lm_eval/models/openai_completions.py","file_url":"https://github.com/vilm-ai/viet-llm-eval/blob/HEAD/lm_eval/models/openai_completions.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"signature":"response:dict/0/?, ctxlen:int/0/int","fixture_out_type":"tuple","level_recorded_at":"2026-09-24T22:24:42.659479+00:00","code_page":"/code/get-result","census":null,"licence":"MIT","inline_ok":true,"inline_ok_link":true,"withheld_clause":null,"mcp_get_code":{"code_sha256":"89b22f3da55a855c"}},{"code_sha256":"00f5e7d9ee6636ba","code_sha256_prefix":"00f5e7d9ee6636ba","entry":"get_sample_size","repo":"Sypherd/lm-evaluation-harness","repo_kind":"listed","path":"lm_eval/evaluator_utils.py","file_url":"https://github.com/Sypherd/lm-evaluation-harness/blob/HEAD/lm_eval/evaluator_utils.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"signature":"task:MockTask/0/?, limit:float/0/float","fixture_out_type":"int","level_recorded_at":"2026-09-24T15:16:18.137819+00:00","code_page":"/code/get-sample-size","census":null,"licence":"MIT","inline_ok":true,"inline_ok_link":true,"withheld_clause":null,"mcp_get_code":{"code_sha256":"00f5e7d9ee6636ba"}},{"code_sha256":"5b315604fb608f3b","code_sha256_prefix":"5b315604fb608f3b","entry":"get_sample_size","repo":"opengptx/lm-evaluation-harness","repo_kind":"listed","path":"lm_eval/evaluator_utils.py","file_url":"https://github.com/opengptx/lm-evaluation-harness/blob/HEAD/lm_eval/evaluator_utils.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"signature":"task:MockTask/0/?, limit:float/0/float","fixture_out_type":"int","level_recorded_at":"2026-09-24T15:16:18.137819+00:00","code_page":"/code/get-sample-size","census":null,"licence":"MIT","inline_ok":true,"inline_ok_link":true,"withheld_clause":null,"mcp_get_code":{"code_sha256":"5b315604fb608f3b"}},{"code_sha256":"0db35c379a280bd8","code_sha256_prefix":"0db35c379a280bd8","entry":"get_sample_size","repo":"neuralmagic/lm-evaluation-harness","repo_kind":"listed","path":"lm_eval/evaluator_utils.py","file_url":"https://github.com/neuralmagic/lm-evaluation-harness/blob/HEAD/lm_eval/evaluator_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"signature":"task:MockTask/0/?, limit:float/0/float","fixture_out_type":"int","level_recorded_at":"2026-09-25T09:36:09.529386+00:00","code_page":"/code/get-sample-size","census":null,"licence":"MIT","inline_ok":true,"inline_ok_link":true,"withheld_clause":null,"mcp_get_code":{"code_sha256":"0db35c379a280bd8"}},{"code_sha256":"0fb487fd2495f33f","code_sha256_prefix":"0fb487fd2495f33f","entry":"get_sample_size","repo":"juletx/lm-evaluation-harness","repo_kind":"listed","path":"lm_eval/evaluator_utils.py","file_url":"https://github.com/juletx/lm-evaluation-harness/blob/HEAD/lm_eval/evaluator_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"signature":"task:MockTask/0/?, limit:float/0/float","fixture_out_type":"int","level_recorded_at":"2026-09-24T22:24:42.659479+00:00","code_page":"/code/get-sample-size","census":null,"licence":"MIT","inline_ok":true,"inline_ok_link":true,"withheld_clause":null,"mcp_get_code":{"code_sha256":"0fb487fd2495f33f"}},{"code_sha256":"0ee5d4d20cf990bd","code_sha256_prefix":"0ee5d4d20cf990bd","entry":"handle_arg_string","repo":"vilm-ai/viet-llm-eval","repo_kind":"listed","path":"lm_eval/utils.py","file_url":"https://github.com/vilm-ai/viet-llm-eval/blob/HEAD/lm_eval/utils.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":null,"behaviour_fingerprint":false,"signature":"arg:str/0/str","fixture_out_type":"bool","level_recorded_at":"2026-09-24T15:16:18.137819+00:00","code_page":null,"census":null,"licence":"MIT","inline_ok":true,"inline_ok_link":true,"withheld_clause":null,"mcp_get_code":{"code_sha256":"0ee5d4d20cf990bd"}},{"code_sha256":"20a7cc804eb22661","code_sha256_prefix":"20a7cc804eb22661","entry":"hash_args","repo":"Sypherd/lm-evaluation-harness","repo_kind":"listed","path":"lm_eval/api/model.py","file_url":"https://github.com/Sypherd/lm-evaluation-harness/blob/HEAD/lm_eval/api/model.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"signature":"attr:str/0/str, args:tuple/1/str","fixture_out_type":"str","retrieve_with":{"tool":"get_code","code_sha256":"20a7cc804eb22661"},"code_page":"/code/hash-args","census":null,"licence":"MIT","inline_ok":true,"inline_ok_link":true,"withheld_clause":null,"mcp_get_code":{"code_sha256":"20a7cc804eb22661"}},{"code_sha256":"ea06eaae4fc1eaf0","code_sha256_prefix":"ea06eaae4fc1eaf0","entry":"hash_args","repo":"vilm-ai/viet-llm-eval","repo_kind":"listed","path":"lm_eval/api/model.py","file_url":"https://github.com/vilm-ai/viet-llm-eval/blob/HEAD/lm_eval/api/model.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"signature":"attr:str/0/str, args:tuple/1/int","fixture_out_type":"str","level_recorded_at":"2026-09-22T14:31:46.068493+00:00","retrieve_with":{"tool":"get_code","code_sha256":"ea06eaae4fc1eaf0"},"code_page":"/code/hash-args","census":null,"licence":"MIT","inline_ok":false,"inline_ok_link":true,"withheld_clause":"this paper's link allows the body inline (MIT), but get_code serves the stored copy, first harvested from opengvlab/videochat-r1, and GitHub detected no licence in that repository when it was checked","mcp_get_code":{"code_sha256":"ea06eaae4fc1eaf0"}},{"code_sha256":"26519d22d8af6d48","code_sha256_prefix":"26519d22d8af6d48","entry":"hash_args","repo":"neuralmagic/lm-evaluation-harness","repo_kind":"listed","path":"lm_eval/api/model.py","file_url":"https://github.com/neuralmagic/lm-evaluation-harness/blob/HEAD/lm_eval/api/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"signature":"attr:str/0/str, args:list/1/str","fixture_out_type":"str","level_recorded_at":"2026-09-25T09:36:09.529386+00:00","retrieve_with":{"tool":"get_code","code_sha256":"26519d22d8af6d48"},"code_page":"/code/hash-args","census":null,"licence":"MIT","inline_ok":true,"inline_ok_link":true,"withheld_clause":null,"mcp_get_code":{"code_sha256":"26519d22d8af6d48"}},{"code_sha256":"72f5b387b64e9d9b","code_sha256_prefix":"72f5b387b64e9d9b","entry":"hash_args","repo":"EleutherAI/lm_evaluation_harness","repo_kind":"listed","path":"lm_eval/api/model.py","file_url":"https://github.com/EleutherAI/lm_evaluation_harness/blob/HEAD/lm_eval/api/model.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"signature":"attr:str/0/str, args:list/1/str","fixture_out_type":"str","level_recorded_at":"2026-09-24T22:24:42.659479+00:00","retrieve_with":{"tool":"get_code","code_sha256":"72f5b387b64e9d9b"},"code_page":"/code/hash-args","census":null,"licence":"MIT","inline_ok":true,"inline_ok_link":true,"withheld_clause":null,"mcp_get_code":{"code_sha256":"72f5b387b64e9d9b"}},{"code_sha256":"3e9d38ecacd03fa7","code_sha256_prefix":"3e9d38ecacd03fa7","entry":"hash_args","repo":"juletx/lm-evaluation-harness","repo_kind":"listed","path":"lm_eval/api/model.py","file_url":"https://github.com/juletx/lm-evaluation-harness/blob/HEAD/lm_eval/api/model.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"signature":"attr:str/0/str, args:list/1/int","fixture_out_type":"str","level_recorded_at":"2026-09-24T22:55:07.240920+00:00","code_page":"/code/hash-args","census":null,"licence":"MIT","inline_ok":true,"inline_ok_link":true,"withheld_clause":null,"mcp_get_code":{"code_sha256":"3e9d38ecacd03fa7"}},{"code_sha256":"2e6ad295f92e3d9d","code_sha256_prefix":"2e6ad295f92e3d9d","entry":"is_even","repo":"volcengine/vegiantmodel","repo_kind":"listed","path":"src/veGiantModel/engine/engine.py","file_url":"https://github.com/volcengine/vegiantmodel/blob/HEAD/src/veGiantModel/engine/engine.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"signature":"number:int/0/int","fixture_out_type":"bool","level_recorded_at":"2026-09-24T22:24:42.659479+00:00","retrieve_with":{"tool":"get_code","code_sha256":"2e6ad295f92e3d9d"},"code_page":null,"census":null,"licence":"Apache-2.0","inline_ok":true,"inline_ok_link":true,"withheld_clause":null,"mcp_get_code":{"code_sha256":"2e6ad295f92e3d9d"}},{"code_sha256":"deba15e38b692b23","code_sha256_prefix":"deba15e38b692b23","entry":"list_split","repo":"nlx-group/overlapy","repo_kind":"listed","path":"overlapy.py","file_url":"https://github.com/nlx-group/overlapy/blob/HEAD/overlapy.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"signature":"lst:list/1/int, sections:int/0/int","fixture_out_type":"list","level_recorded_at":"2026-09-24T22:24:42.659479+00:00","retrieve_with":{"tool":"get_code","code_sha256":"deba15e38b692b23"},"code_page":null,"census":null,"licence":"MIT","inline_ok":true,"inline_ok_link":true,"withheld_clause":null,"mcp_get_code":{"code_sha256":"deba15e38b692b23"}},{"code_sha256":"2ee40684ab58dcb9","code_sha256_prefix":"2ee40684ab58dcb9","entry":"load_object","repo":"VachanVY/gpt.jax","repo_kind":"listed","path":"model/model.py","file_url":"https://github.com/VachanVY/gpt.jax/blob/HEAD/model/model.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"signature":"path:str/0/str","fixture_out_type":"dict","level_recorded_at":"2026-09-24T22:24:42.659479+00:00","code_page":"/code/load-object","census":null,"licence":"MIT","inline_ok":true,"inline_ok_link":true,"withheld_clause":null,"mcp_get_code":{"code_sha256":"2ee40684ab58dcb9"}},{"code_sha256":"fdde66f5d1999801","code_sha256_prefix":"fdde66f5d1999801","entry":"loss_nce","repo":"asahi417/relbert","repo_kind":"listed","path":"relbert/util.py","file_url":"https://github.com/asahi417/relbert/blob/HEAD/relbert/util.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"signature":"tensor_positive:Tensor/2/float, tensor_negative:Tensor/2/float, temperature:float/0/float, info_loob:bool/0/bool, linear:callable/0/?, device:str/0/str","fixture_out_type":"Tensor","level_recorded_at":"2026-09-24T22:24:42.659479+00:00","code_page":null,"census":null,"licence":"MIT","inline_ok":true,"inline_ok_link":true,"withheld_clause":null,"mcp_get_code":{"code_sha256":"fdde66f5d1999801"}},{"code_sha256":"bd67f40255aa71d3","code_sha256_prefix":"bd67f40255aa71d3","entry":"loss_triplet","repo":"asahi417/relbert","repo_kind":"listed","path":"relbert/util.py","file_url":"https://github.com/asahi417/relbert/blob/HEAD/relbert/util.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"signature":"tensor_anchor:Tensor/2/float, tensor_positive:Tensor/2/float, tensor_negative:Tensor/2/float, tensor_positive_parent:Tensor/2/float, tensor_negative_parent:Tensor/2/float, margin:float/0/float, linear:callable/0/?, device:str/0/str","fixture_out_type":"Tensor","level_recorded_at":"2026-09-24T22:24:42.659479+00:00","code_page":null,"census":null,"licence":"MIT","inline_ok":true,"inline_ok_link":true,"withheld_clause":null,"mcp_get_code":{"code_sha256":"bd67f40255aa71d3"}},{"code_sha256":"7cacc3b113e10007","code_sha256_prefix":"7cacc3b113e10007","entry":"oa_chat_completion","repo":"vilm-ai/viet-llm-eval","repo_kind":"listed","path":"lm_eval/models/openai_completions.py","file_url":"https://github.com/vilm-ai/viet-llm-eval/blob/HEAD/lm_eval/models/openai_completions.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"signature_not_shown":true,"signature_not_shown_because":"signature_is_harness_object_repr","fixture_out_type":"Mock","level_recorded_at":"2026-09-24T22:55:07.240920+00:00","code_page":null,"census":null,"licence":"MIT","inline_ok":true,"inline_ok_link":true,"withheld_clause":null,"mcp_get_code":{"code_sha256":"7cacc3b113e10007"}},{"code_sha256":"cc82794b6ec72130","code_sha256_prefix":"cc82794b6ec72130","entry":"pad_vocab","repo":"karpathy/llm.c","repo_kind":"listed","path":"train_gpt2.py","file_url":"https://github.com/karpathy/llm.c/blob/HEAD/train_gpt2.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"signature":"tensor:Tensor/2/float, multiple:int/0/int, value:int/0/int","fixture_out_type":"Tensor","level_recorded_at":"2026-09-22T14:31:46.068493+00:00","code_page":null,"census":null,"licence":"MIT","inline_ok":true,"inline_ok_link":true,"withheld_clause":null,"mcp_get_code":{"code_sha256":"cc82794b6ec72130"}},{"code_sha256":"400ea4f64ccce5ef","code_sha256_prefix":"400ea4f64ccce5ef","entry":"parse_eval_args","repo":"Sypherd/lm-evaluation-harness","repo_kind":"listed","path":"lm_eval/__main__.py","file_url":"https://github.com/Sypherd/lm-evaluation-harness/blob/HEAD/lm_eval/__main__.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"signature":"parser:ArgumentParser/0/?","fixture_out_type":"Namespace","level_recorded_at":"2026-09-22T14:31:46.068493+00:00","code_page":"/code/parse-eval-args","census":null,"licence":"MIT","inline_ok":false,"inline_ok_link":true,"withheld_clause":"this paper's link allows the body inline (MIT), but get_code serves the stored copy, first harvested from BatsResearch/crosslingual-test-time-scaling, and GitHub detected no licence in that repository when it was checked","mcp_get_code":{"code_sha256":"400ea4f64ccce5ef"}},{"code_sha256":"e60f912106425ef4","code_sha256_prefix":"e60f912106425ef4","entry":"positional_function","repo":"abhaskumarsinha/MinimalGPT","repo_kind":"listed","path":"GPT.py","file_url":"https://github.com/abhaskumarsinha/MinimalGPT/blob/HEAD/GPT.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"signature":"words:int/0/int, embedding:int/0/int","fixture_out_type":"ndarray","level_recorded_at":"2026-09-22T14:31:46.068493+00:00","retrieve_with":{"tool":"get_code","code_sha256":"e60f912106425ef4"},"code_page":null,"census":null,"licence":"MIT","inline_ok":true,"inline_ok_link":true,"withheld_clause":null,"mcp_get_code":{"code_sha256":"e60f912106425ef4"}},{"code_sha256":"809bc9208dd99dce","code_sha256_prefix":"809bc9208dd99dce","entry":"prepare_print_tasks","repo":"opengptx/lm-evaluation-harness","repo_kind":"listed","path":"lm_eval/evaluator_utils.py","file_url":"https://github.com/opengptx/lm-evaluation-harness/blob/HEAD/lm_eval/evaluator_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"signature":"task_hierarchy:dict/0/?, results:dict/0/?, tab:int/0/int","fixture_out_type":"tuple","level_recorded_at":"2026-09-24T15:16:18.137819+00:00","code_page":null,"census":null,"licence":"MIT","inline_ok":true,"inline_ok_link":true,"withheld_clause":null,"mcp_get_code":{"code_sha256":"809bc9208dd99dce"}},{"code_sha256":"63b45afd720a808e","code_sha256_prefix":"63b45afd720a808e","entry":"rand_sample","repo":"asahi417/relbert","repo_kind":"listed","path":"relbert/trainer.py","file_url":"https://github.com/asahi417/relbert/blob/HEAD/relbert/trainer.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"signature":"_list:list/1/int","fixture_out_type":"int","level_recorded_at":"2026-09-24T22:24:42.659479+00:00","retrieve_with":{"tool":"get_code","code_sha256":"63b45afd720a808e"},"code_page":"/code/rand-sample","census":null,"licence":"MIT","inline_ok":true,"inline_ok_link":true,"withheld_clause":null,"mcp_get_code":{"code_sha256":"63b45afd720a808e"}},{"code_sha256":"de43c7cdbf869350","code_sha256_prefix":"de43c7cdbf869350","entry":"remove_none_pattern","repo":"sambanova/lm-evaluation-harness","repo_kind":"listed","path":"lm_eval/logging_utils.py","file_url":"https://github.com/sambanova/lm-evaluation-harness/blob/HEAD/lm_eval/logging_utils.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"signature":"input_string:str/0/str","fixture_out_type":"tuple","level_recorded_at":"2026-09-24T15:16:18.137819+00:00","retrieve_with":{"tool":"get_code","code_sha256":"de43c7cdbf869350"},"code_page":"/code/remove-none-pattern","census":null,"licence":"MIT","inline_ok":true,"inline_ok_link":true,"withheld_clause":null,"mcp_get_code":{"code_sha256":"de43c7cdbf869350"}},{"code_sha256":"c44830a7722ff5f7","code_sha256_prefix":"c44830a7722ff5f7","entry":"request_caching_arg_to_dict","repo":"Sypherd/lm-evaluation-harness","repo_kind":"listed","path":"lm_eval/evaluator.py","file_url":"https://github.com/Sypherd/lm-evaluation-harness/blob/HEAD/lm_eval/evaluator.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"signature":"cache_requests:str/0/str","fixture_out_type":"dict","retrieve_with":{"tool":"get_code","code_sha256":"c44830a7722ff5f7"},"code_page":"/code/request-caching-arg-to-dict","census":null,"licence":"MIT","inline_ok":true,"inline_ok_link":true,"withheld_clause":null,"mcp_get_code":{"code_sha256":"c44830a7722ff5f7"}},{"code_sha256":"99b9d9d7a4d6ca29","code_sha256_prefix":"99b9d9d7a4d6ca29","entry":"restore_list","repo":"asahi417/relbert","repo_kind":"listed","path":"relbert/list_keeper.py","file_url":"https://github.com/asahi417/relbert/blob/HEAD/relbert/list_keeper.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"signature":"original_list:list/1/int, flatten_value:list/1/int","fixture_out_type":"list","level_recorded_at":"2026-09-24T22:55:07.240920+00:00","retrieve_with":{"tool":"get_code","code_sha256":"99b9d9d7a4d6ca29"},"code_page":null,"census":null,"licence":"MIT","inline_ok":true,"inline_ok_link":true,"withheld_clause":null,"mcp_get_code":{"code_sha256":"99b9d9d7a4d6ca29"}},{"code_sha256":"c7a7a9dca42a1347","code_sha256_prefix":"c7a7a9dca42a1347","entry":"safe_json_load","repo":"asahi417/relbert","repo_kind":"listed","path":"relbert/relbert_cl/push_to_hub.py","file_url":"https://github.com/asahi417/relbert/blob/HEAD/relbert/relbert_cl/push_to_hub.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"signature":"_file:str/0/str","fixture_out_type":"dict","level_recorded_at":"2026-09-24T22:24:42.659479+00:00","retrieve_with":{"tool":"get_code","code_sha256":"c7a7a9dca42a1347"},"code_page":null,"census":null,"licence":"MIT","inline_ok":true,"inline_ok_link":true,"withheld_clause":null,"mcp_get_code":{"code_sha256":"c7a7a9dca42a1347"}},{"code_sha256":"dfed6e9291316231","code_sha256_prefix":"dfed6e9291316231","entry":"simple_parse_args_string","repo":"vilm-ai/viet-llm-eval","repo_kind":"listed","path":"lm_eval/utils.py","file_url":"https://github.com/vilm-ai/viet-llm-eval/blob/HEAD/lm_eval/utils.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"signature":"args_string:str/0/str","fixture_out_type":"dict","level_recorded_at":"2026-09-24T15:16:18.137819+00:00","retrieve_with":{"tool":"get_code","code_sha256":"dfed6e9291316231"},"code_page":"/code/simple-parse-args-string","census":null,"licence":"MIT","inline_ok":true,"inline_ok_link":true,"withheld_clause":null,"mcp_get_code":{"code_sha256":"dfed6e9291316231"}},{"code_sha256":"cabc905de3f72170","code_sha256_prefix":"cabc905de3f72170","entry":"steer","repo":"juletx/lm-evaluation-harness","repo_kind":"listed","path":"lm_eval/models/hf_steered.py","file_url":"https://github.com/juletx/lm-evaluation-harness/blob/HEAD/lm_eval/models/hf_steered.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"signature":"model:callable/0/?, hook_to_steer:dict/0/?","fixture_out_type":"_GeneratorContextManager","level_recorded_at":"2026-09-24T22:55:07.240920+00:00","code_page":null,"census":null,"licence":"MIT","inline_ok":true,"inline_ok_link":true,"withheld_clause":null,"mcp_get_code":{"code_sha256":"cabc905de3f72170"}},{"code_sha256":"11b1822611370e59","code_sha256_prefix":"11b1822611370e59","entry":"to_tensor","repo":"asahi417/relbert","repo_kind":"listed","path":"relbert/trainer.py","file_url":"https://github.com/asahi417/relbert/blob/HEAD/relbert/trainer.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"signature":"name:str/0/str, data:list/1/list","fixture_out_type":"Tensor","level_recorded_at":"2026-09-24T22:55:07.240920+00:00","code_page":"/code/to-tensor","census":{"url":"/census/to-tensor","implementations":8,"papers":8,"distinct_outputs":3,"n_buckets":3,"bucket_key":null,"bucket_index":null,"bucket_label":null},"census_bucket_fragment":null,"licence":"MIT","inline_ok":true,"inline_ok_link":true,"withheld_clause":null,"mcp_get_code":{"code_sha256":"11b1822611370e59"}},{"code_sha256":"3bb8fcfcb4c12121","code_sha256_prefix":"3bb8fcfcb4c12121","entry":"add_byte_giant_model_customize_args","repo":"volcengine/vegiantmodel","repo_kind":"listed","path":"src/veGiantModel/initialize.py","file_url":"https://github.com/volcengine/vegiantmodel/blob/HEAD/src/veGiantModel/initialize.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"code_page":null,"census":null,"licence":"Apache-2.0","inline_ok":true,"inline_ok_link":true,"withheld_clause":null,"mcp_get_code":{"code_sha256":"3bb8fcfcb4c12121"}},{"code_sha256":"e66a197775436632","code_sha256_prefix":"e66a197775436632","entry":"bps_recv","repo":"volcengine/vegiantmodel","repo_kind":"listed","path":"src/veGiantModel/engine/p2p.py","file_url":"https://github.com/volcengine/vegiantmodel/blob/HEAD/src/veGiantModel/engine/p2p.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"code_page":null,"census":null,"licence":"Apache-2.0","inline_ok":true,"inline_ok_link":true,"withheld_clause":null,"mcp_get_code":{"code_sha256":"e66a197775436632"}},{"code_sha256":"7ca8b4cd5f558e02","code_sha256_prefix":"7ca8b4cd5f558e02","entry":"bps_send","repo":"volcengine/vegiantmodel","repo_kind":"listed","path":"src/veGiantModel/engine/p2p.py","file_url":"https://github.com/volcengine/vegiantmodel/blob/HEAD/src/veGiantModel/engine/p2p.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"code_page":null,"census":null,"licence":"Apache-2.0","inline_ok":true,"inline_ok_link":true,"withheld_clause":null,"mcp_get_code":{"code_sha256":"7ca8b4cd5f558e02"}},{"code_sha256":"9dd643ee2669464c","code_sha256_prefix":"9dd643ee2669464c","entry":"build_model","repo":"VachanVY/gpt.jax","repo_kind":"listed","path":"model/model.py","file_url":"https://github.com/VachanVY/gpt.jax/blob/HEAD/model/model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"code_page":"/code/build-model","census":null,"licence":"MIT","inline_ok":true,"inline_ok_link":true,"withheld_clause":null,"mcp_get_code":{"code_sha256":"9dd643ee2669464c"}},{"code_sha256":"e1d89d7579eaaeec","code_sha256_prefix":"e1d89d7579eaaeec","entry":"em_accuracy_helper","repo":null,"repo_kind":"listed","path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"code_page":null,"census":null,"placed_by_identical_code":{"repo":"tonyzhaozh/few-shot-learning","path":"run_extraction.py","file_url":"https://github.com/tonyzhaozh/few-shot-learning/blob/HEAD/run_extraction.py","papers":["2102.09690"],"is":"identical code Syntology harvested from this repository for another paper; this paper's link records no repository, path or licence of its own"},"licence":null,"inline_ok":false,"inline_ok_link":false,"withheld_clause":"this paper's copy of the code was not recorded (its link names no repository or source path), so no licence was read for it","mcp_get_code":{"code_sha256":"e1d89d7579eaaeec"}},{"code_sha256":"7753000f9482cac5","code_sha256_prefix":"7753000f9482cac5","entry":"eval_accuracy","repo":null,"repo_kind":"listed","path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"code_page":"/code/eval-accuracy","census":null,"placed_by_identical_code":{"repo":"tonyzhaozh/few-shot-learning","path":"run_classification.py","file_url":"https://github.com/tonyzhaozh/few-shot-learning/blob/HEAD/run_classification.py","papers":["2102.09690"],"is":"identical code Syntology harvested from this repository for another paper; this paper's link records no repository, path or licence of its own"},"licence":null,"inline_ok":false,"inline_ok_link":false,"withheld_clause":"this paper's copy of the code was not recorded (its link names no repository or source path), so no licence was read for it","mcp_get_code":{"code_sha256":"7753000f9482cac5"}},{"code_sha256":"34f65890e3e58ec4","code_sha256_prefix":"34f65890e3e58ec4","entry":"evaluate_analogy","repo":"asahi417/relbert","repo_kind":"listed","path":"relbert/evaluation/analogy_questions.py","file_url":"https://github.com/asahi417/relbert/blob/HEAD/relbert/evaluation/analogy_questions.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"code_page":null,"census":null,"licence":"MIT","inline_ok":true,"inline_ok_link":true,"withheld_clause":null,"mcp_get_code":{"code_sha256":"34f65890e3e58ec4"}},{"code_sha256":"87cfebcf19ab803b","code_sha256_prefix":"87cfebcf19ab803b","entry":"evaluate_classification","repo":"asahi417/relbert","repo_kind":"listed","path":"relbert/evaluation/lexical_relation_classification.py","file_url":"https://github.com/asahi417/relbert/blob/HEAD/relbert/evaluation/lexical_relation_classification.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"code_page":"/code/evaluate-classification","census":null,"licence":"MIT","inline_ok":true,"inline_ok_link":true,"withheld_clause":null,"mcp_get_code":{"code_sha256":"87cfebcf19ab803b"}},{"code_sha256":"c7271f2f2ce3c4fa","code_sha256_prefix":"c7271f2f2ce3c4fa","entry":"evaluate_relation_mapping","repo":"asahi417/relbert","repo_kind":"listed","path":"relbert/evaluation/relation_mapping.py","file_url":"https://github.com/asahi417/relbert/blob/HEAD/relbert/evaluation/relation_mapping.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"code_page":null,"census":null,"licence":"MIT","inline_ok":true,"inline_ok_link":true,"withheld_clause":null,"mcp_get_code":{"code_sha256":"c7271f2f2ce3c4fa"}},{"code_sha256":"c56b548854fce3e6","code_sha256_prefix":"c56b548854fce3e6","entry":"flatten_list","repo":"asahi417/relbert","repo_kind":"listed","path":"relbert/list_keeper.py","file_url":"https://github.com/asahi417/relbert/blob/HEAD/relbert/list_keeper.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"code_page":"/code/flatten-list","census":null,"licence":"MIT","inline_ok":true,"inline_ok_link":true,"withheld_clause":null,"mcp_get_code":{"code_sha256":"c56b548854fce3e6"}},{"code_sha256":"f6825b4eb5fe78c8","code_sha256_prefix":"f6825b4eb5fe78c8","entry":"get_commit_from_path","repo":"sambanova/lm-evaluation-harness","repo_kind":"listed","path":"lm_eval/logging_utils.py","file_url":"https://github.com/sambanova/lm-evaluation-harness/blob/HEAD/lm_eval/logging_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"code_page":null,"census":null,"licence":"MIT","inline_ok":true,"inline_ok_link":true,"withheld_clause":null,"mcp_get_code":{"code_sha256":"f6825b4eb5fe78c8"}},{"code_sha256":"306b51e6dcfb30ef","code_sha256_prefix":"306b51e6dcfb30ef","entry":"get_distribution_scope","repo":"abhaskumarsinha/Corpus2GPT","repo_kind":"listed","path":"training_utils/distribution_utils.py","file_url":"https://github.com/abhaskumarsinha/Corpus2GPT/blob/HEAD/training_utils/distribution_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"code_page":null,"census":null,"licence":"Apache-2.0","inline_ok":true,"inline_ok_link":true,"withheld_clause":null,"mcp_get_code":{"code_sha256":"306b51e6dcfb30ef"}},{"code_sha256":"77347b697708c203","code_sha256_prefix":"77347b697708c203","entry":"get_label_probs","repo":null,"repo_kind":"listed","path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"code_page":null,"census":null,"placed_by_identical_code":{"repo":"tonyzhaozh/few-shot-learning","path":"run_classification.py","file_url":"https://github.com/tonyzhaozh/few-shot-learning/blob/HEAD/run_classification.py","papers":["2102.09690"],"is":"identical code Syntology harvested from this repository for another paper; this paper's link records no repository, path or licence of its own"},"licence":null,"inline_ok":false,"inline_ok_link":false,"withheld_clause":"this paper's copy of the code was not recorded (its link names no repository or source path), so no licence was read for it","mcp_get_code":{"code_sha256":"77347b697708c203"}},{"code_sha256":"d09226e56c2085ea","code_sha256_prefix":"d09226e56c2085ea","entry":"get_p_content_free","repo":null,"repo_kind":"listed","path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"code_page":"/code/get-p-content-free","census":null,"placed_by_identical_code":{"repo":"tonyzhaozh/few-shot-learning","path":"run_classification.py","file_url":"https://github.com/tonyzhaozh/few-shot-learning/blob/HEAD/run_classification.py","papers":["2102.09690"],"is":"identical code Syntology harvested from this repository for another paper; this paper's link records no repository, path or licence of its own"},"licence":null,"inline_ok":false,"inline_ok_link":false,"withheld_clause":"this paper's copy of the code was not recorded (its link names no repository or source path), so no licence was read for it","mcp_get_code":{"code_sha256":"d09226e56c2085ea"}},{"code_sha256":"8e3012dd13fbb68a","code_sha256_prefix":"8e3012dd13fbb68a","entry":"get_readme","repo":"asahi417/relbert","repo_kind":"listed","path":"relbert/relbert_cl/readme_template.py","file_url":"https://github.com/asahi417/relbert/blob/HEAD/relbert/relbert_cl/readme_template.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"code_page":null,"census":null,"licence":"MIT","inline_ok":true,"inline_ok_link":true,"withheld_clause":null,"mcp_get_code":{"code_sha256":"8e3012dd13fbb68a"}},{"code_sha256":"4ff6d4e326d3c0db","code_sha256_prefix":"4ff6d4e326d3c0db","entry":"load_model","repo":"mbzuai-paris/lm-evaluation-harness-atlas-chat","repo_kind":"listed","path":"lm_eval/models/nemo_lm.py","file_url":"https://github.com/mbzuai-paris/lm-evaluation-harness-atlas-chat/blob/HEAD/lm_eval/models/nemo_lm.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"code_page":"/code/load-model","census":null,"licence":"MIT","inline_ok":true,"inline_ok_link":true,"withheld_clause":null,"mcp_get_code":{"code_sha256":"4ff6d4e326d3c0db"}},{"code_sha256":"9d752127437e6b8b","code_sha256_prefix":"9d752127437e6b8b","entry":"run_inference_one_model","repo":"vilm-ai/viet-llm-eval","repo_kind":"listed","path":"lm_eval/models/vllm_causallms.py","file_url":"https://github.com/vilm-ai/viet-llm-eval/blob/HEAD/lm_eval/models/vllm_causallms.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"code_page":null,"census":null,"licence":"MIT","inline_ok":true,"inline_ok_link":true,"withheld_clause":null,"mcp_get_code":{"code_sha256":"9d752127437e6b8b"}},{"code_sha256":"2fe668161976f857","code_sha256_prefix":"2fe668161976f857","entry":"send","repo":"volcengine/vegiantmodel","repo_kind":"listed","path":"src/veGiantModel/engine/p2p.py","file_url":"https://github.com/volcengine/vegiantmodel/blob/HEAD/src/veGiantModel/engine/p2p.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"code_page":null,"census":null,"licence":"Apache-2.0","inline_ok":true,"inline_ok_link":true,"withheld_clause":null,"mcp_get_code":{"code_sha256":"2fe668161976f857"}},{"code_sha256":"5036094f3af0dc56","code_sha256_prefix":"5036094f3af0dc56","entry":"setup_distributed_environment","repo":"mbzuai-paris/lm-evaluation-harness-atlas-chat","repo_kind":"listed","path":"lm_eval/models/nemo_lm.py","file_url":"https://github.com/mbzuai-paris/lm-evaluation-harness-atlas-chat/blob/HEAD/lm_eval/models/nemo_lm.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"code_page":null,"census":null,"licence":"MIT","inline_ok":true,"inline_ok_link":true,"withheld_clause":null,"mcp_get_code":{"code_sha256":"5036094f3af0dc56"}},{"code_sha256":"8cb749113ad26cc3","code_sha256_prefix":"8cb749113ad26cc3","entry":"wrap_constant_batch_size","repo":"insait-institute/lm-evaluation-harness-bg","repo_kind":"listed","path":"lm_eval/models/neuron_optimum.py","file_url":"https://github.com/insait-institute/lm-evaluation-harness-bg/blob/HEAD/lm_eval/models/neuron_optimum.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"code_page":null,"census":null,"licence":"MIT","inline_ok":true,"inline_ok_link":true,"withheld_clause":null,"mcp_get_code":{"code_sha256":"8cb749113ad26cc3"}}]},"code_links_note":null,"arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"af9daeae356ce83ae073bef5cbe0988a9f797d3c477aa36374ded14e7d141d58","record_changed_at":"2026-10-05","record_changed_at_basis":"changed"}