{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/dataset/beir","name":"BEIR","full_name":"Benchmarking IR","description_markdown":"**BEIR** (Benchmarking IR) is a heterogeneous benchmark containing different information retrieval (IR) tasks. Through BEIR, it is possible to systematically study the zero-shot generalization capabilities of multiple neural retrieval approaches.\r\n\r\nThe benchmark contains a total of 9 information retrieval tasks (Fact Checking, Citation Prediction, Duplicate Question Retrieval, Argument Retrieval, News Retrieval, Question Answering, Tweet Retrieval, Biomedical IR, Entity Retrieval) from 19 different datasets:\r\n\r\n* [MS MARCO](ms-marco)\r\n* [TREC-COVID](trec-covid)\r\n* [NFCorpus](nfcorpus)\r\n* [BioASQ](bioasq)\r\n* [Natural Questions](natural-questions)\r\n* [HotpotQA](hotpotqa)\r\n* FiQA-2018\r\n* [Signal-1M](signal-1m-related-tweets)\r\n* [TREC-News](trec-news-1)\r\n* ArguAna\r\n* [Touche 2020](webis-touche-2020)\r\n* [CQADupStack](cqadupstack)\r\n* [Quora Question Pairs](quora-question-pairs)\r\n* [DBPedia](dbpedia)\r\n* [SciDocs](scidocs)\r\n* [FEVER](fever)\r\n* [Climate-FEVER](climate-fever)\r\n* [SciFact](scifact)\r\n* [Robust04](robust04)","description_withheld":null,"homepage":"https://github.com/UKPLab/beir","introduced_date":"2021-04-17","introduced_date_note":null,"introduced_by":{"paper":"/paper/beir-a-heterogenous-benchmark-for-zero-shot","title":"BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models","first_author":"Nandan Thakur","url":null},"license":{"name":"Multiple licenses","url":null},"modalities":[{"name":"Texts","url":"/datasets/modality/texts"}],"tasks":[{"name":"Question Answering","url":"/task/question-answering","datasets_with_task":"/datasets/task/question-answering"},{"name":"Passage Retrieval","url":"/task/passage-retrieval","datasets_with_task":"/datasets/task/passage-retrieval"},{"name":"Zero-shot Text Search","url":"/task/zero-shot-text-search","datasets_with_task":"/datasets/task/zero-shot-text-search"},{"name":"Fact Checking","url":"/task/fact-checking","datasets_with_task":"/datasets/task/fact-checking"},{"name":"Biomedical Information Retrieval","url":"/task/biomedical-information-retrieval","datasets_with_task":"/datasets/task/biomedical-information-retrieval"},{"name":"Citation Prediction","url":"/task/citation-prediction","datasets_with_task":"/datasets/task/citation-prediction"},{"name":"Argument Retrieval","url":"/task/argument-retrieval","datasets_with_task":"/datasets/task/argument-retrieval"},{"name":"Duplicate-Question Retrieval","url":"/task/duplicate-question-retrieval","datasets_with_task":"/datasets/task/duplicate-question-retrieval"},{"name":"Entity Retrieval","url":"/task/entity-retrieval","datasets_with_task":"/datasets/task/entity-retrieval"},{"name":"Tweet Retrieval","url":"/task/tweet-retrieval","datasets_with_task":"/datasets/task/tweet-retrieval"},{"name":"News Retrieval","url":"/task/news-retrieval","datasets_with_task":"/datasets/task/news-retrieval"},{"name":"Zero Shot on BEIR (Inference Free Model)","url":"/task/zero-shot-on-beir-inference-free-model","datasets_with_task":"/datasets/task/zero-shot-on-beir-inference-free-model"}],"languages":[{"name":"English","url":"/datasets/language/english"}],"variants":["CQADupStack (BEIR)","TREC-NEWS (BEIR)","TREC-COVID (BEIR)","Tóuche-2020 (BEIR)","Signal-1M (RT) (BEIR)","SciFact (BEIR)","SciDocs (BEIR)","Quora (BEIR)","NQ (BEIR)","NFCorpus (BEIR)","MSMARCO (BEIR)","HotpotQA (BEIR)","FiQA-2018 (BEIR)","FEVER (BEIR)","DBpedia (BEIR)","CLIMATE-FEVER (BEIR)","BioASQ (BEIR)","ArguAna (BEIR)","BEIR"],"data_loaders":[{"repo":"https://github.com/huggingface/datasets","url":"https://huggingface.co/datasets/income/fever-top-20-gen-queries","frameworks":["tf","pytorch","jax"]},{"repo":"https://github.com/huggingface/datasets","url":"https://huggingface.co/datasets/BeIR/beir","frameworks":["tf","pytorch","jax"]},{"repo":"https://github.com/huggingface/datasets","url":"https://huggingface.co/datasets/BeIR/beir-corpus","frameworks":["tf","pytorch","jax"]},{"repo":"https://github.com/huggingface/datasets","url":"https://huggingface.co/datasets/income/hotpotqa-top-20-gen-queries","frameworks":["tf","pytorch","jax"]},{"repo":"https://github.com/huggingface/datasets","url":"https://huggingface.co/datasets/income/dbpedia-entity-top-20-gen-queries","frameworks":["tf","pytorch","jax"]},{"repo":"https://github.com/huggingface/datasets","url":"https://huggingface.co/datasets/BeIR/fever-qrels","frameworks":["tf","pytorch","jax"]},{"repo":"https://github.com/huggingface/datasets","url":"https://huggingface.co/datasets/BeIR/climate-fever-qrels","frameworks":["tf","pytorch","jax"]},{"repo":"https://github.com/huggingface/datasets","url":"https://huggingface.co/datasets/BeIR/trec-covid","frameworks":["tf","pytorch","jax"]},{"repo":"https://github.com/huggingface/datasets","url":"https://huggingface.co/datasets/BeIR/scifact","frameworks":["tf","pytorch","jax"]},{"repo":"https://github.com/huggingface/datasets","url":"https://huggingface.co/datasets/BeIR/nfcorpus","frameworks":["tf","pytorch","jax"]},{"repo":"https://github.com/huggingface/datasets","url":"https://huggingface.co/datasets/BeIR/msmarco","frameworks":["tf","pytorch","jax"]},{"repo":"https://github.com/huggingface/datasets","url":"https://huggingface.co/datasets/BeIR/fiqa","frameworks":["tf","pytorch","jax"]},{"repo":"https://github.com/huggingface/datasets","url":"https://huggingface.co/datasets/BeIR/nq","frameworks":["tf","pytorch","jax"]},{"repo":"https://github.com/huggingface/datasets","url":"https://huggingface.co/datasets/BeIR/hotpotqa","frameworks":["tf","pytorch","jax"]},{"repo":"https://github.com/huggingface/datasets","url":"https://huggingface.co/datasets/BeIR/arguana","frameworks":["tf","pytorch","jax"]},{"repo":"https://github.com/huggingface/datasets","url":"https://huggingface.co/datasets/BeIR/webis-touche2020","frameworks":["tf","pytorch","jax"]},{"repo":"https://github.com/huggingface/datasets","url":"https://huggingface.co/datasets/BeIR/quora","frameworks":["tf","pytorch","jax"]},{"repo":"https://github.com/huggingface/datasets","url":"https://huggingface.co/datasets/BeIR/dbpedia-entity","frameworks":["tf","pytorch","jax"]},{"repo":"https://github.com/huggingface/datasets","url":"https://huggingface.co/datasets/BeIR/scidocs","frameworks":["tf","pytorch","jax"]},{"repo":"https://github.com/huggingface/datasets","url":"https://huggingface.co/datasets/BeIR/fever","frameworks":["tf","pytorch","jax"]},{"repo":"https://github.com/huggingface/datasets","url":"https://huggingface.co/datasets/BeIR/climate-fever","frameworks":["tf","pytorch","jax"]},{"repo":"https://github.com/huggingface/datasets","url":"https://huggingface.co/datasets/BeIR/trec-covid-queries","frameworks":["tf","pytorch","jax"]},{"repo":"https://github.com/huggingface/datasets","url":"https://huggingface.co/datasets/BeIR/trec-covid-qrels","frameworks":["tf","pytorch","jax"]},{"repo":"https://github.com/huggingface/datasets","url":"https://huggingface.co/datasets/BeIR/scifact-queries","frameworks":["tf","pytorch","jax"]},{"repo":"https://github.com/huggingface/datasets","url":"https://huggingface.co/datasets/BeIR/nfcorpus-queries","frameworks":["tf","pytorch","jax"]},{"repo":"https://github.com/huggingface/datasets","url":"https://huggingface.co/datasets/BeIR/msmarco-queries","frameworks":["tf","pytorch","jax"]},{"repo":"https://github.com/huggingface/datasets","url":"https://huggingface.co/datasets/BeIR/hotpotqa-queries","frameworks":["tf","pytorch","jax"]},{"repo":"https://github.com/huggingface/datasets","url":"https://huggingface.co/datasets/BeIR/fiqa-queries","frameworks":["tf","pytorch","jax"]},{"repo":"https://github.com/huggingface/datasets","url":"https://huggingface.co/datasets/BeIR/arguana-queries","frameworks":["tf","pytorch","jax"]},{"repo":"https://github.com/huggingface/datasets","url":"https://huggingface.co/datasets/BeIR/webis-touche2020-queries","frameworks":["tf","pytorch","jax"]},{"repo":"https://github.com/huggingface/datasets","url":"https://huggingface.co/datasets/BeIR/quora-queries","frameworks":["tf","pytorch","jax"]},{"repo":"https://github.com/huggingface/datasets","url":"https://huggingface.co/datasets/BeIR/dbpedia-entity-queries","frameworks":["tf","pytorch","jax"]},{"repo":"https://github.com/huggingface/datasets","url":"https://huggingface.co/datasets/BeIR/scidocs-queries","frameworks":["tf","pytorch","jax"]},{"repo":"https://github.com/huggingface/datasets","url":"https://huggingface.co/datasets/BeIR/fever-queries","frameworks":["tf","pytorch","jax"]},{"repo":"https://github.com/huggingface/datasets","url":"https://huggingface.co/datasets/BeIR/climate-fever-queries","frameworks":["tf","pytorch","jax"]},{"repo":"https://github.com/huggingface/datasets","url":"https://huggingface.co/datasets/BeIR/scifact-qrels","frameworks":["tf","pytorch","jax"]},{"repo":"https://github.com/huggingface/datasets","url":"https://huggingface.co/datasets/BeIR/nfcorpus-qrels","frameworks":["tf","pytorch","jax"]},{"repo":"https://github.com/huggingface/datasets","url":"https://huggingface.co/datasets/BeIR/msmarco-qrels","frameworks":["tf","pytorch","jax"]},{"repo":"https://github.com/huggingface/datasets","url":"https://huggingface.co/datasets/BeIR/hotpotqa-qrels","frameworks":["tf","pytorch","jax"]},{"repo":"https://github.com/huggingface/datasets","url":"https://huggingface.co/datasets/BeIR/fiqa-qrels","frameworks":["tf","pytorch","jax"]},{"repo":"https://github.com/huggingface/datasets","url":"https://huggingface.co/datasets/BeIR/arguana-qrels","frameworks":["tf","pytorch","jax"]},{"repo":"https://github.com/huggingface/datasets","url":"https://huggingface.co/datasets/BeIR/webis-touche2020-qrels","frameworks":["tf","pytorch","jax"]},{"repo":"https://github.com/huggingface/datasets","url":"https://huggingface.co/datasets/BeIR/quora-qrels","frameworks":["tf","pytorch","jax"]},{"repo":"https://github.com/huggingface/datasets","url":"https://huggingface.co/datasets/BeIR/dbpedia-entity-qrels","frameworks":["tf","pytorch","jax"]},{"repo":"https://github.com/huggingface/datasets","url":"https://huggingface.co/datasets/BeIR/scidocs-qrels","frameworks":["tf","pytorch","jax"]},{"repo":"https://github.com/huggingface/datasets","url":"https://huggingface.co/datasets/BeIR/nq-qrels","frameworks":["tf","pytorch","jax"]},{"repo":"https://github.com/huggingface/datasets","url":"https://huggingface.co/datasets/BeIR/arguana-generated-queries","frameworks":["tf","pytorch","jax"]},{"repo":"https://github.com/huggingface/datasets","url":"https://huggingface.co/datasets/BeIR/climate-fever-generated-queries","frameworks":["tf","pytorch","jax"]},{"repo":"https://github.com/huggingface/datasets","url":"https://huggingface.co/datasets/BeIR/dbpedia-entity-generated-queries","frameworks":["tf","pytorch","jax"]},{"repo":"https://github.com/huggingface/datasets","url":"https://huggingface.co/datasets/BeIR/fever-generated-queries","frameworks":["tf","pytorch","jax"]},{"repo":"https://github.com/huggingface/datasets","url":"https://huggingface.co/datasets/BeIR/nfcorpus-generated-queries","frameworks":["tf","pytorch","jax"]},{"repo":"https://github.com/huggingface/datasets","url":"https://huggingface.co/datasets/BeIR/scifact-generated-queries","frameworks":["tf","pytorch","jax"]},{"repo":"https://github.com/huggingface/datasets","url":"https://huggingface.co/datasets/BeIR/scidocs-generated-queries","frameworks":["tf","pytorch","jax"]},{"repo":"https://github.com/huggingface/datasets","url":"https://huggingface.co/datasets/BeIR/fiqa-generated-queries","frameworks":["tf","pytorch","jax"]},{"repo":"https://github.com/huggingface/datasets","url":"https://huggingface.co/datasets/BeIR/trec-covid-generated-queries","frameworks":["tf","pytorch","jax"]},{"repo":"https://github.com/huggingface/datasets","url":"https://huggingface.co/datasets/BeIR/trec-news-generated-queries","frameworks":["tf","pytorch","jax"]},{"repo":"https://github.com/huggingface/datasets","url":"https://huggingface.co/datasets/BeIR/cqadupstack-qrels","frameworks":["tf","pytorch","jax"]},{"repo":"https://github.com/huggingface/datasets","url":"https://huggingface.co/datasets/BeIR/webis-touche2020-generated-queries","frameworks":["tf","pytorch","jax"]},{"repo":"https://github.com/huggingface/datasets","url":"https://huggingface.co/datasets/BeIR/robust04-generated-queries","frameworks":["tf","pytorch","jax"]},{"repo":"https://github.com/huggingface/datasets","url":"https://huggingface.co/datasets/BeIR/signal1m-generated-queries","frameworks":["tf","pytorch","jax"]},{"repo":"https://github.com/huggingface/datasets","url":"https://huggingface.co/datasets/BeIR/quora-generated-queries","frameworks":["tf","pytorch","jax"]},{"repo":"https://github.com/huggingface/datasets","url":"https://huggingface.co/datasets/BeIR/cqadupstack-generated-queries","frameworks":["tf","pytorch","jax"]},{"repo":"https://github.com/huggingface/datasets","url":"https://huggingface.co/datasets/BeIR/nq-generated-queries","frameworks":["tf","pytorch","jax"]},{"repo":"https://github.com/huggingface/datasets","url":"https://huggingface.co/datasets/BeIR/hotpotqa-generated-queries","frameworks":["tf","pytorch","jax"]},{"repo":"https://github.com/huggingface/datasets","url":"https://huggingface.co/datasets/BeIR/bioasq-generated-queries","frameworks":["tf","pytorch","jax"]},{"repo":"https://github.com/huggingface/datasets","url":"https://huggingface.co/datasets/income/climate-fever-top-20-gen-queries","frameworks":["tf","pytorch","jax"]},{"repo":"https://github.com/huggingface/datasets","url":"https://huggingface.co/datasets/income/signal1m-top-20-gen-queries","frameworks":["tf","pytorch","jax"]},{"repo":"https://github.com/huggingface/datasets","url":"https://huggingface.co/datasets/income/nq-top-20-gen-queries","frameworks":["tf","pytorch","jax"]},{"repo":"https://github.com/huggingface/datasets","url":"https://huggingface.co/datasets/income/cqadupstack-android-top-20-gen-queries","frameworks":["tf","pytorch","jax"]},{"repo":"https://github.com/huggingface/datasets","url":"https://huggingface.co/datasets/income/cqadupstack-english-top-20-gen-queries","frameworks":["tf","pytorch","jax"]},{"repo":"https://github.com/huggingface/datasets","url":"https://huggingface.co/datasets/income/cqadupstack-gaming-top-20-gen-queries","frameworks":["tf","pytorch","jax"]},{"repo":"https://github.com/huggingface/datasets","url":"https://huggingface.co/datasets/income/cqadupstack-gis-top-20-gen-queries","frameworks":["tf","pytorch","jax"]},{"repo":"https://github.com/huggingface/datasets","url":"https://huggingface.co/datasets/income/cqadupstack-mathematica-top-20-gen-queries","frameworks":["tf","pytorch","jax"]},{"repo":"https://github.com/huggingface/datasets","url":"https://huggingface.co/datasets/income/cqadupstack-physics-top-20-gen-queries","frameworks":["tf","pytorch","jax"]},{"repo":"https://github.com/huggingface/datasets","url":"https://huggingface.co/datasets/income/cqadupstack-programmers-top-20-gen-queries","frameworks":["tf","pytorch","jax"]},{"repo":"https://github.com/huggingface/datasets","url":"https://huggingface.co/datasets/income/cqadupstack-stats-top-20-gen-queries","frameworks":["tf","pytorch","jax"]},{"repo":"https://github.com/huggingface/datasets","url":"https://huggingface.co/datasets/income/cqadupstack-tex-top-20-gen-queries","frameworks":["tf","pytorch","jax"]},{"repo":"https://github.com/huggingface/datasets","url":"https://huggingface.co/datasets/income/cqadupstack-unix-top-20-gen-queries","frameworks":["tf","pytorch","jax"]},{"repo":"https://github.com/huggingface/datasets","url":"https://huggingface.co/datasets/income/cqadupstack-webmasters-top-20-gen-queries","frameworks":["tf","pytorch","jax"]},{"repo":"https://github.com/huggingface/datasets","url":"https://huggingface.co/datasets/income/cqadupstack-wordpress-top-20-gen-queries","frameworks":["tf","pytorch","jax"]},{"repo":"https://github.com/huggingface/datasets","url":"https://huggingface.co/datasets/income/bioasq-top-20-gen-queries","frameworks":["tf","pytorch","jax"]},{"repo":"https://github.com/huggingface/datasets","url":"https://huggingface.co/datasets/income/nfcorpus-top-20-gen-queries","frameworks":["tf","pytorch","jax"]},{"repo":"https://github.com/huggingface/datasets","url":"https://huggingface.co/datasets/income/fiqa-top-20-gen-queries","frameworks":["tf","pytorch","jax"]},{"repo":"https://github.com/huggingface/datasets","url":"https://huggingface.co/datasets/income/scifact-top-20-gen-queries","frameworks":["tf","pytorch","jax"]},{"repo":"https://github.com/huggingface/datasets","url":"https://huggingface.co/datasets/income/trec-news-top-20-gen-queries","frameworks":["tf","pytorch","jax"]},{"repo":"https://github.com/huggingface/datasets","url":"https://huggingface.co/datasets/income/robust04-top-20-gen-queries","frameworks":["tf","pytorch","jax"]},{"repo":"https://github.com/huggingface/datasets","url":"https://huggingface.co/datasets/income/scidocs-top-20-gen-queries","frameworks":["tf","pytorch","jax"]},{"repo":"https://github.com/huggingface/datasets","url":"https://huggingface.co/datasets/income/arguana-top-20-gen-queries","frameworks":["tf","pytorch","jax"]},{"repo":"https://github.com/huggingface/datasets","url":"https://huggingface.co/datasets/income/trec-covid-top-20-gen-queries","frameworks":["tf","pytorch","jax"]},{"repo":"https://github.com/huggingface/datasets","url":"https://huggingface.co/datasets/income/quora-top-20-gen-queries","frameworks":["tf","pytorch","jax"]},{"repo":"https://github.com/huggingface/datasets","url":"https://huggingface.co/datasets/income/webis-touche2020-top-20-gen-queries","frameworks":["tf","pytorch","jax"]},{"repo":"https://github.com/tensorflow/datasets","url":"https://www.tensorflow.org/datasets/catalog/beir","frameworks":["tf","pytorch","jax"]},{"repo":"https://github.com/UKPLab/beir","url":"https://github.com/UKPLab/beir","frameworks":["tf"]}],"num_papers_in_archive":311,"source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28"},"benchmarks":[{"leaderboard":"/sota/passage-retrieval-on-msmarco-beir","task":"Passage Retrieval","dataset_variant":"MSMARCO (BEIR)","rows":12,"metrics":["nDCG@10"],"first_row_in_archive_order":{"model":"BM25+CE","paper":"/paper/beir-a-heterogenous-benchmark-for-zero-shot","metrics":{"nDCG@10":"0.413"},"code_links":[{"title":"osu-nlp-group/hipporag","url":"https://github.com/osu-nlp-group/hipporag"},{"title":"UKPLab/beir","url":"https://github.com/UKPLab/beir"},{"title":"beir-cellar/beir","url":"https://github.com/beir-cellar/beir"}]},"note":"rows are the archive's own order at snapshot; nothing here re-ranks them"},{"leaderboard":"/sota/biomedical-information-retrieval-on-nfcorpus-1","task":"Biomedical Information Retrieval","dataset_variant":"NFCorpus (BEIR)","rows":7,"metrics":["nDCG@10"],"first_row_in_archive_order":{"model":"monoT5-3B","paper":"/paper/no-parameter-left-behind-how-distillation-and","metrics":{"nDCG@10":"0.383"},"code_links":[{"title":"guilhermemr04/scaling-zero-shot-retrieval","url":"https://github.com/guilhermemr04/scaling-zero-shot-retrieval"}]},"note":"rows are the archive's own order at snapshot; nothing here re-ranks them"},{"leaderboard":"/sota/biomedical-information-retrieval-on-bioasq-1","task":"Biomedical Information Retrieval","dataset_variant":"BioASQ (BEIR)","rows":6,"metrics":["nDCG@10"],"first_row_in_archive_order":{"model":"monoT5-3B","paper":"/paper/no-parameter-left-behind-how-distillation-and","metrics":{"nDCG@10":"0.579"},"code_links":[{"title":"guilhermemr04/scaling-zero-shot-retrieval","url":"https://github.com/guilhermemr04/scaling-zero-shot-retrieval"}]},"note":"rows are the archive's own order at snapshot; nothing here re-ranks them"},{"leaderboard":"/sota/biomedical-information-retrieval-on-trec-1","task":"Biomedical Information Retrieval","dataset_variant":"TREC-COVID (BEIR)","rows":6,"metrics":["nDCG@10"],"first_row_in_archive_order":{"model":"SGPT-BE-5.8B","paper":"/paper/sgpt-gpt-sentence-embeddings-for-semantic","metrics":{"nDCG@10":"0.873"},"code_links":[{"title":"muennighoff/sgpt","url":"https://github.com/muennighoff/sgpt"}]},"note":"rows are the archive's own order at snapshot; nothing here re-ranks them"},{"leaderboard":"/sota/zero-shot-on-beir-inference-free-model-on","task":"Zero Shot on BEIR (Inference Free Model)","dataset_variant":"BEIR","rows":6,"metrics":["NCDG@10"],"first_row_in_archive_order":{"model":"$\\ell_0$  Mask","paper":"/paper/exploring-ell-0-sparsification-for-inference","metrics":{"NCDG@10":"50.43"},"code_links":[{"title":"zhichao-aws/opensearch-sparse-model-tuning-sample","url":"https://github.com/zhichao-aws/opensearch-sparse-model-tuning-sample"}]},"note":"rows are the archive's own order at snapshot; nothing here re-ranks them"},{"leaderboard":"/sota/fact-checking-on-scifact-beir","task":"Fact Checking","dataset_variant":"SciFact (BEIR)","rows":5,"metrics":["nDCG@10"],"first_row_in_archive_order":{"model":"monoT5-3B","paper":"/paper/no-parameter-left-behind-how-distillation-and","metrics":{"nDCG@10":"0.777"},"code_links":[{"title":"guilhermemr04/scaling-zero-shot-retrieval","url":"https://github.com/guilhermemr04/scaling-zero-shot-retrieval"}]},"note":"rows are the archive's own order at snapshot; nothing here re-ranks them"},{"leaderboard":"/sota/fact-checking-on-climate-fever-beir","task":"Fact Checking","dataset_variant":"CLIMATE-FEVER (BEIR)","rows":4,"metrics":["nDCG@10"],"first_row_in_archive_order":{"model":"SGPT-BE-5.8B","paper":"/paper/sgpt-gpt-sentence-embeddings-for-semantic","metrics":{"nDCG@10":"0.305"},"code_links":[{"title":"muennighoff/sgpt","url":"https://github.com/muennighoff/sgpt"}]},"note":"rows are the archive's own order at snapshot; nothing here re-ranks them"},{"leaderboard":"/sota/fact-checking-on-fever-beir","task":"Fact Checking","dataset_variant":"FEVER (BEIR)","rows":4,"metrics":["nDCG@10"],"first_row_in_archive_order":{"model":"monoT5-3B","paper":"/paper/no-parameter-left-behind-how-distillation-and","metrics":{"nDCG@10":"0.849"},"code_links":[{"title":"guilhermemr04/scaling-zero-shot-retrieval","url":"https://github.com/guilhermemr04/scaling-zero-shot-retrieval"}]},"note":"rows are the archive's own order at snapshot; nothing here re-ranks them"},{"leaderboard":"/sota/question-answering-on-fiqa-2018-beir","task":"Question Answering","dataset_variant":"FiQA-2018 (BEIR)","rows":4,"metrics":["nDCG@10"],"first_row_in_archive_order":{"model":"monoT5-3B","paper":"/paper/no-parameter-left-behind-how-distillation-and","metrics":{"nDCG@10":"0.513"},"code_links":[{"title":"guilhermemr04/scaling-zero-shot-retrieval","url":"https://github.com/guilhermemr04/scaling-zero-shot-retrieval"}]},"note":"rows are the archive's own order at snapshot; nothing here re-ranks them"},{"leaderboard":"/sota/question-answering-on-hotpotqa-beir","task":"Question Answering","dataset_variant":"HotpotQA (BEIR)","rows":4,"metrics":["nDCG@10"],"first_row_in_archive_order":{"model":"monoT5-3B","paper":"/paper/no-parameter-left-behind-how-distillation-and","metrics":{"nDCG@10":"0.759"},"code_links":[{"title":"guilhermemr04/scaling-zero-shot-retrieval","url":"https://github.com/guilhermemr04/scaling-zero-shot-retrieval"}]},"note":"rows are the archive's own order at snapshot; nothing here re-ranks them"}],"papers":{"order":"date (newest first), then slug","population":"every paper with a leaderboard row on this dataset's benchmarks (the benchmark-backed subset): the archive's own papers-using-this-dataset list was never published, so this is not that list; num_papers_in_archive is the archive's own count","page":1,"pages_in_order":1,"rows_per_page":100,"shown":8,"of":8,"counts":{"papers_with_a_benchmark_row":8,"with_a_code_link":7,"where_syntology_ran_a_sample":4,"not_listed_spam_title":0,"listed":8,"listed_where_code_ran":4,"where_syntology_ran_a_sample_split":{"with_a_run_with_no_instrument_failure":3,"every_run_a_failure_of_syntologys_instrument":1,"listed_with_a_run_with_no_instrument_failure":3,"listed_every_run_a_failure_of_syntologys_instrument":1,"filter":{"states":["a run with no instrument failure","any run, instrument failures included"],"default":"a run with no instrument failure","note":"on the 'only where code ran' pages the default hides, in the browser, the rows where every run was a failure of Syntology's instrument; the second state shows them again. Rows are hidden, never re-ordered; these twins list every row"}},"definition":"distinct papers with at least one leaderboard row on this dataset's benchmarks; 'where Syntology ran a sample' counts papers with at least one harvested sample that ran, which is not a correctness claim"},"items":[{"paper":"/paper/exploring-ell-0-sparsification-for-inference","slug":"exploring-ell-0-sparsification-for-inference","title":"Exploring $\\ell_0$ Sparsification for Inference-free Sparse Retrievers","date":"2025-04-21","arxiv_id":"2504.14839","rows_on_this_dataset":2,"code_links":1,"syntology":{"read_at":"2026-10-05T00:36:11+00:00","samples_harvested":4,"samples_ran":3,"samples_constructed":0,"samples_ran_checked":3,"samples_ran_instrument_failed":0,"samples_unverified":1,"pointer_only_for_licence":0,"official":{"repos":["zhichao-aws/opensearch-sparse-model-tuning-sample"],"state":"official (archive's flag): 3 ran","n_ran":3,"n_constructed":0,"n_ran_no_instrument_failure":3,"n_unverified":1,"ran_from_kinds":["official"]},"claim":"Per-sample execution on synthesized fixtures; not a correctness claim.","sample_list":"/paper/exploring-ell-0-sparsification-for-inference#ran","syntology_url":"https://syntology.ai/paper/2504.14839","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2504.14839"}}}},{"paper":"/paper/towards-competitive-search-relevance-for","slug":"towards-competitive-search-relevance-for","title":"Towards Competitive Search Relevance For Inference-Free Learned Sparse Retrievers","date":"2024-11-07","arxiv_id":"2411.04403","rows_on_this_dataset":1,"code_links":1,"syntology":null},{"paper":"/paper/splade-v3-new-baselines-for-splade","slug":"splade-v3-new-baselines-for-splade","title":"SPLADE-v3: New baselines for SPLADE","date":"2024-03-11","arxiv_id":"2403.06789","rows_on_this_dataset":1,"code_links":1,"syntology":{"read_at":"2026-10-05T00:36:11+00:00","samples_harvested":3,"samples_ran":3,"samples_constructed":0,"samples_ran_checked":1,"samples_ran_instrument_failed":2,"samples_unverified":0,"pointer_only_for_licence":3,"official":null,"claim":"Per-sample execution on synthesized fixtures; not a correctness claim.","sample_list":"/paper/splade-v3-new-baselines-for-splade#ran","syntology_url":"https://syntology.ai/paper/2403.06789","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2403.06789"}}}},{"paper":"/paper/bm25-query-augmentation-learned-end-to-end","slug":"bm25-query-augmentation-learned-end-to-end","title":"BM25 Query Augmentation Learned End-to-End","date":"2023-05-23","arxiv_id":"2305.14087","rows_on_this_dataset":1,"code_links":0,"syntology":null},{"paper":"/paper/no-parameter-left-behind-how-distillation-and","slug":"no-parameter-left-behind-how-distillation-and","title":"No Parameter Left Behind: How Distillation and Model Size Affect Zero-Shot Retrieval","date":"2022-06-06","arxiv_id":"2206.02873","rows_on_this_dataset":8,"code_links":1,"syntology":null},{"paper":"/paper/sgpt-gpt-sentence-embeddings-for-semantic","slug":"sgpt-gpt-sentence-embeddings-for-semantic","title":"SGPT: GPT Sentence Embeddings for Semantic Search","date":"2022-02-17","arxiv_id":"2202.08904","rows_on_this_dataset":23,"code_links":1,"syntology":{"read_at":"2026-10-05T00:36:11+00:00","samples_harvested":1,"samples_ran":1,"samples_constructed":0,"samples_ran_checked":0,"samples_ran_instrument_failed":1,"samples_unverified":0,"pointer_only_for_licence":0,"official":{"repos":["muennighoff/sgpt"],"state":"official (archive's flag): 1 ran","n_ran":1,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":0,"ran_from_kinds":["official"]},"claim":"Per-sample execution on synthesized fixtures; not a correctness claim.","sample_list":"/paper/sgpt-gpt-sentence-embeddings-for-semantic#ran","syntology_url":"https://syntology.ai/paper/2202.08904","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2202.08904"}}}},{"paper":"/paper/splade-v2-sparse-lexical-and-expansion-model","slug":"splade-v2-sparse-lexical-and-expansion-model","title":"SPLADE v2: Sparse Lexical and Expansion Model for Information Retrieval","date":"2021-09-21","arxiv_id":"2109.10086","rows_on_this_dataset":1,"code_links":2,"syntology":null},{"paper":"/paper/beir-a-heterogenous-benchmark-for-zero-shot","slug":"beir-a-heterogenous-benchmark-for-zero-shot","title":"BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models","date":"2021-04-17","arxiv_id":"2104.08663","rows_on_this_dataset":21,"code_links":3,"syntology":{"read_at":"2026-10-05T00:36:11+00:00","samples_harvested":4,"samples_ran":3,"samples_constructed":0,"samples_ran_checked":3,"samples_ran_instrument_failed":0,"samples_unverified":1,"pointer_only_for_licence":0,"official":{"repos":["UKPLab/beir"],"state":"official (archive's flag): 3 ran","n_ran":3,"n_constructed":0,"n_ran_no_instrument_failure":3,"n_unverified":1,"ran_from_kinds":["official"]},"claim":"Per-sample execution on synthesized fixtures; not a correctness claim.","sample_list":"/paper/beir-a-heterogenous-benchmark-for-zero-shot#ran","syntology_url":"https://syntology.ai/paper/2104.08663","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2104.08663"}}}}],"full_list":"papers_with_a_benchmark_row: every listed paper, in this order"},"papers_with_a_benchmark_row":[{"paper":"/paper/exploring-ell-0-sparsification-for-inference","slug":"exploring-ell-0-sparsification-for-inference","title":"Exploring $\\ell_0$ Sparsification for Inference-free Sparse Retrievers","date":"2025-04-21","arxiv_id":"2504.14839","rows_on_this_dataset":2,"code_links":1,"syntology":{"read_at":"2026-10-05T00:36:11+00:00","samples_harvested":4,"samples_ran":3,"samples_constructed":0,"samples_ran_checked":3,"samples_ran_instrument_failed":0,"samples_unverified":1,"pointer_only_for_licence":0,"official":{"repos":["zhichao-aws/opensearch-sparse-model-tuning-sample"],"state":"official (archive's flag): 3 ran","n_ran":3,"n_constructed":0,"n_ran_no_instrument_failure":3,"n_unverified":1,"ran_from_kinds":["official"]},"claim":"Per-sample execution on synthesized fixtures; not a correctness claim.","sample_list":"/paper/exploring-ell-0-sparsification-for-inference#ran","syntology_url":"https://syntology.ai/paper/2504.14839","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2504.14839"}}}},{"paper":"/paper/towards-competitive-search-relevance-for","slug":"towards-competitive-search-relevance-for","title":"Towards Competitive Search Relevance For Inference-Free Learned Sparse Retrievers","date":"2024-11-07","arxiv_id":"2411.04403","rows_on_this_dataset":1,"code_links":1,"syntology":null},{"paper":"/paper/splade-v3-new-baselines-for-splade","slug":"splade-v3-new-baselines-for-splade","title":"SPLADE-v3: New baselines for SPLADE","date":"2024-03-11","arxiv_id":"2403.06789","rows_on_this_dataset":1,"code_links":1,"syntology":{"read_at":"2026-10-05T00:36:11+00:00","samples_harvested":3,"samples_ran":3,"samples_constructed":0,"samples_ran_checked":1,"samples_ran_instrument_failed":2,"samples_unverified":0,"pointer_only_for_licence":3,"official":null,"claim":"Per-sample execution on synthesized fixtures; not a correctness claim.","sample_list":"/paper/splade-v3-new-baselines-for-splade#ran","syntology_url":"https://syntology.ai/paper/2403.06789","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2403.06789"}}}},{"paper":"/paper/bm25-query-augmentation-learned-end-to-end","slug":"bm25-query-augmentation-learned-end-to-end","title":"BM25 Query Augmentation Learned End-to-End","date":"2023-05-23","arxiv_id":"2305.14087","rows_on_this_dataset":1,"code_links":0,"syntology":null},{"paper":"/paper/no-parameter-left-behind-how-distillation-and","slug":"no-parameter-left-behind-how-distillation-and","title":"No Parameter Left Behind: How Distillation and Model Size Affect Zero-Shot Retrieval","date":"2022-06-06","arxiv_id":"2206.02873","rows_on_this_dataset":8,"code_links":1,"syntology":null},{"paper":"/paper/sgpt-gpt-sentence-embeddings-for-semantic","slug":"sgpt-gpt-sentence-embeddings-for-semantic","title":"SGPT: GPT Sentence Embeddings for Semantic Search","date":"2022-02-17","arxiv_id":"2202.08904","rows_on_this_dataset":23,"code_links":1,"syntology":{"read_at":"2026-10-05T00:36:11+00:00","samples_harvested":1,"samples_ran":1,"samples_constructed":0,"samples_ran_checked":0,"samples_ran_instrument_failed":1,"samples_unverified":0,"pointer_only_for_licence":0,"official":{"repos":["muennighoff/sgpt"],"state":"official (archive's flag): 1 ran","n_ran":1,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":0,"ran_from_kinds":["official"]},"claim":"Per-sample execution on synthesized fixtures; not a correctness claim.","sample_list":"/paper/sgpt-gpt-sentence-embeddings-for-semantic#ran","syntology_url":"https://syntology.ai/paper/2202.08904","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2202.08904"}}}},{"paper":"/paper/splade-v2-sparse-lexical-and-expansion-model","slug":"splade-v2-sparse-lexical-and-expansion-model","title":"SPLADE v2: Sparse Lexical and Expansion Model for Information Retrieval","date":"2021-09-21","arxiv_id":"2109.10086","rows_on_this_dataset":1,"code_links":2,"syntology":null},{"paper":"/paper/beir-a-heterogenous-benchmark-for-zero-shot","slug":"beir-a-heterogenous-benchmark-for-zero-shot","title":"BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models","date":"2021-04-17","arxiv_id":"2104.08663","rows_on_this_dataset":21,"code_links":3,"syntology":{"read_at":"2026-10-05T00:36:11+00:00","samples_harvested":4,"samples_ran":3,"samples_constructed":0,"samples_ran_checked":3,"samples_ran_instrument_failed":0,"samples_unverified":1,"pointer_only_for_licence":0,"official":{"repos":["UKPLab/beir"],"state":"official (archive's flag): 3 ran","n_ran":3,"n_constructed":0,"n_ran_no_instrument_failure":3,"n_unverified":1,"ran_from_kinds":["official"]},"claim":"Per-sample execution on synthesized fixtures; not a correctness claim.","sample_list":"/paper/beir-a-heterogenous-benchmark-for-zero-shot#ran","syntology_url":"https://syntology.ai/paper/2104.08663","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2104.08663"}}}}],"papers_where_code_ran":4,"papers_where_code_ran_first_page":"/dataset/beir/papers/ran/1","papers_where_code_ran_caption":"We ran code from the paper's repository; we did not run it on this dataset or check it against this dataset's benchmarks.","pages":[{"order":"ran","page":1,"of":1,"url":"/dataset/beir/papers/ran/1","json":"/dataset/beir/papers/ran/1.json","rows":4}],"row_route":"each paper's syntology object carries sample_list (this site's paper page, #ran), syntology_url (syntology.ai's page for the paper, only when that route probed live) and mcp (the agent call that lists the paper's harvested samples; not a verdict on the paper)","developers":"https://syntology.ai/developers","syntology_totals":{"read_at":"2026-10-05T00:36:11+00:00","papers_with_samples":4,"samples_harvested":12,"samples_ran":10,"samples_constructed":0,"samples_ran_checked":7,"samples_ran_instrument_failed":3,"samples_unverified":2,"pointer_only_for_licence":3,"papers_with_no_sample_that_ran":0,"papers_with_a_sample_that_ran":4,"papers_with_a_run_with_no_instrument_failure":3,"papers_where_every_run_was_an_instrument_failure":1,"note":"the per-paper counts of every listed paper (all pages), summed; not a rate"},"key_notes":{"samples_ran_checked":"legacy name, kept unchanged so existing readers do not break: it counts the samples that ran with no instrument failure (honoured, violated, and ran with no contract checked); it does not mean a contract was checked, and the pages print it as 'K with no instrument failure', not 'K checked'","samples_constructed":"a sub-count of the samples that ran, never subtracted from them and never a failure: an executed sample whose run returned an instance of its own class (fixture_out_type equals the entry name): the run built an object and did not compute a result (Syntology's RAN record, counts.constructed)"},"since_archive_judged":{"papers":10,"papers_placed":1,"judged_by":{"model":"Claude Sonnet 4.5","models_as_recorded":["global.anthropic.claude-sonnet-4-5-20250929-v1:0"],"from":"lb_rows_run_verified.jsonl: each record's `model` field (the extractor's call that wrote decisions[])"},"note":"a model's judgement from the paper's own tables that they report on this leaderboard, not a result: no value from a paper that was not placed is printed or carried here; papers counts each paper once over these leaderboards","backfill":"8,886 of 18,637 newer papers checked so far (733 more had nothing to place; 41 could not be read)","backfill_complete":false,"leaderboards":[{"leaderboard":"/sota/biomedical-information-retrieval-on-nfcorpus-1","since_archive":"/sota/biomedical-information-retrieval-on-nfcorpus-1#since-judged","papers_judged":8,"placed":0},{"leaderboard":"/sota/biomedical-information-retrieval-on-trec-1","since_archive":"/sota/biomedical-information-retrieval-on-trec-1#since-judged","papers_judged":8,"placed":1},{"leaderboard":"/sota/fact-checking-on-scifact-beir","since_archive":"/sota/fact-checking-on-scifact-beir#since-judged","papers_judged":6,"placed":1},{"leaderboard":"/sota/question-answering-on-fiqa-2018-beir","since_archive":"/sota/question-answering-on-fiqa-2018-beir#since-judged","papers_judged":1,"placed":0}]},"papers_note":"The archive never published its papers-using-dataset list; these are papers with a leaderboard row on this dataset's benchmarks.","record_sha256":"1d78fa62707422d9d65c8a8a6701b5749283f51ab018f3abbcbccda7120ae15d","record_changed_at":"2026-10-05","record_changed_at_basis":"changed"}