{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/on-the-effect-of-low-frequency-terms-on","title":"On the Effect of Low-Frequency Terms on Neural-IR Models","arxiv_id":"1904.12683","date":"2019-04-29","proceeding":null,"authors":["Sebastian Hofstätter","Navid Rekabsaz","Carsten Eickhoff","Allan Hanbury"],"abstract":"Low-frequency terms are a recurring challenge for information retrieval\nmodels, especially neural IR frameworks struggle with adequately capturing\ninfrequently observed words. While these terms are often removed from neural\nmodels - mainly as a concession to efficiency demands - they traditionally play\nan important role in the performance of IR models. In this paper, we analyze\nthe effects of low-frequency terms on the performance and robustness of neural\nIR models. We conduct controlled experiments on three recent neural IR models,\ntrained on a large-scale passage retrieval collection. We evaluate the neural\nIR models with various vocabulary sizes for their respective word embeddings,\nconsidering different levels of constraints on the available GPU memory. We\nobserve that despite the significant benefits of using larger vocabularies, the\nperformance gap between the vocabularies can be, to a great extent, mitigated\nby extensive tuning of a related parameter: the number of documents to re-rank.\nWe further investigate the use of subword-token embedding models, and in\nparticular FastText, for neural IR models. Our experiments show that using\nFastText brings slight improvements to the overall performance of the neural IR\nmodels in comparison to models trained on the full vocabulary, while the\nimprovement becomes much more pronounced for queries containing low-frequency\nterms.","url_abs":"http://arxiv.org/abs/1904.12683v2","url_pdf":"http://arxiv.org/pdf/1904.12683v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"on-the-effect-of-low-frequency-terms-on","repo_url":"https://github.com/sebastian-hofstaetter/sigir19-neural-ir","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":null,"task_name":"GPU"},{"task_slug":"information-retrieval","task_name":"Information Retrieval"},{"task_slug":"passage-retrieval","task_name":"Passage Retrieval"},{"task_slug":"retrieval","task_name":"Retrieval"},{"task_slug":"word-embeddings","task_name":"Word Embeddings"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/1904.12683","atlas_url":"https://app.syntology.ai/?focus=1904.12683","mcp":null,"developers":"https://syntology.ai/developers","held":true,"reason":"arxiv_id_in_atlas","checked_against":{"atlas_ids":549449,"atlas_ids_sha256":"afd72399dafd9906e9062927f70655952385a3d75488558e1f0484e950ab09b7","read_at":"2026-10-05T01:46:21Z"},"agent_calls":[{"tool":"get_citation_path","arguments":{"paper_1":"1904.12683"},"arguments_in":{"paper_2":"another paper's arXiv id or title"},"call":"get_citation_path(paper_1=\"1904.12683\", paper_2=\"…\")"},{"tool":"get_concepts_for_paper","arguments":{"arxiv_id":"1904.12683"},"call":"get_concepts_for_paper(arxiv_id=\"1904.12683\")"}],"by_repo":[{"repo":"sebastian-hofstaetter/sigir19-neural-ir","kind":"official","samples":0,"ran":0,"instrument":0,"unverified":0}],"by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":null,"arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"2bbac944fc72d387047f035532eca2740cd6a8bfb9677b69a18970fff9fc3bc0","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}