{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/a-batch-noise-contrastive-estimation-approach","title":"A Batch Noise Contrastive Estimation Approach for Training Large Vocabulary Language Models","arxiv_id":"1708.05997","date":"2017-08-20","proceeding":null,"authors":["Youssef Oualil","Dietrich Klakow"],"abstract":"Training large vocabulary Neural Network Language Models (NNLMs) is a\ndifficult task due to the explicit requirement of the output layer\nnormalization, which typically involves the evaluation of the full softmax\nfunction over the complete vocabulary. This paper proposes a Batch Noise\nContrastive Estimation (B-NCE) approach to alleviate this problem. This is\nachieved by reducing the vocabulary, at each time step, to the target words in\nthe batch and then replacing the softmax by the noise contrastive estimation\napproach, where these words play the role of targets and noise samples at the\nsame time. In doing so, the proposed approach can be fully formulated and\nimplemented using optimal dense matrix operations. Applying B-NCE to train\ndifferent NNLMs on the Large Text Compression Benchmark (LTCB) and the One\nBillion Word Benchmark (OBWB) shows a significant reduction of the training\ntime with no noticeable degradation of the models performance. This paper also\npresents a new baseline comparative study of different standard NNLMs on the\nlarge OBWB on a single Titan-X GPU.","url_abs":"http://arxiv.org/abs/1708.05997v2","url_pdf":"http://arxiv.org/pdf/1708.05997v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"a-batch-noise-contrastive-estimation-approach","repo_url":"https://github.com/Stonesjtu/Pytorch-NCE","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":null,"task_name":"GPU"},{"task_slug":"text-compression","task_name":"Text Compression"}],"methods":[{"method_slug":"softmax","method_name":"Softmax"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers","held":null,"reason":"arxiv_id_not_in_atlas","checked_against":{"atlas_ids":549449,"atlas_ids_sha256":"afd72399dafd9906e9062927f70655952385a3d75488558e1f0484e950ab09b7","read_at":"2026-10-05T01:46:21Z"},"by_repo":[{"repo":"Stonesjtu/Pytorch-NCE","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0}],"by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":null,"arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"92556f81efbf55001df03b1df83f9ed014f51f449d829fda48fb14d1b6dd24da","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}