{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/improved-training-of-neural-trans-dimensional","title":"Improved training of neural trans-dimensional random field language models with dynamic noise-contrastive estimation","arxiv_id":"1807.00993","date":"2018-07-03","proceeding":null,"authors":["Bin Wang","Zhijian Ou"],"abstract":"A new whole-sentence language model - neural trans-dimensional random field\nlanguage model (neural TRF LM), where sentences are modeled as a collection of\nrandom fields, and the potential function is defined by a neural network, has\nbeen introduced and successfully trained by noise-contrastive estimation (NCE).\nIn this paper, we extend NCE and propose dynamic noise-contrastive estimation\n(DNCE) to solve the two problems observed in NCE training. First, a dynamic\nnoise distribution is introduced and trained simultaneously to converge to the\ndata distribution. This helps to significantly cut down the noise sample number\nused in NCE and reduce the training cost. Second, DNCE discriminates between\nsentences generated from the noise distribution and sentences generated from\nthe interpolation of the data distribution and the noise distribution. This\nalleviates the overfitting problem caused by the sparseness of the training\nset. With DNCE, we can successfully and efficiently train neural TRF LMs on\nlarge corpus (about 0.8 billion words) with large vocabulary (about 568 K\nwords). Neural TRF LMs perform as good as LSTM LMs with less parameters and\nbeing 5x~114x faster in rescoring sentences. Interpolating neural TRF LMs with\nLSTM LMs and n-gram LMs can further reduce the error rates.","url_abs":"http://arxiv.org/abs/1807.00993v1","url_pdf":"http://arxiv.org/pdf/1807.00993v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"improved-training-of-neural-trans-dimensional","repo_url":"https://github.com/wbengine/TRF-NN-Tensorflow","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"tf","reach":{"status":"ok"}}],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"sentence","task_name":"Sentence"}],"methods":[{"method_slug":"lstm","method_name":"LSTM"},{"method_slug":"sigmoid-activation","method_name":"Sigmoid Activation"},{"method_slug":"tanh-activation","method_name":"Tanh Activation"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/1807.00993","atlas_url":"https://app.syntology.ai/?focus=1807.00993","mcp":null,"developers":"https://syntology.ai/developers","held":true,"reason":"arxiv_id_in_atlas","checked_against":{"atlas_ids":549449,"atlas_ids_sha256":"afd72399dafd9906e9062927f70655952385a3d75488558e1f0484e950ab09b7","read_at":"2026-10-05T01:46:21Z"},"agent_calls":[{"tool":"get_citation_path","arguments":{"paper_1":"1807.00993"},"arguments_in":{"paper_2":"another paper's arXiv id or title"},"call":"get_citation_path(paper_1=\"1807.00993\", paper_2=\"…\")"},{"tool":"get_concepts_for_paper","arguments":{"arxiv_id":"1807.00993"},"call":"get_concepts_for_paper(arxiv_id=\"1807.00993\")"}],"by_repo":[{"repo":"wbengine/TRF-NN-Tensorflow","kind":"official","samples":0,"ran":0,"instrument":0,"unverified":0}],"by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":null,"arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"bf6661cd249e71e9c656970472e85385e020b7e1c3fb6a2597aa175e6a0e41cb","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}