{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/improving-neural-language-models-with-a","title":"Improving Neural Language Models with a Continuous Cache","arxiv_id":"1612.04426","date":"2016-12-13","proceeding":null,"authors":["Edouard Grave","Armand Joulin","Nicolas Usunier"],"abstract":"We propose an extension to neural network language models to adapt their\nprediction to the recent history. Our model is a simplified version of memory\naugmented networks, which stores past hidden activations as memory and accesses\nthem through a dot product with the current hidden activation. This mechanism\nis very efficient and scales to very large memory sizes. We also draw a link\nbetween the use of external memory in neural network and cache models used with\ncount based language models. We demonstrate on several language model datasets\nthat our approach performs significantly better than recent memory augmented\nnetworks.","url_abs":"http://arxiv.org/abs/1612.04426v1","url_pdf":"http://arxiv.org/pdf/1612.04426v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"improving-neural-language-models-with-a","repo_url":"https://github.com/Asteur/RERITES-AvgWeightDescentLSTM-PoetryGeneration","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"improving-neural-language-models-with-a","repo_url":"https://github.com/AtheMathmo/lookahead-lstm","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"BSD-3-Clause"}},{"paper_slug":"improving-neural-language-models-with-a","repo_url":"https://github.com/SachinIchake/KALM","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"BSD-3-Clause"}},{"paper_slug":"improving-neural-language-models-with-a","repo_url":"https://github.com/ari-holtzman/genlm","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"BSD-3-Clause"}},{"paper_slug":"improving-neural-language-models-with-a","repo_url":"https://github.com/arvieFrydenlund/awd-lstm-lm","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"BSD-3-Clause"}},{"paper_slug":"improving-neural-language-models-with-a","repo_url":"https://github.com/jb33k/awd-lstm-lm-ThinkNet","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"BSD-3-Clause"}},{"paper_slug":"improving-neural-language-models-with-a","repo_url":"https://github.com/jhave/RERITES-AvgWeightDescentLSTM-PoetryGeneration","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"unanswered"}},{"paper_slug":"improving-neural-language-models-with-a","repo_url":"https://github.com/llppff/ptb-lstmorqrnn-pytorch","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"BSD-3-Clause"}},{"paper_slug":"improving-neural-language-models-with-a","repo_url":"https://github.com/philippwirth/awd-lstm-test","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"BSD-3-Clause"}},{"paper_slug":"improving-neural-language-models-with-a","repo_url":"https://github.com/philippwirth/treelangrnn","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"BSD-3-Clause"}},{"paper_slug":"improving-neural-language-models-with-a","repo_url":"https://github.com/salesforce/awd-lstm-lm","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"BSD-3-Clause"}},{"paper_slug":"improving-neural-language-models-with-a","repo_url":"https://github.com/soyoung97/awd-lstm-gru","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"BSD-3-Clause"}},{"paper_slug":"improving-neural-language-models-with-a","repo_url":"https://github.com/uclanlp/NamedEntityLanguageModel","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"BSD-3-Clause"}},{"paper_slug":"improving-neural-language-models-with-a","repo_url":"https://github.com/dmlc/gluon-nlp","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mxnet","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"}],"methods":[{"method_slug":"adagrad","method_name":"AdaGrad"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"gradient-clipping","method_name":"Gradient Clipping"},{"method_slug":"lstm","method_name":"LSTM"},{"method_slug":"neural-cache","method_name":"Neural Cache"},{"method_slug":"sigmoid-activation","method_name":"Sigmoid Activation"},{"method_slug":"tanh-activation","method_name":"Tanh Activation"}],"datasets_introduced":[],"methods_introduced":[{"slug":"neural-cache","name":"Neural Cache","full_name":"Neural Cache"}],"results":[{"leaderboard":"/sota/language-modelling-on-wikitext-103","task":"Language Modelling","dataset":"WikiText-103","model":"Neural cache model (size = 2,000)","rank_in_archive_order":80,"of":89,"metrics":{"Test perplexity":"40.8"},"uses_additional_data":false},{"leaderboard":"/sota/language-modelling-on-wikitext-103","task":"Language Modelling","dataset":"WikiText-103","model":"Neural cache model (size = 100)","rank_in_archive_order":81,"of":89,"metrics":{"Test perplexity":"44.8"},"uses_additional_data":false},{"leaderboard":"/sota/language-modelling-on-wikitext-103","task":"Language Modelling","dataset":"WikiText-103","model":"LSTM","rank_in_archive_order":85,"of":89,"metrics":{"Test perplexity":"48.7"},"uses_additional_data":false},{"leaderboard":"/sota/language-modelling-on-wikitext-2","task":"Language Modelling","dataset":"WikiText-2","model":"Grave et al. (2016) - LSTM + continuous cache pointer","rank_in_archive_order":33,"of":38,"metrics":{"Test perplexity":"68.9"},"uses_additional_data":false},{"leaderboard":"/sota/language-modelling-on-wikitext-2","task":"Language Modelling","dataset":"WikiText-2","model":"Grave et al. (2016) - LSTM","rank_in_archive_order":37,"of":38,"metrics":{"Test perplexity":"99.3"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/1612.04426","atlas_url":"https://app.syntology.ai/?focus=1612.04426","mcp":null,"developers":"https://syntology.ai/developers","held":true,"reason":"arxiv_id_in_atlas","checked_against":{"atlas_ids":549449,"atlas_ids_sha256":"afd72399dafd9906e9062927f70655952385a3d75488558e1f0484e950ab09b7","read_at":"2026-10-05T01:46:21Z"},"agent_calls":[{"tool":"get_citation_path","arguments":{"paper_1":"1612.04426"},"arguments_in":{"paper_2":"another paper's arXiv id or title"},"call":"get_citation_path(paper_1=\"1612.04426\", paper_2=\"…\")"},{"tool":"get_concepts_for_paper","arguments":{"arxiv_id":"1612.04426"},"call":"get_concepts_for_paper(arxiv_id=\"1612.04426\")"}],"by_repo":[{"repo":"Asteur/RERITES-AvgWeightDescentLSTM-PoetryGeneration","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0,"harvested_for_other_papers":1,"harvested_for":["1708.02182"],"harvested_for_calls":[{"arxiv_id":"1708.02182","tool":"get_harvested_code_for_paper","arguments":{"arxiv_id":"1708.02182"},"call":"get_harvested_code_for_paper(arxiv_id=\"1708.02182\")"}]},{"repo":"AtheMathmo/lookahead-lstm","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0,"harvested_for_other_papers":1,"harvested_for":["1803.08240"],"harvested_for_calls":[{"arxiv_id":"1803.08240","tool":"get_harvested_code_for_paper","arguments":{"arxiv_id":"1803.08240"},"call":"get_harvested_code_for_paper(arxiv_id=\"1803.08240\")"}]},{"repo":"SachinIchake/KALM","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0,"harvested_for_other_papers":1,"harvested_for":["1803.08240"],"harvested_for_calls":[{"arxiv_id":"1803.08240","tool":"get_harvested_code_for_paper","arguments":{"arxiv_id":"1803.08240"},"call":"get_harvested_code_for_paper(arxiv_id=\"1803.08240\")"}]},{"repo":"ari-holtzman/genlm","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0,"harvested_for_other_papers":1,"harvested_for":["1803.08240"],"harvested_for_calls":[{"arxiv_id":"1803.08240","tool":"get_harvested_code_for_paper","arguments":{"arxiv_id":"1803.08240"},"call":"get_harvested_code_for_paper(arxiv_id=\"1803.08240\")"}]},{"repo":"arvieFrydenlund/awd-lstm-lm","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0},{"repo":"jb33k/awd-lstm-lm-ThinkNet","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0},{"repo":"jhave/RERITES-AvgWeightDescentLSTM-PoetryGeneration","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0},{"repo":"llppff/ptb-lstmorqrnn-pytorch","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0},{"repo":"philippwirth/awd-lstm-test","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0},{"repo":"philippwirth/treelangrnn","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0,"harvested_for_other_papers":1,"harvested_for":["1803.08240"],"harvested_for_calls":[{"arxiv_id":"1803.08240","tool":"get_harvested_code_for_paper","arguments":{"arxiv_id":"1803.08240"},"call":"get_harvested_code_for_paper(arxiv_id=\"1803.08240\")"}]},{"repo":"salesforce/awd-lstm-lm","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0,"harvested_for_other_papers":1,"harvested_for":["2010.02986"],"harvested_for_calls":[{"arxiv_id":"2010.02986","tool":"get_harvested_code_for_paper","arguments":{"arxiv_id":"2010.02986"},"call":"get_harvested_code_for_paper(arxiv_id=\"2010.02986\")"}]},{"repo":"soyoung97/awd-lstm-gru","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0},{"repo":"uclanlp/NamedEntityLanguageModel","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0,"harvested_for_other_papers":1,"harvested_for":["1805.04836"],"harvested_for_calls":[{"arxiv_id":"1805.04836","tool":"get_harvested_code_for_paper","arguments":{"arxiv_id":"1805.04836"},"call":"get_harvested_code_for_paper(arxiv_id=\"1805.04836\")"}]},{"repo":"dmlc/gluon-nlp","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0,"harvested_for_other_papers":1,"harvested_for":["1907.04433"],"harvested_for_calls":[{"arxiv_id":"1907.04433","tool":"get_harvested_code_for_paper","arguments":{"arxiv_id":"1907.04433"},"call":"get_harvested_code_for_paper(arxiv_id=\"1907.04433\")"}]}],"by_repo_harvested_for_is":"harvested_for: the arXiv ids (as Syntology's graph holds them) of every paper whose samples Syntology's graph links to this repository, in arXiv-id order, complete; the page names the first 3 and says how many more. A link only: it does not say the papers are the same work, that one copied the other, or that the code implements this paper's method; harvested_for_calls gives the get_harvested_code_for_paper call for each arXiv-shaped id among them","by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":null,"arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"d7c3112a80cf15655c83faef4eb37cf691e3ca69ee5d23aa7a75663a10e235c4","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}