{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/decaying-momentum-helps-neural-network","title":"Demon: Improved Neural Network Training with Momentum Decay","arxiv_id":"1910.04952","date":"2019-10-11","proceeding":null,"authors":["John Chen","Cameron Wolfe","Zhao Li","Anastasios Kyrillidis"],"abstract":"Momentum is a widely used technique for gradient-based optimizers in deep learning. In this paper, we propose a decaying momentum (\\textsc{Demon}) rule. We conduct the first large-scale empirical analysis of momentum decay methods for modern neural network optimization, in addition to the most popular learning rate decay schedules. Across 28 relevant combinations of models, epochs, datasets, and optimizers, \\textsc{Demon} achieves the highest number of Top-1 and Top-3 finishes at 39\\% and 85\\% respectively, almost doubling the second-placed learning rate cosine schedule at 17\\% and 60\\%, respectively. \\textsc{Demon} also outperforms other widely used schedulers including, but not limited to, the learning rate step schedule, linear schedule, OneCycle schedule, and exponential schedule. Compared with the widely used learning rate step schedule, \\textsc{Demon} is observed to be less sensitive to parameter tuning, which is critical to training neural networks in practice. Results are demonstrated across a variety of settings and architectures, including image classification, generative models, and language models. \\textsc{Demon} is easy to implement, requires no additional tuning, and incurs almost no extra computational overhead compared to the vanilla counterparts. Code is readily available.","url_abs":"https://arxiv.org/abs/1910.04952v4","url_pdf":"https://arxiv.org/pdf/1910.04952v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"decaying-momentum-helps-neural-network","repo_url":"https://github.com/JRC1995/DemonRangerOptimizer","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"decaying-momentum-helps-neural-network","repo_url":"https://github.com/autasi/demon_sgd","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":null}],"tasks":[{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"image-classification","task_name":"image-classification"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"demon","method_name":"Demon"},{"method_slug":"demon-adam","method_name":"Demon ADAM"},{"method_slug":"demon-cm","method_name":"Demon CM"},{"method_slug":"sgd","method_name":"SGD"},{"method_slug":"sgd-with-momentum","method_name":"SGD with Momentum"}],"datasets_introduced":[],"methods_introduced":[{"slug":"demon","name":"Demon","full_name":"Demon"},{"slug":"demon-adam","name":"Demon ADAM","full_name":"Demon ADAM"},{"slug":"demon-cm","name":"Demon CM","full_name":"Demon CM"}],"results":[],"syntology":{"syntology_url":null,"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers","held":null,"reason":"arxiv_id_not_in_atlas","checked_against":{"atlas_ids":549449,"atlas_ids_sha256":"afd72399dafd9906e9062927f70655952385a3d75488558e1f0484e950ab09b7","read_at":"2026-10-05T01:46:21Z"},"by_repo":[{"repo":"JRC1995/DemonRangerOptimizer","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0},{"repo":"autasi/demon_sgd","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0}],"by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":null,"arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"cc4b6b4a83683e6057c4eea17b905651147cae13989c0c91bfb8611d63379fca","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}