{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/method/demon","slug":"demon","name":"Demon","full_name":"Demon","full_name_withheld":false,"description_markdown":"**Decaying Momentum**, or **Demon**, is a stochastic optimizer motivated by decaying the total contribution of a gradient to all future updates. By decaying the momentum parameter, the total contribution of a gradient to all future updates is decayed. A particular gradient term $g\\_{t}$ contributes a total of  $\\eta\\sum\\_{i}\\beta^{i}$ of its \"energy\" to all future gradient updates, and this results in the geometric sum, $\\sum^{\\infty}\\_{i=1}\\beta^{i} = \\beta\\sum^{\\infty}\\_{i=0}\\beta^{i} = \\frac{\\beta}{\\left(1-\\beta\\right)}$. Decaying this sum results in the Demon algorithm. Letting $\\beta\\_{init}$ be the initial $\\beta$; then at the current step $t$ with total $T$ steps, the decay routine is given by solving the below for $\\beta\\_{t}$:\r\n\r\n$$ \\frac{\\beta\\_{t}}{\\left(1-\\beta\\_{t}\\right)} =  \\left(1-t/T\\right)\\beta\\_{init}/\\left(1-\\beta\\_{init}\\right)$$\r\n\r\nWhere $\\left(1-t/T\\right)$ refers to the proportion of iterations remaining. Note that Demon typically requires no hyperparameter tuning as it is usually decayed to $0$ or a small negative value at time \r\n$T$. Improved performance is observed by delaying the decaying. Demon can be applied to any gradient descent algorithm with a momentum parameter.","description_state":"present","introduced_year":null,"introduced_by":{"title":"Demon: Improved Neural Network Training with Momentum Decay","paper":"/paper/decaying-momentum-helps-neural-network","first_author":"John Chen","n_authors":4,"url_abs":null,"archive_paper_url":"https://paperswithcode.com/paper/decaying-momentum-helps-neural-network"},"source":{"url":"https://arxiv.org/abs/1910.04952v4","title":"Demon: Improved Neural Network Training with Momentum Decay","url_on_a_paper_host":true},"code_snippet_url":"https://github.com/JRC1995/DemonRangerOptimizer/blob/5a3e6e352ab766f96cd8d20eabd5b71843c595fe/optimizers.py#L205","code_snippet_url_on_a_code_host":true,"categories":[{"area":"General","area_id":"general","collection":"Momentum Rules","url":"/methods/category/momentum-rules","pwc_aliases":[]}],"n_papers_tagged":16,"archive_num_papers":16,"papers_newest_first":[{"paper":null,"slug":"representation-and-interpretation-in","title":"Representation and Interpretation in Artificial and Natural Computing","date":"2025-02-14","arxiv_id":"2502.10383","n_code_links":0,"syntology":null},{"paper":null,"slug":"demonet-underwater-acoustic-target","title":"DEMONet: Underwater Acoustic Target Recognition based on Multi-Expert Network and Cross-Temporal Variational Autoencoder","date":"2024-11-05","arxiv_id":"2411.02758","n_code_links":0,"syntology":null},{"paper":"/paper/training-free-diffusion-model-alignment-with","slug":"training-free-diffusion-model-alignment-with","title":"Training-free Diffusion Model Alignment with Sampling Demons","date":"2024-10-08","arxiv_id":"2410.05760","n_code_links":1,"syntology":{"ran":2,"of":3,"n_ran_checked":0,"n_instrument":2,"unverified":1,"pointer_only":0,"phrase":"2 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 1 unverified","official":{"repos":["aiiu-lab/DemonSampling"],"state":"official (archive's flag): 2 ran","n_ran":2,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":1,"ran_from_kinds":["official"]}}},{"paper":null,"slug":"neural-entropy","title":"Neural Entropy","date":"2024-09-05","arxiv_id":"2409.03817","n_code_links":0,"syntology":null},{"paper":null,"slug":"str-l-pose-integrating-point-and-structured","title":"Str-L Pose: Integrating Point and Structured Line for Relative Pose Estimation in Dual-Graph","date":"2024-08-28","arxiv_id":"2408.15750","n_code_links":0,"syntology":null},{"paper":"/paper/a-decentralized-and-self-adaptive-approach","slug":"a-decentralized-and-self-adaptive-approach","title":"A Decentralized and Self-Adaptive Approach for Monitoring Volatile Edge Environments","date":"2024-05-13","arxiv_id":"2405.07806","n_code_links":1,"syntology":null},{"paper":null,"slug":"reflective-linguistic-programming-rlp-a","title":"Reflective Linguistic Programming (RLP): A Stepping Stone in Socially-Aware AGI (SocialAGI)","date":"2023-05-22","arxiv_id":"2305.12647","n_code_links":0,"syntology":null},{"paper":null,"slug":"how-to-train-your-demon-to-do-fast","title":"How to train your demon to do fast information erasure without heat production","date":"2023-05-17","arxiv_id":"2305.10607","n_code_links":0,"syntology":null},{"paper":null,"slug":"thermodynamic-ai-and-the-fluctuation-frontier","title":"Thermodynamic AI and the fluctuation frontier","date":"2023-02-09","arxiv_id":"2302.06584","n_code_links":0,"syntology":null},{"paper":"/paper/differentiable-neural-computers-with-memory","slug":"differentiable-neural-computers-with-memory","title":"Differentiable Neural Computers with Memory Demon","date":"2022-11-05","arxiv_id":"2211.02987","n_code_links":1,"syntology":{"ran":3,"of":5,"n_ran_checked":0,"n_instrument":3,"unverified":2,"pointer_only":5,"phrase":"3 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 2 unverified","official":{"repos":["azarafrooz/dnc-with-demon"],"state":"official (archive's flag): 3 ran","n_ran":3,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":2,"ran_from_kinds":["official"]}}},{"paper":null,"slug":"static-knowledge-vs-dynamic-argumentation-a","title":"Static Knowledge vs. Dynamic Argumentation: A Dual Theory Based on Kripke Semantics","date":"2022-09-27","arxiv_id":"2209.13082","n_code_links":0,"syntology":null},{"paper":null,"slug":"boosting-adversarial-transferability-of-mlp","title":"Boosting Adversarial Transferability of MLP-Mixer","date":"2022-04-26","arxiv_id":"2204.12204","n_code_links":0,"syntology":null},{"paper":"/paper/learning-relational-rules-from-rewards","slug":"learning-relational-rules-from-rewards","title":"Learning Relational Rules from Rewards","date":"2022-03-25","arxiv_id":"2203.13599","n_code_links":1,"syntology":null},{"paper":null,"slug":"nonequilibrium-thermodynamics-of-self","title":"Nonequilibrium thermodynamics of self-supervised learning","date":"2021-06-16","arxiv_id":"2106.08981","n_code_links":0,"syntology":null},{"paper":null,"slug":"fusing-the-old-with-the-new-learning-relative","title":"Fusing the Old with the New: Learning Relative Camera Pose with Geometry-Guided Uncertainty","date":"2021-04-16","arxiv_id":"2104.08278","n_code_links":0,"syntology":null},{"paper":"/paper/decaying-momentum-helps-neural-network","slug":"decaying-momentum-helps-neural-network","title":"Demon: Improved Neural Network Training with Momentum Decay","date":"2019-10-11","arxiv_id":"1910.04952","n_code_links":2,"syntology":null}],"papers_shown":16,"papers_order":"date (newest first), then slug","paper_counts":{"archive_papers_tagged":16,"with_a_code_link":5,"where_syntology_ran_a_sample":2,"not_listed_spam_title":0,"listed":16,"listed_where_code_ran":2,"where_syntology_ran_a_sample_split":{"with_a_run_with_no_instrument_failure":0,"every_run_a_failure_of_syntologys_instrument":2,"listed_with_a_run_with_no_instrument_failure":0,"listed_every_run_a_failure_of_syntologys_instrument":2,"filter":{"states":["a run with no instrument failure","any run, instrument failures included"],"default":"a run with no instrument failure","note":"on the 'only where code ran' pages the default hides, in the browser, the rows where every run was a failure of Syntology's instrument; the second state shows them again. Rows are hidden, never re-ordered; these twins list every row"}},"definition":"distinct papers the archive tags; 'where Syntology ran a sample' counts papers with at least one harvested sample that ran, which is not a correctness claim"},"papers_where_code_ran":2,"papers_where_code_ran_first_page":"/method/demon/papers/ran/1","papers_where_code_ran_caption":"We ran code from the paper's repository; we did not isolate this method inside it.","pages":[{"order":"ran","page":1,"of":1,"url":"/method/demon/papers/ran/1","json":"/method/demon/papers/ran/1.json","rows":2}],"tasks":[{"task":"/task/graph-neural-network","name":"Graph Neural Network","papers":2},{"task":"/task/pose-estimation","name":"Pose Estimation","papers":2},{"task":"/task/adversarial-attack","name":"Adversarial Attack","papers":1},{"task":"/task/atari-games","name":"Atari Games","papers":1},{"task":"/task/autonomous-driving","name":"Autonomous Driving","papers":1},{"task":"/task/camera-pose-estimation","name":"Camera Pose Estimation","papers":1},{"task":"/task/denoising","name":"Denoising","papers":1},{"task":"/task/edge-computing","name":"Edge-computing","papers":1},{"task":"/task/image-classification","name":"Image Classification","papers":1},{"task":"/task/image-generation","name":"Image Generation","papers":1},{"task":"/task/language-modeling","name":"Language Modeling","papers":1},{"task":"/task/language-modelling","name":"Language Modelling","papers":1},{"task":"/task/scene-understanding","name":"Scene Understanding","papers":1},{"task":"/task/self-supervised-learning","name":"Self-Supervised Learning","papers":1},{"task":"/task/stochastic-optimization","name":"Stochastic Optimization","papers":1},{"task":"/task/text-to-image-generation-1","name":"Text to Image Generation","papers":1},{"task":"/task/text-to-image-generation","name":"Text-to-Image Generation","papers":1},{"task":"/task/image-classification","name":"image-classification","papers":1}],"tasks_shown":18,"n_tasks":18,"usage_by_year":[{"year":"2019","papers":1},{"year":"2021","papers":2},{"year":"2022","papers":4},{"year":"2023","papers":3},{"year":"2024","papers":5},{"year":"2025","papers":1}],"row_source":"methods_table","archive":{"source":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","archive_url":"https://paperswithcode.com/method/demon"},"syntology_read_at":"2026-10-05T00:36:11+00:00","syntology":{"source":"Syntology's have check for this name (the free coverage lookup), read from Syntology's graph","have_url":"https://syntology.ai/api/graph/have?x=Demon","read_at":"2026-10-05T00:54:43+00:00","query":"Demon","kind":"paper","have":"unverified_code","tier":"harvested","verified_implementation":false,"level":1,"level_meaning":"the served verification level, per answer; never summed across methods","resolved_to":{"arxiv_id":"2308.04152"},"resolution":"paper","resolved_name_differs":false,"absence_asserted":false,"absence_unchecked":null,"no_code_is_about_this_method":false,"subject":"different_paper","archive_paper_arxiv_id":"1910.04952","repositories":{"listed":2,"in_graph":2,"list_truncated":false,"strongest_link":"official"},"retrieve_with":"get_harvested_code_for_paper(arxiv_id='2308.04152')","name_competition":{"verdict":"outranked","may_be_wrong_sense":true,"bearers_found":3,"served":{"arxiv_id":"2308.04152","citations":98},"leading_other_bearer":{"arxiv_id":"1612.02401","title":"DeMoN: Depth and Motion Network for Learning Monocular Stereo","citations":751,"named_by":["title_prefix"],"bearer_name":"DeMoN: Depth and Motion Network for Learning Monocular Stereo","ask_with":"have('1612.02401')"},"n_other_bearers_listed":2,"note":"'Demon' is ALSO the name of the paper 'DeMoN: Depth and Motion Network for Learning Monocular Stereo' (arXiv:1612.02401, 751 citations), which is cited FAR more than the paper this answer is about (98). This answer may be about the wrong Demon. If you meant the better-known one, ask again with that entry of `also_named`."},"name_competition_text":"The same name also belongs to the paper “DeMoN: Depth and Motion Network for Learning Monocular Stereo” (arXiv:1612.02401, 751 citations), which the lookup finds cited far more than the paper this answer is about (98 citations), so this answer may be about a different sense of the name.","near_matches":null,"code_nearby":[],"code_nearby_caption":null,"origin_paper":{"arxiv_id":"2308.04152","page":"/paper/empowering-vision-language-models-to-follow","syntology_url":"https://syntology.ai/paper/2308.04152","in_graph_layer":true,"samples":12,"ran":10,"n_ran_checked":2,"n_instrument":8,"unverified":2,"phrase":"10 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 1 violated, 1 with no contract checked; 8 where Syntology's instrument failed) · 2 unverified","instrument_means":"failures of Syntology's instrument, not of the code"},"line":"about another paper, arXiv:2308.04152: harvested code stored, below machine verification (unverified_code; harvested), served level V1. Repositories: 2 listed of 2 in the graph, strongest link: official. The lookup matched a paper by its title, arXiv:2308.04152, not a method entry. About another paper: this answer is about arXiv:2308.04152, while the archive credits arXiv:1910.04952.","level_legend":{"state":"quoted","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","card_sha256":"479785a84658dd682a11b10237b3620d651d7f87799f5dc7add970aaa83f6d39","note":"Syntology's served verification levels, quoted verbatim from its MCP server card; a row is split where the card separates clauses with a double hyphen, and the separator is not reproduced","levels":[{"level":"V0","quoted":["imports"]},{"level":"V1","quoted":["executes against a synthesized fixture and returns a value of the recorded type"]},{"level":"V2","quoted":["two implementations GENERATED from the paper's signature and spec agree numerically (1e-6) on fixture inputs","neither of the two ever read the authors' own code, so V2 measures how unambiguously the paper specifies the method, not fidelity to the authors' implementation (measured 2026-09-11, commit 75eaa77: cross_checked_by is a second LLM on 2,628 of 2,831 served samples)"]},{"level":"V3","quoted":["paper-derived property tests pass on both of those same two generations."]}],"no_rung":"No rung on this ladder compares the served code against the authors' implementation, so treat every level as evidence about runnability and internal agreement, not about matching the paper's results."}},"syntology_have_state":"loaded","record_sha256":"9975df2069a9faf3f1c58d39dc0736b9e53b9f621957ffca27d98460a9336070","record_changed_at":"2026-10-05","record_changed_at_basis":"changed"}