{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/parameter-continuation-methods-for-the","title":"Parameter Continuation Methods for the Optimization of Deep Neural Networks","arxiv_id":null,"date":"2019-12-16","proceeding":"2019 18th IEEE International Conference On Machine Learning And Applications (ICMLA) 2019 12","authors":["Harsh Nilesh Pathank","Randy Clinton Paffenroth"],"abstract":"There are many extant methods for approximating the solutions of non-convex optimization problems arising in deep neural networks, including stochastic gradient descent, RMSProp, AdaGrad, and ADAM. In this paper, we propose a novel training strategy for deep neural networks based on ideas from numerical parameter continuation methods. Parameter continuation methods have a long history in many application domains such as bifurcation analysis and the study of systems of differential equations. However, as far as we are aware, such powerful methods have seen relatively limited use in the optimization of deep neural networks. Specifically, herein we derive a homotopy formulation of standard activation functions. Such a derivation allows one to decompose the optimization of deep neural networks into a sequence of optimization problems, each of which is armed with a good initial guess based upon the solution of the previous problem. The whole process is initiated by using a closed-form solution for the first of these problems provided by the homotopy formulation. Intuitively, there is a deep connection between our homotopy techniques and many ideas used in transfer and curriculum learning. However, our proposed methods leverage decades of theoretical and computational work in parameter continuation methods and can be viewed as an initial bridge between those techniques and deep neural networks. In particular, we propose a method that we call Natural Parameter Adaption Continuation with Secant approximation (NPACS). This method provides an effective optimization technique that uses standard algorithms such as ADAM in a novel way to achieve faster and more stable convergence. We demonstrate the effectiveness of our method on standard benchmark problems, and we compute local minima more quickly and with lower train and test loss values than current state-of-the-art techniques in a majority of cases.","url_abs":"https://ieeexplore.ieee.org/abstract/document/8999318","url_pdf":"https://github.com/harsh306/NPACS","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"parameter-continuation-methods-for-the","repo_url":"https://github.com/harsh306/NPACS","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"tf","reach":null}],"tasks":[],"methods":[{"method_slug":"adagrad","method_name":"AdaGrad"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"rmsprop","method_name":"RMSProp"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers","held":null,"reason":"no_arxiv_id","checked_against":null,"by_repo":[{"repo":"harsh306/NPACS","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0}],"by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":null,"arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"1e1fe44779917f949042027833ce162ce318e834ace8fde87d7df1c01f4c6760","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}