{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/towards-understanding-why-lookahead","title":"Towards Understanding Why Lookahead Generalizes Better Than SGD and Beyond","arxiv_id":null,"date":"2021-12-01","proceeding":"NeurIPS 2021 12","authors":["Pan Zhou","Hanshu Yan","Xiaotong Yuan","Jiashi Feng","Shuicheng Yan"],"abstract":"To train networks, lookahead algorithm~\\cite{zhang2019lookahead} updates its fast weights $k$ times via  an  inner-loop   optimizer before updating its slow weights once by using the latest  fast weights. Any optimizer, e.g. SGD,  can serve as the inner-loop optimizer, and the derived lookahead  generally enjoys remarkable test performance improvement over the vanilla optimizer.  But theoretical understandings on the test performance improvement   of lookahead remain absent yet. To solve this issue, we theoretically justify  the advantages of lookahead  in terms of the excess risk error which measures the test performance. Specifically, we prove that lookahead using  SGD as its inner-loop optimizer can better balance the optimization error and generalization error to achieve smaller excess risk error than vanilla SGD  on (strongly) convex problems and nonconvex problems with Polyak-{\\L}ojasiewicz condition which has been observed/proved in neural networks.   Moreover,  we show the stagewise optimization strategy~\\cite{barshan2015stage} which decays learning rate several times during training can also benefit lookahead in  improving  its optimization and generalization errors on strongly convex problems. Finally, we propose a  stagewise locally-regularized lookahead (SLRLA) algorithm which sums up the vanilla objective and a local regularizer to minimize at each stage and  provably enjoys optimization and generalization improvement  over the conventional (stagewise) lookahead.    Experimental results on   CIFAR10/100 and ImageNet  testify its  advantages. Codes is available at  \\url{https://github.com/sail-sg/SLRLA-optimizer}. ","url_abs":"http://proceedings.neurips.cc/paper/2021/hash/e53a0a2978c28872a4505bdb51db06dc-Abstract.html","url_pdf":"http://proceedings.neurips.cc/paper/2021/file/e53a0a2978c28872a4505bdb51db06dc-Paper.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"towards-understanding-why-lookahead","repo_url":"https://github.com/sail-sg/slrla-optimizer","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[],"methods":[{"method_slug":"sgd","method_name":"SGD"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers","held":null,"reason":"no_arxiv_id","checked_against":null,"by_repo":[{"repo":"sail-sg/slrla-optimizer","kind":"official","samples":0,"ran":0,"instrument":0,"unverified":0}],"by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":null,"arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"0f9808de3fdb2f3d412b8d5c216ef9029a421820b338c3e29f83b92425e05116","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}