{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/method/nesterov-accelerated-gradient","slug":"nesterov-accelerated-gradient","name":"Nesterov Accelerated Gradient","full_name":"Nesterov Accelerated Gradient","full_name_withheld":false,"description_markdown":"**Nesterov Accelerated Gradient** is a momentum-based [SGD](https://paperswithcode.com/method/sgd) optimizer that \"looks ahead\" to where the parameters will be to calculate the gradient **ex post** rather than **ex ante**:\r\n\r\n$$ v\\_{t} = \\gamma{v}\\_{t-1} - \\eta\\nabla\\_{\\theta}J\\left(\\theta_{t-1}+\\gamma{v\\_{t-1}}\\right) $$\r\n$$ \\theta\\_{t} = \\theta\\_{t-1} + v\\_{t} $$\r\n$$ \\gamma, \\eta \\in \\mathbb{R}^+ $$\r\n\r\nLike SGD with momentum $\\gamma$ is usually set to $0.9$. $\\eta$ and $\\gamma$ are usually less than $1$.\r\n\r\nThe intuition is that the [standard momentum](https://paperswithcode.com/method/sgd-with-momentum) method first computes the gradient at the current location and then takes a big jump in the direction of the updated accumulated gradient. In contrast Nesterov momentum first makes a big jump in the direction of the previous accumulated gradient and then measures the gradient where it ends up and makes a correction. The idea being that it is better to correct a mistake after you have made it. \r\n\r\nImage Source: [Geoff Hinton lecture notes](http://www.cs.toronto.edu/~tijmen/csc321/slides/lecture_slides_lec6.pdf)","description_state":"present","introduced_year":1983,"introduced_by":{"title":null,"paper":null,"first_author":null,"n_authors":0,"url_abs":null,"archive_paper_url":null},"source":{"url":null,"title":null,"url_on_a_paper_host":false},"code_snippet_url":null,"code_snippet_url_on_a_code_host":false,"categories":[{"area":"General","area_id":"general","collection":"Large Batch Optimization","url":"/methods/category/large-batch-optimization","pwc_aliases":[]},{"area":"General","area_id":"general","collection":"Stochastic Optimization","url":"/methods/category/stochastic-optimization","pwc_aliases":[]}],"n_papers_tagged":34,"archive_num_papers":34,"papers_newest_first":[{"paper":null,"slug":"convergence-of-momentum-based-optimization","title":"Convergence of Momentum-Based Optimization Algorithms with Time-Varying Parameters","date":"2025-06-13","arxiv_id":"2506.11904","n_code_links":0,"syntology":null},{"paper":"/paper/nesterov-method-for-asynchronous-pipeline","slug":"nesterov-method-for-asynchronous-pipeline","title":"Nesterov Method for Asynchronous Pipeline Parallel Optimization","date":"2025-05-02","arxiv_id":"2505.01099","n_code_links":1,"syntology":null},{"paper":"/paper/advancing-rvfl-networks-robust-classification","slug":"advancing-rvfl-networks-robust-classification","title":"Advancing RVFL networks: Robust classification with the HawkEye loss function","date":"2024-10-01","arxiv_id":"2410.00510","n_code_links":1,"syntology":null},{"paper":"/paper/an-accelerated-algorithm-for-stochastic","slug":"an-accelerated-algorithm-for-stochastic","title":"An Accelerated Algorithm for Stochastic Bilevel Optimization under Unbounded Smoothness","date":"2024-09-28","arxiv_id":"2409.19212","n_code_links":1,"syntology":{"ran":3,"of":4,"n_ran_checked":2,"n_instrument":1,"unverified":1,"pointer_only":0,"phrase":"3 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 1 where Syntology's instrument failed) · 1 unverified","official":{"repos":["mingruiliu-ml-lab/accelerated-bilevel-optimization-unbounded-smoothness"],"state":"official (archive's flag): 3 ran","n_ran":3,"n_constructed":0,"n_ran_no_instrument_failure":2,"n_unverified":1,"ran_from_kinds":["official"]}}},{"paper":"/paper/optimizing-time-series-forecasting-a","slug":"optimizing-time-series-forecasting-a","title":"Optimizing Time Series Forecasting: A Comparative Study of Adam and Nesterov Accelerated Gradient on LSTM and GRU networks Using Stock Market data","date":"2024-09-28","arxiv_id":"2410.01843","n_code_links":1,"syntology":null},{"paper":"/paper/denomamba-a-fused-state-space-model-for-low","slug":"denomamba-a-fused-state-space-model-for-low","title":"DenoMamba: A fused state-space model for low-dose CT denoising","date":"2024-09-19","arxiv_id":"2409.13094","n_code_links":1,"syntology":null},{"paper":null,"slug":"3d-cbct-challenge-2024-improved-cone-beam-ct","title":"3D CBCT Challenge 2024: Improved Cone Beam CT Reconstruction using SwinIR-Based Sinogram and Image Enhancement","date":"2024-06-12","arxiv_id":"2406.08048","n_code_links":0,"syntology":null},{"paper":"/paper/momentum-sam-sharpness-aware-minimization","slug":"momentum-sam-sharpness-aware-minimization","title":"Momentum-SAM: Sharpness Aware Minimization without Computational Overhead","date":"2024-01-22","arxiv_id":"2401.12033","n_code_links":1,"syntology":{"ran":2,"of":2,"n_ran_checked":1,"n_instrument":1,"unverified":0,"pointer_only":2,"phrase":"2 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 1 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified","official":{"repos":["marlonbecker/msam"],"state":"official (archive's flag): 2 ran","n_ran":2,"n_constructed":0,"n_ran_no_instrument_failure":1,"n_unverified":0,"ran_from_kinds":["official"]}}},{"paper":null,"slug":"accelerated-gradient-methods-for-nonconvex-1","title":"Accelerated gradient methods for nonconvex optimization: Escape trajectories from strict saddle points and convergence to local minima","date":"2023-07-13","arxiv_id":"2307.07030","n_code_links":0,"syntology":null},{"paper":null,"slug":"riemannian-accelerated-gradient-methods-via","title":"Riemannian accelerated gradient methods via extrapolation","date":"2022-08-13","arxiv_id":"2208.06619","n_code_links":0,"syntology":null},{"paper":null,"slug":"last-iterate-convergence-analysis-of","title":"Last-iterate convergence analysis of stochastic momentum methods for neural networks","date":"2022-05-30","arxiv_id":"2205.14811","n_code_links":0,"syntology":null},{"paper":null,"slug":"automated-parking-space-detection-using","title":"Automated Parking Space Detection Using Convolutional Neural Networks","date":"2021-06-14","arxiv_id":"2106.07228","n_code_links":0,"syntology":null},{"paper":null,"slug":"a-discrete-variational-derivation-of","title":"A Discrete Variational Derivation of Accelerated Methods in Optimization","date":"2021-06-04","arxiv_id":"2106.02700","n_code_links":0,"syntology":null},{"paper":"/paper/a-large-batch-optimizer-reality-check","slug":"a-large-batch-optimizer-reality-check","title":"A Large Batch Optimizer Reality Check: Traditional, Generic Optimizers Suffice Across Batch Sizes","date":"2021-02-12","arxiv_id":"2102.06356","n_code_links":0,"syntology":null},{"paper":null,"slug":"stochastic-optimization-with-momentum","title":"Stochastic optimization with momentum: convergence, fluctuations, and traps avoidance","date":"2020-12-07","arxiv_id":"2012.04002","n_code_links":0,"syntology":null},{"paper":"/paper/a-dynamical-view-on-optimization-algorithms","slug":"a-dynamical-view-on-optimization-algorithms","title":"A Dynamical View on Optimization Algorithms of Overparameterized Neural Networks","date":"2020-10-25","arxiv_id":"2010.13165","n_code_links":1,"syntology":null},{"paper":"/paper/improving-convergence-for-nonconvex-composite","slug":"improving-convergence-for-nonconvex-composite","title":"Accelerated Gradient Methods for Sparse Statistical Learning with Nonconvex Penalties","date":"2020-09-22","arxiv_id":"2009.10629","n_code_links":1,"syntology":null},{"paper":null,"slug":"federated-learning-with-nesterov-accelerated","title":"Federated Learning with Nesterov Accelerated Gradient","date":"2020-09-18","arxiv_id":"2009.08716","n_code_links":0,"syntology":null},{"paper":"/paper/greedynas-towards-fast-one-shot-nas-with","slug":"greedynas-towards-fast-one-shot-nas-with","title":"GreedyNAS: Towards Fast One-Shot NAS with Greedy Supernet","date":"2020-03-25","arxiv_id":"2003.11236","n_code_links":0,"syntology":null},{"paper":"/paper/scheduled-restart-momentum-for-accelerated","slug":"scheduled-restart-momentum-for-accelerated","title":"Scheduled Restart Momentum for Accelerated Stochastic Gradient Descent","date":"2020-02-24","arxiv_id":"2002.10583","n_code_links":1,"syntology":{"ran":0,"of":1,"n_ran_checked":0,"n_instrument":0,"unverified":1,"pointer_only":1,"phrase":"0 ran · 1 unverified","official":{"repos":["minhtannguyen/SRSGD"],"state":"official: harvested, nothing ran","n_ran":0,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":1,"ran_from_kinds":[]}}},{"paper":null,"slug":"second-order-information-in-first-order","title":"Second-order Information in First-order Optimization Methods","date":"2019-12-20","arxiv_id":"1912.09926","n_code_links":0,"syntology":null},{"paper":"/paper/liteseg-a-novel-lightweight-convnet-for","slug":"liteseg-a-novel-lightweight-convnet-for","title":"LiteSeg: A Novel Lightweight ConvNet for Semantic Segmentation","date":"2019-12-13","arxiv_id":"1912.06683","n_code_links":2,"syntology":null},{"paper":null,"slug":"best-practices-for-convolutional-neural","title":"Best Practices for Convolutional Neural Networks Applied to Object Recognition in Images","date":"2019-10-29","arxiv_id":"1910.13029","n_code_links":0,"syntology":null},{"paper":"/paper/nesterov-accelerated-gradient-and-scale","slug":"nesterov-accelerated-gradient-and-scale","title":"Nesterov Accelerated Gradient and Scale Invariance for Adversarial Attacks","date":"2019-08-17","arxiv_id":"1908.06281","n_code_links":3,"syntology":{"ran":1,"of":1,"n_ran_checked":0,"n_instrument":1,"unverified":0,"pointer_only":1,"phrase":"1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified","official":{"repos":["JHL-HUST/SI-NI-FGSM"],"state":"official (archive's flag): 1 ran","n_ran":1,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":0,"ran_from_kinds":["official"]}}},{"paper":null,"slug":"power-gradient-descent","title":"Power Gradient Descent","date":"2019-06-11","arxiv_id":"1906.04787","n_code_links":0,"syntology":null},{"paper":null,"slug":"dana-scalable-out-of-the-box-distributed-asgd","title":"DANA: Scalable Out-of-the-box Distributed ASGD Without Retuning","date":"2019-05-01","arxiv_id":null,"n_code_links":0,"syntology":null},{"paper":"/paper/bag-of-tricks-for-image-classification-with","slug":"bag-of-tricks-for-image-classification-with","title":"Bag of Tricks for Image Classification with Convolutional Neural Networks","date":"2018-12-04","arxiv_id":"1812.01187","n_code_links":28,"syntology":{"ran":11,"of":15,"n_ran_checked":9,"n_instrument":2,"unverified":4,"pointer_only":5,"phrase":"11 ran (of which 0 constructed an object rather than computing a result; 9 with no instrument failure: 2 honoured, 0 violated, 7 with no contract checked; 2 where Syntology's instrument failed) · 4 unverified","official":{"repos":["dmlc/gluon-cv"],"state":"community repositories only","n_ran":0,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":0,"ran_from_kinds":["listed"]}}},{"paper":"/paper/nesterov-acceleration-of-alternating-least","slug":"nesterov-acceleration-of-alternating-least","title":"Nesterov Acceleration of Alternating Least Squares for Canonical Tensor Decomposition: Momentum Step Size Selection and Restart Mechanisms","date":"2018-10-13","arxiv_id":"1810.05846","n_code_links":1,"syntology":null},{"paper":null,"slug":"on-the-convergence-of-weighted-adagrad-with","title":"A Unified Analysis of AdaGrad with Weighted Aggregation and Momentum Acceleration","date":"2018-08-10","arxiv_id":"1808.03408","n_code_links":0,"syntology":null},{"paper":"/paper/big-little-net-an-efficient-multi-scale","slug":"big-little-net-an-efficient-multi-scale","title":"Big-Little Net: An Efficient Multi-Scale Feature Representation for Visual and Speech Recognition","date":"2018-07-10","arxiv_id":"1807.03848","n_code_links":3,"syntology":null},{"paper":"/paper/language-modeling-with-gated-convolutional","slug":"language-modeling-with-gated-convolutional","title":"Language Modeling with Gated Convolutional Networks","date":"2016-12-23","arxiv_id":"1612.08083","n_code_links":11,"syntology":null},{"paper":"/paper/deep-pyramidal-residual-networks","slug":"deep-pyramidal-residual-networks","title":"Deep Pyramidal Residual Networks","date":"2016-10-10","arxiv_id":"1610.02915","n_code_links":7,"syntology":{"ran":1,"of":1,"n_ran_checked":0,"n_instrument":1,"unverified":0,"pointer_only":1,"phrase":"1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified","official":{"repos":["jhkim89/PyramidNet"],"state":"community repositories only","n_ran":0,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":0,"ran_from_kinds":["listed"]}}},{"paper":"/paper/densely-connected-convolutional-networks","slug":"densely-connected-convolutional-networks","title":"Densely Connected Convolutional Networks","date":"2016-08-25","arxiv_id":"1608.06993","n_code_links":146,"syntology":{"ran":48,"of":71,"n_ran_checked":32,"n_instrument":16,"unverified":23,"pointer_only":8,"phrase":"48 ran (of which 0 constructed an object rather than computing a result; 32 with no instrument failure: 2 honoured, 0 violated, 30 with no contract checked; 16 where Syntology's instrument failed) · 23 unverified","official":{"repos":["liuzhuang13/DenseNet"],"state":"community repositories only","n_ran":0,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":0,"ran_from_kinds":["listed"]}}},{"paper":"/paper/wide-residual-networks","slug":"wide-residual-networks","title":"Wide Residual Networks","date":"2016-05-23","arxiv_id":"1605.07146","n_code_links":72,"syntology":{"ran":66,"of":96,"n_ran_checked":47,"n_instrument":19,"unverified":30,"pointer_only":50,"phrase":"66 ran (of which 31 constructed an object rather than computing a result; 47 with no instrument failure: 0 honoured, 0 violated, 47 with no contract checked; 19 where Syntology's instrument failed) · 30 unverified","official":{"repos":["szagoruyko/wide-residual-networks"],"state":"official (archive's flag): 3 ran","n_ran":3,"n_constructed":0,"n_ran_no_instrument_failure":2,"n_unverified":0,"ran_from_kinds":["listed","official","unlocated"]}}}],"papers_shown":34,"papers_order":"date (newest first), then slug","paper_counts":{"archive_papers_tagged":34,"with_a_code_link":18,"where_syntology_ran_a_sample":7,"not_listed_spam_title":0,"listed":34,"listed_where_code_ran":7,"where_syntology_ran_a_sample_split":{"with_a_run_with_no_instrument_failure":5,"every_run_a_failure_of_syntologys_instrument":2,"listed_with_a_run_with_no_instrument_failure":5,"listed_every_run_a_failure_of_syntologys_instrument":2,"filter":{"states":["a run with no instrument failure","any run, instrument failures included"],"default":"a run with no instrument failure","note":"on the 'only where code ran' pages the default hides, in the browser, the rows where every run was a failure of Syntology's instrument; the second state shows them again. Rows are hidden, never re-ordered; these twins list every row"}},"definition":"distinct papers the archive tags; 'where Syntology ran a sample' counts papers with at least one harvested sample that ran, which is not a correctness claim"},"papers_where_code_ran":7,"papers_where_code_ran_first_page":"/method/nesterov-accelerated-gradient/papers/ran/1","papers_where_code_ran_caption":"We ran code from the paper's repository; we did not isolate this method inside it.","pages":[{"order":"ran","page":1,"of":1,"url":"/method/nesterov-accelerated-gradient/papers/ran/1","json":"/method/nesterov-accelerated-gradient/papers/ran/1.json","rows":7}],"tasks":[{"task":"/task/image-classification","name":"Image Classification","papers":8},{"task":"/task/stochastic-optimization","name":"Stochastic Optimization","papers":5},{"task":"/task/image-classification","name":"image-classification","papers":4},{"task":"/task/classification","name":"General Classification","papers":3},{"task":"/task/object-recognition","name":"Object Recognition","papers":3},{"task":"/task/denoising","name":"Denoising","papers":2},{"task":"/task/language-modelling","name":"Language Modelling","papers":2},{"task":"/task/semantic-segmentation","name":"Semantic Segmentation","papers":2},{"task":"/task/2d-human-pose-estimation","name":"2D Human Pose Estimation","papers":1},{"task":"/task/adversarial-attack","name":"Adversarial Attack","papers":1},{"task":"/task/all","name":"All","papers":1},{"task":"/task/bilevel-optimization","name":"Bilevel Optimization","papers":1},{"task":"/task/breast-tumour-classification","name":"Breast Tumour Classification","papers":1},{"task":"/task/ct-reconstruction","name":"CT Reconstruction","papers":1},{"task":"/task/classification-1","name":"Classification","papers":1},{"task":"/task/computational-efficiency","name":"Computational Efficiency","papers":1},{"task":"/task/crowd-counting","name":"Crowd Counting","papers":1},{"task":"/task/decoder","name":"Decoder","papers":1},{"task":"/task/diagnostic","name":"Diagnostic","papers":1},{"task":"/task/distributed-computing","name":"Distributed Computing","papers":1}],"tasks_shown":20,"n_tasks":59,"usage_by_year":[{"year":"2016","papers":4},{"year":"2018","papers":4},{"year":"2019","papers":6},{"year":"2020","papers":6},{"year":"2021","papers":3},{"year":"2022","papers":2},{"year":"2023","papers":1},{"year":"2024","papers":6},{"year":"2025","papers":2}],"row_source":"methods_table","archive":{"source":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","archive_url":"https://paperswithcode.com/method/nesterov-accelerated-gradient"},"syntology_read_at":"2026-10-05T00:36:11+00:00","syntology":{"source":"Syntology's have check for this name (the free coverage lookup), read from Syntology's graph","have_url":"https://syntology.ai/api/graph/have?x=Nesterov%20Accelerated%20Gradient","read_at":"2026-10-05T00:59:33+00:00","query":"Nesterov Accelerated Gradient","kind":"method","have":"unknown_name","tier":null,"verified_implementation":null,"level":null,"level_meaning":"the served verification level, per answer; never summed across methods","resolved_to":null,"resolution":null,"resolved_name_differs":false,"absence_asserted":false,"absence_unchecked":null,"no_code_is_about_this_method":false,"subject":null,"archive_paper_arxiv_id":null,"repositories":null,"retrieve_with":null,"name_competition":{"verdict":"none","may_be_wrong_sense":false,"bearers_found":0,"served":null,"leading_other_bearer":null,"n_other_bearers_listed":0,"note":"Nothing in this corpus is named 'Nesterov Accelerated Gradient' -- no method node, no paper announcing it in its title, no expanded form declaring it as an acronym."},"name_competition_text":null,"near_matches":["Randomized Subspace Nesterov Accelerated Gradient","Nesterov-accelerated gradient (NAG) enhanced SDE-BNN model","Accelerated Gradient descent algorithm based on Nesterov's smoothing technique","Nesterov-type accelerated scheme","Accelerated Reflected Gradient (ARG) method","Accelerated Optimistic Gradient (AOG) algorithm","Federated Accelerated Stochastic Gradient Descent (FedAc)","Accelerated Bregman Gradient Policy Optimization (VR-BGPO)"],"code_nearby":[],"code_nearby_caption":null,"origin_paper":null,"line":"name not resolved (unknown_name). The lookup matched no method, paper or venue by this name; that asserts nothing about whether code exists.","level_legend":null},"syntology_have_state":"loaded","record_sha256":"c7432ce80a9a94cb354600e13b7d582919feea37ed0601a8a47adaf9db9f373c","record_changed_at":"2026-10-05","record_changed_at_basis":"changed"}