{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/method/adagrad","slug":"adagrad","name":"AdaGrad","full_name":"AdaGrad","full_name_withheld":false,"description_markdown":"**AdaGrad** is a stochastic optimization method that adapts the learning rate to the parameters. It performs smaller updates for parameters associated with frequently occurring features, and larger updates for parameters associated with infrequently occurring features. In its update rule, Adagrad modifies the general learning rate $\\eta$ at each time step $t$ for every parameter $\\theta\\_{i}$ based on the past gradients for $\\theta\\_{i}$: \r\n\r\n$$ \\theta\\_{t+1, i} = \\theta\\_{t, i} - \\frac{\\eta}{\\sqrt{G\\_{t, ii} + \\epsilon}}g\\_{t, i} $$\r\n\r\nThe benefit of AdaGrad is that it eliminates the need to manually tune the learning rate; most leave it at a default value of $0.01$. Its main weakness is the accumulation of the squared gradients in the denominator. Since every added term is positive, the accumulated sum keeps growing during training, causing the learning rate to shrink and becoming infinitesimally small.\r\n\r\nImage: [Alec Radford](https://twitter.com/alecrad)","description_state":"present","introduced_year":2011,"introduced_by":{"title":null,"paper":null,"first_author":null,"n_authors":0,"url_abs":null,"archive_paper_url":null},"source":{"url":null,"title":null,"url_on_a_paper_host":false},"code_snippet_url":"https://github.com/Dawn-Of-Eve/nadir/blob/main/src/nadir/adagrad.py","code_snippet_url_on_a_code_host":true,"categories":[{"area":"General","area_id":"general","collection":"Large Batch Optimization","url":"/methods/category/large-batch-optimization","pwc_aliases":[]},{"area":"General","area_id":"general","collection":"Stochastic Optimization","url":"/methods/category/stochastic-optimization","pwc_aliases":[]}],"n_papers_tagged":191,"archive_num_papers":192,"papers_newest_first":[{"paper":null,"slug":"recursive-bound-constrained-adagrad-with","title":"Recursive Bound-Constrained AdaGrad with Applications to Multilevel and Domain Decomposition Minimization","date":"2025-07-15","arxiv_id":"2507.11513","n_code_links":0,"syntology":null},{"paper":null,"slug":"lightsam-parameter-agnostic-sharpness-aware","title":"LightSAM: Parameter-Agnostic Sharpness-Aware Minimization","date":"2025-05-30","arxiv_id":"2505.24399","n_code_links":0,"syntology":null},{"paper":"/paper/sample-and-computationally-efficient-1","slug":"sample-and-computationally-efficient-1","title":"Sample and Computationally Efficient Continuous-Time Reinforcement Learning with General Function Approximation","date":"2025-05-20","arxiv_id":"2505.14821","n_code_links":1,"syntology":null},{"paper":null,"slug":"complexity-lower-bounds-of-adaptive-gradient","title":"Complexity Lower Bounds of Adaptive Gradient Algorithms for Non-convex Stochastic Optimization under Relaxed Smoothness","date":"2025-05-07","arxiv_id":"2505.04599","n_code_links":0,"syntology":null},{"paper":null,"slug":"structured-preconditioners-in-adaptive","title":"Structured Preconditioners in Adaptive Optimization: A Unified Analysis","date":"2025-03-13","arxiv_id":"2503.10537","n_code_links":0,"syntology":null},{"paper":null,"slug":"tractable-representations-for-convergent","title":"Tractable Representations for Convergent Approximation of Distributional HJB Equations","date":"2025-03-07","arxiv_id":"2503.05563","n_code_links":0,"syntology":null},{"paper":null,"slug":"symmetric-rank-one-quasi-newton-methods-for","title":"Symmetric Rank-One Quasi-Newton Methods for Deep Learning Using Cubic Regularization","date":"2025-02-17","arxiv_id":"2502.12298","n_code_links":0,"syntology":null},{"paper":null,"slug":"integrating-llms-with-its-recent-advances","title":"Integrating LLMs with ITS: Recent Advances, Potentials, Challenges, and Future Directions","date":"2025-01-08","arxiv_id":"2501.04437","n_code_links":0,"syntology":null},{"paper":null,"slug":"towards-simple-and-provable-parameter-free","title":"Towards Simple and Provable Parameter-Free Adaptive Gradient Methods","date":"2024-12-27","arxiv_id":"2412.19444","n_code_links":0,"syntology":null},{"paper":null,"slug":"adaptive-optimization-for-enhanced-efficiency","title":"Adaptive Optimization for Enhanced Efficiency in Large-Scale Language Model Training","date":"2024-12-06","arxiv_id":"2412.04718","n_code_links":0,"syntology":null},{"paper":null,"slug":"a-combined-encoder-and-transformer-approach","title":"A Combined Encoder and Transformer Approach for Coherent and High-Quality Text Generation","date":"2024-11-19","arxiv_id":"2411.12157","n_code_links":0,"syntology":null},{"paper":"/paper/modeling-adagrad-rmsprop-and-adam-with","slug":"modeling-adagrad-rmsprop-and-adam-with","title":"Modeling AdaGrad, RMSProp, and Adam with Integro-Differential Equations","date":"2024-11-14","arxiv_id":"2411.09734","n_code_links":1,"syntology":{"ran":1,"of":1,"n_ran_checked":0,"n_instrument":1,"unverified":0,"pointer_only":0,"phrase":"1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified","official":{"repos":["carlosherediapimienta/numerical_simulations_nonlocal"],"state":"official (archive's flag): 1 ran","n_ran":1,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":0,"ran_from_kinds":["official"]}}},{"paper":null,"slug":"new-insight-in-cervical-cancer-diagnosis","title":"New Insight in Cervical Cancer Diagnosis Using Convolution Neural Network Architecture","date":"2024-10-23","arxiv_id":"2410.17735","n_code_links":0,"syntology":null},{"paper":null,"slug":"preconditioning-for-accelerated-gradient","title":"Preconditioning for Accelerated Gradient Descent Optimization and Regularization","date":"2024-09-30","arxiv_id":"2410.00232","n_code_links":0,"syntology":null},{"paper":null,"slug":"asymptotic-and-non-asymptotic-convergence","title":"Stability and convergence analysis of AdaGrad for non-convex optimization via novel stopping time-based techniques","date":"2024-09-08","arxiv_id":"2409.05023","n_code_links":0,"syntology":null},{"paper":"/paper/causal-temporal-representation-learning-with","slug":"causal-temporal-representation-learning-with","title":"Causal Temporal Representation Learning with Nonstationary Sparse Transition","date":"2024-09-05","arxiv_id":"2409.03142","n_code_links":1,"syntology":{"ran":7,"of":9,"n_ran_checked":3,"n_instrument":4,"unverified":2,"pointer_only":9,"phrase":"7 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 0 violated, 3 with no contract checked; 4 where Syntology's instrument failed) · 2 unverified","official":{"repos":["xiangchensong/ctrlns"],"state":"official (archive's flag): 7 ran","n_ran":7,"n_constructed":0,"n_ran_no_instrument_failure":3,"n_unverified":2,"ran_from_kinds":["official"]}}},{"paper":"/paper/machine-learning-models-for-daily-rainfall","slug":"machine-learning-models-for-daily-rainfall","title":"Machine learning models for daily rainfall forecasting in Northern Tropical Africa using tropical wave predictors","date":"2024-08-29","arxiv_id":"2408.16349","n_code_links":1,"syntology":null},{"paper":null,"slug":"a-methodology-establishing-linear-convergence","title":"A Methodology Establishing Linear Convergence of Adaptive Gradient Methods under PL Inequality","date":"2024-07-17","arxiv_id":"2407.12629","n_code_links":0,"syntology":null},{"paper":null,"slug":"large-batch-analysis-for-adagrad-under","title":"AdaGrad under Anisotropic Smoothness","date":"2024-06-21","arxiv_id":"2406.15244","n_code_links":0,"syntology":null},{"paper":null,"slug":"convergence-analysis-of-adaptive-gradient","title":"Provable Complexity Improvement of AdaGrad over SGD: Upper and Lower Bounds in Stochastic Non-Convex Optimization","date":"2024-06-07","arxiv_id":"2406.04592","n_code_links":0,"syntology":null},{"paper":"/paper/gradient-clipping-improves-adagrad-when-the","slug":"gradient-clipping-improves-adagrad-when-the","title":"Clipping Improves Adam-Norm and AdaGrad-Norm when the Noise Is Heavy-Tailed","date":"2024-06-06","arxiv_id":"2406.04443","n_code_links":1,"syntology":null},{"paper":null,"slug":"tight-bounds-for-online-convex-optimization","title":"Tight Bounds for Online Convex Optimization with Adversarial Constraints","date":"2024-05-15","arxiv_id":"2405.09296","n_code_links":0,"syntology":null},{"paper":null,"slug":"ctrl-continuous-time-representation-learning","title":"CTRL: Continuous-Time Representation Learning on Temporal Heterogeneous Information Network","date":"2024-05-11","arxiv_id":"2405.08013","n_code_links":0,"syntology":null},{"paper":"/paper/classp-a-biologically-inspired-approach-to-1","slug":"classp-a-biologically-inspired-approach-to-1","title":"CLASSP: a Biologically-Inspired Approach to Continual Learning through Adjustment Suppression and Sparsity Promotion","date":"2024-04-29","arxiv_id":"2405.09637","n_code_links":1,"syntology":null},{"paper":null,"slug":"deep-learning-with-parametric-lenses","title":"Deep Learning with Parametric Lenses","date":"2024-03-30","arxiv_id":"2404.00408","n_code_links":0,"syntology":null},{"paper":"/paper/an-embarrassingly-simple-defense-against","slug":"an-embarrassingly-simple-defense-against","title":"Towards Adversarial Robustness And Backdoor Mitigation in SSL","date":"2024-03-23","arxiv_id":"2403.15918","n_code_links":1,"syntology":null},{"paper":null,"slug":"adaptive-federated-learning-over-the-air","title":"Adaptive Federated Learning Over the Air","date":"2024-03-11","arxiv_id":"2403.06528","n_code_links":0,"syntology":null},{"paper":"/paper/remove-that-square-root-a-new-efficient-scale","slug":"remove-that-square-root-a-new-efficient-scale","title":"Remove that Square Root: A New Efficient Scale-Invariant Version of AdaGrad","date":"2024-03-05","arxiv_id":"2403.02648","n_code_links":1,"syntology":{"ran":2,"of":2,"n_ran_checked":2,"n_instrument":0,"unverified":0,"pointer_only":2,"phrase":"2 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 2 violated, 0 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified","official":{"repos":["nazya/kate"],"state":"official (archive's flag): 2 ran","n_ran":2,"n_constructed":0,"n_ran_no_instrument_failure":2,"n_unverified":0,"ran_from_kinds":["official"]}}},{"paper":null,"slug":"revisiting-convergence-of-adagrad-with","title":"Revisiting Convergence of AdaGrad with Relaxed Assumptions","date":"2024-02-21","arxiv_id":"2402.13794","n_code_links":0,"syntology":null},{"paper":"/paper/stochastic-hessian-fitting-on-lie-group","slug":"stochastic-hessian-fitting-on-lie-group","title":"Stochastic Hessian Fittings with Lie Groups","date":"2024-02-19","arxiv_id":"2402.11858","n_code_links":1,"syntology":null},{"paper":null,"slug":"adadagrad-adaptive-batch-size-schemes-for","title":"AdAdaGrad: Adaptive Batch Size Schemes for Adaptive Gradient Methods","date":"2024-02-17","arxiv_id":"2402.11215","n_code_links":0,"syntology":null},{"paper":null,"slug":"non-convergence-to-global-minimizers-for-adam","title":"Non-convergence to global minimizers for Adam and stochastic gradient descent optimization and constructions of local minimizers in the training of artificial neural networks","date":"2024-02-07","arxiv_id":"2402.05155","n_code_links":0,"syntology":null},{"paper":"/paper/two-trades-is-not-baffled-condense-graph-via","slug":"two-trades-is-not-baffled-condense-graph-via","title":"Two Trades is not Baffled: Condensing Graph via Crafting Rational Gradient Matching","date":"2024-02-07","arxiv_id":"2402.04924","n_code_links":1,"syntology":{"ran":3,"of":3,"n_ran_checked":2,"n_instrument":1,"unverified":0,"pointer_only":1,"phrase":"3 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 2 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified","official":{"repos":["nus-hpc-ai-lab/ctrl"],"state":"official (archive's flag): 3 ran","n_ran":3,"n_constructed":0,"n_ran_no_instrument_failure":2,"n_unverified":0,"ran_from_kinds":["official"]}}},{"paper":"/paper/non-asymptotic-analysis-of-biased-adaptive","slug":"non-asymptotic-analysis-of-biased-adaptive","title":"Non-asymptotic Analysis of Biased Adaptive Stochastic Approximation","date":"2024-02-05","arxiv_id":"2402.02857","n_code_links":1,"syntology":null},{"paper":null,"slug":"enhancing-stochastic-gradient-descent-a","title":"Enhancing Stochastic Gradient Descent: A Unified Framework and Novel Acceleration Methods for Faster Convergence","date":"2024-02-02","arxiv_id":"2402.01515","n_code_links":0,"syntology":null},{"paper":"/paper/hift-a-hierarchical-full-parameter-fine","slug":"hift-a-hierarchical-full-parameter-fine","title":"HiFT: A Hierarchical Full Parameter Fine-Tuning Strategy","date":"2024-01-26","arxiv_id":"2401.15207","n_code_links":1,"syntology":{"ran":2,"of":3,"n_ran_checked":2,"n_instrument":0,"unverified":1,"pointer_only":0,"phrase":"2 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified","official":{"repos":["misonsky/HiFT"],"state":"official (archive's flag): 2 ran","n_ran":2,"n_constructed":0,"n_ran_no_instrument_failure":2,"n_unverified":1,"ran_from_kinds":["official"]}}},{"paper":"/paper/sania-polyak-type-optimization-framework","slug":"sania-polyak-type-optimization-framework","title":"SANIA: Polyak-type Optimization Framework Leads to Scale Invariant Stochastic Algorithms","date":"2023-12-28","arxiv_id":"2312.17369","n_code_links":1,"syntology":{"ran":10,"of":10,"n_ran_checked":6,"n_instrument":4,"unverified":0,"pointer_only":10,"phrase":"10 ran (of which 0 constructed an object rather than computing a result; 6 with no instrument failure: 0 honoured, 0 violated, 6 with no contract checked; 4 where Syntology's instrument failed) · 0 unverified","official":{"repos":["fxrshed/sania"],"state":"official (archive's flag): 10 ran","n_ran":10,"n_constructed":0,"n_ran_no_instrument_failure":6,"n_unverified":0,"ran_from_kinds":["official"]}}},{"paper":null,"slug":"playing-in-the-dark-no-regret-learning-with","title":"Optimal Algorithms for Online Convex Optimization with Adversarial Constraints","date":"2023-10-29","arxiv_id":"2310.18955","n_code_links":0,"syntology":null},{"paper":null,"slug":"towards-hyperparameter-agnostic-dnn-training","title":"Towards Hyperparameter-Agnostic DNN Training via Dynamical System Insights","date":"2023-10-21","arxiv_id":"2310.13901","n_code_links":0,"syntology":null},{"paper":"/paper/a-parallelized-adam-based-solver-for-reserve","slug":"a-parallelized-adam-based-solver-for-reserve","title":"A Parallelized, Adam-Based Solver for Reserve and Security Constrained AC Unit Commitment","date":"2023-10-10","arxiv_id":"2310.06650","n_code_links":1,"syntology":null},{"paper":"/paper/stochastic-gradient-descent-with","slug":"stochastic-gradient-descent-with","title":"Stochastic Gradient Descent with Preconditioned Polyak Step-size","date":"2023-10-03","arxiv_id":"2310.02093","n_code_links":1,"syntology":{"ran":13,"of":15,"n_ran_checked":13,"n_instrument":0,"unverified":2,"pointer_only":15,"phrase":"13 ran (of which 0 constructed an object rather than computing a result; 13 with no instrument failure: 0 honoured, 0 violated, 13 with no contract checked; 0 where Syntology's instrument failed) · 2 unverified","official":{"repos":["fxrshed/scaledsps"],"state":"official (archive's flag): 13 ran","n_ran":13,"n_constructed":0,"n_ran_no_instrument_failure":13,"n_unverified":2,"ran_from_kinds":["official"]}}},{"paper":"/paper/a-distributed-data-parallel-pytorch","slug":"a-distributed-data-parallel-pytorch","title":"A Distributed Data-Parallel PyTorch Implementation of the Distributed Shampoo Optimizer for Training Neural Networks At-Scale","date":"2023-09-12","arxiv_id":"2309.06497","n_code_links":3,"syntology":{"ran":0,"of":2,"n_ran_checked":0,"n_instrument":0,"unverified":2,"pointer_only":2,"phrase":"0 ran · 2 unverified","official":{"repos":["facebookresearch/optimizers"],"state":"official: harvested, nothing ran","n_ran":0,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":2,"ran_from_kinds":[]}}},{"paper":null,"slug":"adaptively-optimised-adaptive-importance","title":"Adaptively Optimised Adaptive Importance Samplers","date":"2023-07-18","arxiv_id":"2307.09341","n_code_links":0,"syntology":null},{"paper":"/paper/error-feedback-can-accurately-compress","slug":"error-feedback-can-accurately-compress","title":"Error Feedback Can Accurately Compress Preconditioners","date":"2023-06-09","arxiv_id":"2306.06098","n_code_links":1,"syntology":{"ran":2,"of":3,"n_ran_checked":1,"n_instrument":1,"unverified":1,"pointer_only":0,"phrase":"2 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 1 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 1 unverified","official":{"repos":["ist-daslab/efcp"],"state":"official (archive's flag): 2 ran","n_ran":2,"n_constructed":0,"n_ran_no_instrument_failure":1,"n_unverified":1,"ran_from_kinds":["official"]}}},{"paper":"/paper/prodigy-an-expeditiously-adaptive-parameter","slug":"prodigy-an-expeditiously-adaptive-parameter","title":"Prodigy: An Expeditiously Adaptive Parameter-Free Learner","date":"2023-06-09","arxiv_id":"2306.06101","n_code_links":1,"syntology":null},{"paper":null,"slug":"ctrl-connect-tabular-and-language-model-for","title":"CTRL: Connect Collaborative and Language Model for CTR Prediction","date":"2023-06-05","arxiv_id":"2306.02841","n_code_links":0,"syntology":null},{"paper":null,"slug":"parameter-free-projected-gradient-descent","title":"Parameter-free projected gradient descent","date":"2023-05-31","arxiv_id":"2305.19605","n_code_links":0,"syntology":null},{"paper":null,"slug":"convergence-of-adagrad-for-non-convex","title":"Convergence of AdaGrad for Non-convex Objectives: Simple Proofs and Relaxed Assumptions","date":"2023-05-29","arxiv_id":"2305.18471","n_code_links":0,"syntology":null},{"paper":"/paper/dowg-unleashed-an-efficient-universal","slug":"dowg-unleashed-an-efficient-universal","title":"DoWG Unleashed: An Efficient Universal Parameter-Free Gradient Descent Method","date":"2023-05-25","arxiv_id":"2305.16284","n_code_links":3,"syntology":null},{"paper":null,"slug":"noddle-node2vec-based-deep-learning-model-for","title":"NODDLE: Node2vec based deep learning model for link prediction","date":"2023-05-25","arxiv_id":"2305.16421","n_code_links":0,"syntology":null},{"paper":null,"slug":"two-sides-of-one-coin-the-limits-of-untuned","title":"Two Sides of One Coin: the Limits of Untuned SGD and the Power of Adaptive Methods","date":"2023-05-21","arxiv_id":"2305.12475","n_code_links":0,"syntology":null},{"paper":null,"slug":"stochastic-ratios-tracking-algorithm-for","title":"Stochastic Ratios Tracking Algorithm for Large Scale Machine Learning Problems","date":"2023-05-17","arxiv_id":"2305.09978","n_code_links":0,"syntology":null},{"paper":null,"slug":"optimized-machine-learning-for-chd-detection","title":"Optimized Machine Learning for CHD Detection using 3D CNN-based Segmentation, Transfer Learning and Adagrad Optimization","date":"2023-04-30","arxiv_id":"2305.00411","n_code_links":0,"syntology":null},{"paper":"/paper/swectrl-mini-a-data-transparent-transformer","slug":"swectrl-mini-a-data-transparent-transformer","title":"SweCTRL-Mini: a data-transparent Transformer-based large language model for controllable text generation in Swedish","date":"2023-04-27","arxiv_id":"2304.13994","n_code_links":1,"syntology":null},{"paper":"/paper/once-detected-never-lost-surpassing-human","slug":"once-detected-never-lost-surpassing-human","title":"Once Detected, Never Lost: Surpassing Human Performance in Offline LiDAR based 3D Object Detection","date":"2023-04-24","arxiv_id":"2304.12315","n_code_links":2,"syntology":{"ran":3,"of":5,"n_ran_checked":0,"n_instrument":3,"unverified":2,"pointer_only":0,"phrase":"3 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 2 unverified","official":{"repos":["tusen-ai/sst"],"state":"official (archive's flag): 3 ran","n_ran":3,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":2,"ran_from_kinds":["official"]}}},{"paper":"/paper/elastic-weight-removal-for-faithful-and","slug":"elastic-weight-removal-for-faithful-and","title":"Elastic Weight Removal for Faithful and Abstractive Dialogue Generation","date":"2023-03-30","arxiv_id":"2303.17574","n_code_links":1,"syntology":null},{"paper":null,"slug":"mixing-backward-with-forward-chaining-for","title":"Mixing Backward- with Forward-Chaining for Metacognitive Skill Acquisition and Transfer","date":"2023-03-18","arxiv_id":"2303.12223","n_code_links":0,"syntology":null},{"paper":null,"slug":"non-asymptotic-analysis-of-adaptive","title":"Non asymptotic analysis of Adaptive stochastic gradient algorithms and applications","date":"2023-03-01","arxiv_id":"2303.01370","n_code_links":0,"syntology":null},{"paper":null,"slug":"high-probability-convergence-of-stochastic","title":"High Probability Convergence of Stochastic Gradient Methods","date":"2023-02-28","arxiv_id":"2302.14843","n_code_links":0,"syntology":null},{"paper":null,"slug":"sgd-with-adagrad-stepsizes-full-adaptivity","title":"SGD with AdaGrad Stepsizes: Full Adaptivity with High Probability to Unknown Parameters, Unbounded Gradients and Affine Variance","date":"2023-02-17","arxiv_id":"2302.08783","n_code_links":0,"syntology":null},{"paper":null,"slug":"multilevel-objective-function-free","title":"Multilevel Objective-Function-Free Optimization with an Application to Neural Networks Training","date":"2023-02-14","arxiv_id":"2302.07049","n_code_links":0,"syntology":null},{"paper":"/paper/automatically-bounding-the-taylor-remainder","slug":"automatically-bounding-the-taylor-remainder","title":"Automatically Bounding the Taylor Remainder Series: Tighter Bounds and New Applications","date":"2022-12-22","arxiv_id":"2212.11429","n_code_links":1,"syntology":null},{"paper":null,"slug":"adatask-a-task-aware-adaptive-learning-rate","title":"AdaTask: A Task-aware Adaptive Learning Rate Approach to Multi-task Learning","date":"2022-11-28","arxiv_id":"2211.15055","n_code_links":0,"syntology":null},{"paper":null,"slug":"alternative-formulations-for-gilthead","title":"Alternative formulations for gilthead seabream diets: towards a more sustainable production","date":"2022-11-03","arxiv_id":"2211.02430","n_code_links":0,"syntology":null},{"paper":"/paper/controllable-factuality-in-document-grounded","slug":"controllable-factuality-in-document-grounded","title":"Controllable Factuality in Document-Grounded Dialog Systems Using a Noisy Channel Model","date":"2022-10-31","arxiv_id":"2210.17418","n_code_links":1,"syntology":null},{"paper":"/paper/unsupervised-learning-of-structured","slug":"unsupervised-learning-of-structured","title":"Unsupervised Learning of Structured Representations via Closed-Loop Transcription","date":"2022-10-30","arxiv_id":"2210.16782","n_code_links":1,"syntology":null},{"paper":"/paper/demystifying-self-supervised-trojan-attacks","slug":"demystifying-self-supervised-trojan-attacks","title":"An Embarrassingly Simple Backdoor Attack on Self-supervised Learning","date":"2022-10-13","arxiv_id":"2210.07346","n_code_links":4,"syntology":{"ran":2,"of":2,"n_ran_checked":0,"n_instrument":2,"unverified":0,"pointer_only":2,"phrase":"2 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified","official":{"repos":["meet-cjli/ctrl","CCCjiang/CTRL"],"state":"community repositories only","n_ran":0,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":0,"ran_from_kinds":["listed"]}}},{"paper":null,"slug":"divergence-results-and-convergence-of-a","title":"Divergence Results and Convergence of a Variance Reduced Version of ADAM","date":"2022-10-11","arxiv_id":"2210.05607","n_code_links":0,"syntology":null},{"paper":null,"slug":"on-the-convergence-of-adagrad-on-r-d-beyond","title":"On the Convergence of AdaGrad(Norm) on $\\R^{d}$: Beyond Convexity, Non-Asymptotic Rate and Acceleration","date":"2022-09-29","arxiv_id":"2209.14827","n_code_links":0,"syntology":null},{"paper":"/paper/continual-learning-fast-and-slow","slug":"continual-learning-fast-and-slow","title":"Continual Learning, Fast and Slow","date":"2022-09-06","arxiv_id":"2209.02370","n_code_links":1,"syntology":null},{"paper":null,"slug":"dynamic-regret-of-adaptive-gradient-methods","title":"Dynamic Regret of Adaptive Gradient Methods for Strongly Convex Problems","date":"2022-09-04","arxiv_id":"2209.01608","n_code_links":0,"syntology":null},{"paper":"/paper/ctrl-clustering-training-losses-for-label","slug":"ctrl-clustering-training-losses-for-label","title":"CTRL: Clustering Training Losses for Label Error Detection","date":"2022-08-17","arxiv_id":"2208.08464","n_code_links":1,"syntology":null},{"paper":"/paper/multinomial-logistic-regression-algorithms","slug":"multinomial-logistic-regression-algorithms","title":"Multinomial Logistic Regression Algorithms via Quadratic Gradient","date":"2022-08-14","arxiv_id":"2208.06828","n_code_links":1,"syntology":null},{"paper":null,"slug":"quantized-adaptive-subgradient-algorithms-and","title":"Quantized Adaptive Subgradient Algorithms and Their Applications","date":"2022-08-11","arxiv_id":"2208.05631","n_code_links":0,"syntology":null},{"paper":null,"slug":"formal-guarantees-for-heuristic-optimization","title":"Formal guarantees for heuristic optimization algorithms used in machine learning","date":"2022-07-31","arxiv_id":"2208.00502","n_code_links":0,"syntology":null},{"paper":null,"slug":"accelerated-federated-learning-with-decoupled","title":"Accelerated Federated Learning with Decoupled Adaptive Optimization","date":"2022-07-14","arxiv_id":"2207.07223","n_code_links":0,"syntology":null},{"paper":null,"slug":"bfe-and-adabfe-a-new-approach-in-learning","title":"BFE and AdaBFE: A New Approach in Learning Rate Automation for Stochastic Optimization","date":"2022-07-06","arxiv_id":"2207.02763","n_code_links":0,"syntology":null},{"paper":"/paper/pursuit-of-a-discriminative-representation","slug":"pursuit-of-a-discriminative-representation","title":"Pursuit of a Discriminative Representation for Multiple Subspaces via Sequential Games","date":"2022-06-18","arxiv_id":"2206.09120","n_code_links":1,"syntology":{"ran":1,"of":1,"n_ran_checked":1,"n_instrument":0,"unverified":0,"pointer_only":0,"phrase":"1 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified","official":{"repos":["DruvPai/MultipleSubspaceRepresentationPursuit"],"state":"official (archive's flag): 1 ran","n_ran":1,"n_constructed":0,"n_ran_no_instrument_failure":1,"n_unverified":0,"ran_from_kinds":["official"]}}},{"paper":null,"slug":"grad-gradagrad-a-non-monotone-adaptive","title":"Grad-GradaGrad? A Non-Monotone Adaptive Stochastic Gradient Method","date":"2022-06-14","arxiv_id":"2206.06900","n_code_links":0,"syntology":null},{"paper":null,"slug":"a-control-theoretic-framework-for-adaptive","title":"A Control Theoretic Framework for Adaptive Gradient Optimizers in Machine Learning","date":"2022-06-04","arxiv_id":"2206.02034","n_code_links":0,"syntology":null},{"paper":null,"slug":"nest-your-adaptive-algorithm-for-parameter","title":"Nest Your Adaptive Algorithm for Parameter-Agnostic Nonconvex Minimax Optimization","date":"2022-06-01","arxiv_id":"2206.00743","n_code_links":0,"syntology":null},{"paper":null,"slug":"incorporating-the-barzilai-borwein-adaptive","title":"Incorporating the Barzilai-Borwein Adaptive Step Size into Sugradient Methods for Deep Network Training","date":"2022-05-27","arxiv_id":"2205.13711","n_code_links":0,"syntology":null},{"paper":"/paper/symmetry-teleportation-for-accelerated","slug":"symmetry-teleportation-for-accelerated","title":"Symmetry Teleportation for Accelerated Optimization","date":"2022-05-21","arxiv_id":"2205.10637","n_code_links":1,"syntology":{"ran":5,"of":5,"n_ran_checked":2,"n_instrument":3,"unverified":0,"pointer_only":5,"phrase":"5 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 2 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified","official":{"repos":["rose-stl-lab/symmetry-teleportation"],"state":"official (archive's flag): 5 ran","n_ran":5,"n_constructed":0,"n_ran_no_instrument_failure":2,"n_unverified":0,"ran_from_kinds":["official"]}}},{"paper":null,"slug":"sampling-with-attribute-related-information","title":"Efficient and Training-Free Control of Language Generation","date":"2022-05-12","arxiv_id":"2205.06036","n_code_links":0,"syntology":null},{"paper":null,"slug":"high-probability-bounds-for-a-class-of-1","title":"High Probability Bounds for a Class of Nonconvex Algorithms with AdaGrad Stepsize","date":"2022-04-06","arxiv_id":"2204.02833","n_code_links":0,"syntology":null},{"paper":"/paper/adasmooth-an-adaptive-learning-rate-method","slug":"adasmooth-an-adaptive-learning-rate-method","title":"AdaSmooth: An Adaptive Learning Rate Method based on Effective Ratio","date":"2022-04-02","arxiv_id":"2204.00825","n_code_links":0,"syntology":null},{"paper":null,"slug":"convergence-and-complexity-of-stochastic-1","title":"Convergence of First-Order Methods for Constrained Nonconvex Optimization with Dependent Data","date":"2022-03-29","arxiv_id":"2203.15797","n_code_links":0,"syntology":null},{"paper":null,"slug":"a-mini-block-natural-gradient-method-for-deep","title":"A Mini-Block Fisher Method for Deep Neural Networks","date":"2022-02-08","arxiv_id":"2202.04124","n_code_links":0,"syntology":null},{"paper":null,"slug":"step-size-adaptation-using-exponentiated","title":"Step-size Adaptation Using Exponentiated Gradient Updates","date":"2022-01-31","arxiv_id":"2202.00145","n_code_links":0,"syntology":null},{"paper":null,"slug":"on-the-convergence-of-msgd-and-adagrad-for-1","title":"On the Convergence of mSGD and AdaGrad for Stochastic Optimization","date":"2022-01-26","arxiv_id":"2201.11204","n_code_links":0,"syntology":null},{"paper":"/paper/convergence-and-complexity-of-stochastic","slug":"convergence-and-complexity-of-stochastic","title":"Stochastic regularized majorization-minimization with weakly convex and multi-convex surrogates","date":"2022-01-05","arxiv_id":"2201.01652","n_code_links":1,"syntology":null},{"paper":"/paper/adaptively-customizing-activation-functions","slug":"adaptively-customizing-activation-functions","title":"Adaptively Customizing Activation Functions for Various Layers","date":"2021-12-17","arxiv_id":"2112.09442","n_code_links":1,"syntology":null},{"paper":null,"slug":"a-comparative-study-of-transformers-on-word","title":"A Comparative Study of Transformers on Word Sense Disambiguation","date":"2021-11-30","arxiv_id":"2111.15417","n_code_links":0,"syntology":null},{"paper":null,"slug":"adaptive-differentially-private-empirical","title":"Adaptive Differentially Private Empirical Risk Minimization","date":"2021-10-14","arxiv_id":"2110.07435","n_code_links":0,"syntology":null},{"paper":null,"slug":"kkt-conditions-first-order-and-second-order","title":"KKT Conditions, First-Order and Second-Order Optimization, and Distributed Optimization: Tutorial and Survey","date":"2021-10-05","arxiv_id":"2110.01858","n_code_links":0,"syntology":null},{"paper":null,"slug":"dynamic-regret-analysis-for-online-meta","title":"Dynamic Regret Analysis for Online Meta-Learning","date":"2021-09-29","arxiv_id":"2109.14375","n_code_links":0,"syntology":null},{"paper":null,"slug":"l-sr1-adaptive-regularization-by-cubics-for","title":"L-SR1 Adaptive Regularization by Cubics for Deep Learning","date":"2021-09-29","arxiv_id":null,"n_code_links":0,"syntology":null},{"paper":"/paper/turingbench-a-benchmark-environment-for","slug":"turingbench-a-benchmark-environment-for","title":"TURINGBENCH: A Benchmark Environment for Turing Test in the Age of Neural Text Generation","date":"2021-09-27","arxiv_id":"2109.13296","n_code_links":3,"syntology":{"ran":6,"of":7,"n_ran_checked":6,"n_instrument":0,"unverified":1,"pointer_only":0,"phrase":"6 ran (of which 0 constructed an object rather than computing a result; 6 with no instrument failure: 0 honoured, 0 violated, 6 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified","official":null}},{"paper":null,"slug":"on-the-convergence-of-decentralized-adaptive","title":"On the Convergence of Decentralized Adaptive Gradient Methods","date":"2021-09-07","arxiv_id":"2109.03194","n_code_links":0,"syntology":null},{"paper":"/paper/tom-leveraging-trend-of-the-observed","slug":"tom-leveraging-trend-of-the-observed","title":"Tom: Leveraging trend of the observed gradients for faster convergence","date":"2021-09-07","arxiv_id":"2109.03820","n_code_links":2,"syntology":null}],"papers_shown":100,"papers_order":"date (newest first), then slug","paper_counts":{"archive_papers_tagged":191,"with_a_code_link":70,"where_syntology_ran_a_sample":23,"not_listed_spam_title":0,"listed":191,"listed_where_code_ran":23,"where_syntology_ran_a_sample_split":{"with_a_run_with_no_instrument_failure":19,"every_run_a_failure_of_syntologys_instrument":4,"listed_with_a_run_with_no_instrument_failure":19,"listed_every_run_a_failure_of_syntologys_instrument":4,"filter":{"states":["a run with no instrument failure","any run, instrument failures included"],"default":"a run with no instrument failure","note":"on the 'only where code ran' pages the default hides, in the browser, the rows where every run was a failure of Syntology's instrument; the second state shows them again. Rows are hidden, never re-ordered; these twins list every row"}},"definition":"distinct papers the archive tags; 'where Syntology ran a sample' counts papers with at least one harvested sample that ran, which is not a correctness claim"},"papers_where_code_ran":23,"papers_where_code_ran_first_page":"/method/adagrad/papers/ran/1","papers_where_code_ran_caption":"We ran code from the paper's repository; we did not isolate this method inside it.","pages":[{"order":"archive","page":2,"of":2,"url":"/method/adagrad/papers/2","json":"/method/adagrad/papers/2.json","rows":91},{"order":"ran","page":1,"of":1,"url":"/method/adagrad/papers/ran/1","json":"/method/adagrad/papers/ran/1.json","rows":23}],"tasks":[{"task":"/task/stochastic-optimization","name":"Stochastic Optimization","papers":26},{"task":"/task/language-modelling","name":"Language Modelling","papers":14},{"task":"/task/language-modeling","name":"Language Modeling","papers":13},{"task":"/task/machine-learning","name":"BIG-bench Machine Learning","papers":7},{"task":"/task/image-classification","name":"Image Classification","papers":7},{"task":"/task/representation-learning","name":"Representation Learning","papers":7},{"task":"/task/second-order-methods","name":"Second-order methods","papers":7},{"task":"/task/computational-efficiency","name":"Computational Efficiency","papers":6},{"task":"/task/image-classification","name":"image-classification","papers":6},{"task":"/task/text-generation","name":"Text Generation","papers":5},{"task":"/task/translation","name":"Translation","papers":4},{"task":"/task/deep-learning","name":"Deep Learning","papers":3},{"task":"/task/distributed-optimization","name":"Distributed Optimization","papers":3},{"task":"/task/federated-learning","name":"Federated Learning","papers":3},{"task":null,"name":"GPU","papers":3},{"task":"/task/link-prediction","name":"Link Prediction","papers":3},{"task":"/task/machine-translation","name":"Machine Translation","papers":3},{"task":"/task/multi-task-learning","name":"Multi-Task Learning","papers":3},{"task":"/task/prediction","name":"Prediction","papers":3},{"task":"/task/self-supervised-learning","name":"Self-Supervised Learning","papers":3}],"tasks_shown":20,"n_tasks":118,"usage_by_year":[{"year":"2013","papers":1},{"year":"2015","papers":4},{"year":"2016","papers":4},{"year":"2017","papers":8},{"year":"2018","papers":12},{"year":"2019","papers":20},{"year":"2020","papers":22},{"year":"2021","papers":29},{"year":"2022","papers":30},{"year":"2023","papers":25},{"year":"2024","papers":28},{"year":"2025","papers":8}],"row_source":"methods_table","archive":{"source":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","archive_url":"https://paperswithcode.com/method/adagrad"},"syntology_read_at":"2026-10-05T00:36:11+00:00","syntology":{"source":"Syntology's have check for this name (the free coverage lookup), read from Syntology's graph","have_url":"https://syntology.ai/api/graph/have?x=AdaGrad","read_at":"2026-10-05T00:51:27+00:00","query":"AdaGrad","kind":"method","have":"unknown_name","tier":null,"verified_implementation":null,"level":null,"level_meaning":"the served verification level, per answer; never summed across methods","resolved_to":null,"resolution":null,"resolved_name_differs":false,"absence_asserted":false,"absence_unchecked":null,"no_code_is_about_this_method":false,"subject":null,"archive_paper_arxiv_id":null,"repositories":null,"retrieve_with":null,"name_competition":{"verdict":"none","may_be_wrong_sense":false,"bearers_found":0,"served":null,"leading_other_bearer":null,"n_other_bearers_listed":0,"note":"Nothing in this corpus is named 'AdaGrad' -- no method node, no paper announcing it in its title, no expanded form declaring it as an acronym."},"name_competition_text":null,"near_matches":["AdaGrad-Diff","AdaGrad variant","Piecewise AdaGrad"],"code_nearby":[],"code_nearby_caption":null,"origin_paper":null,"line":"name not resolved (unknown_name). The lookup matched no method, paper or venue by this name; that asserts nothing about whether code exists.","level_legend":null},"syntology_have_state":"loaded","record_sha256":"a97b6e19320994f117a4134faaf7be50c9bc9ee33d3622a342050b1e3d5bfd16","record_changed_at":"2026-10-05","record_changed_at_basis":"changed"}