{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/method/sgd","slug":"sgd","name":"SGD","full_name":"Stochastic Gradient Descent","full_name_withheld":false,"description_markdown":"**Stochastic Gradient Descent** is an iterative optimization technique that uses minibatches of data to form an expectation of the gradient, rather than the full gradient using all available data. That is for weights $w$ and a loss function $L$ we have:\r\n\r\n$$ w\\_{t+1} = w\\_{t} - \\eta\\hat{\\nabla}\\_{w}{L(w\\_{t})} $$\r\n\r\nWhere $\\eta$ is a learning rate. SGD reduces redundancy compared to batch gradient descent - which recomputes gradients for similar examples before each parameter update - so it is usually much faster.\r\n\r\n(Image Source: [here](http://rasbt.github.io/mlxtend/user_guide/general_concepts/gradient-optimization/))","description_state":"present","introduced_year":1951,"introduced_by":{"title":null,"paper":null,"first_author":null,"n_authors":0,"url_abs":null,"archive_paper_url":null},"source":{"url":null,"title":null,"url_on_a_paper_host":false},"code_snippet_url":"https://github.com/pytorch/pytorch/blob/4e0ac120e9a8b096069c2f892488d630a5c8f358/torch/optim/sgd.py#L97-L112","code_snippet_url_on_a_code_host":true,"categories":[{"area":"General","area_id":"general","collection":"Stochastic Optimization","url":"/methods/category/stochastic-optimization","pwc_aliases":[]}],"n_papers_tagged":2021,"archive_num_papers":2021,"papers_newest_first":[{"paper":null,"slug":"fast-last-iterate-convergence-of-sgd-in-the","title":"Fast Last-Iterate Convergence of SGD in the Smooth Interpolation Regime","date":"2025-07-15","arxiv_id":"2507.11274","n_code_links":0,"syntology":null},{"paper":null,"slug":"a-single-merging-suffices-recovering-server","title":"A Single Merging Suffices: Recovering Server-based Learning Performance in Decentralized Learning","date":"2025-07-09","arxiv_id":"2507.06542","n_code_links":0,"syntology":null},{"paper":null,"slug":"tight-generalization-error-bounds-for","title":"Tight Generalization Error Bounds for Stochastic Gradient Descent in Non-convex Learning","date":"2025-06-23","arxiv_id":"2506.18645","n_code_links":0,"syntology":null},{"paper":"/paper/a-minimalist-optimizer-design-for-llm","slug":"a-minimalist-optimizer-design-for-llm","title":"A Minimalist Optimizer Design for LLM Pretraining","date":"2025-06-20","arxiv_id":"2506.16659","n_code_links":1,"syntology":{"ran":0,"of":1,"n_ran_checked":0,"n_instrument":0,"unverified":1,"pointer_only":1,"phrase":"0 ran · 1 unverified","official":{"repos":["optimai-lab/minimalist_llm_pretraining"],"state":"official: harvested, nothing ran","n_ran":0,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":1,"ran_from_kinds":[]}}},{"paper":null,"slug":"a-simplified-analysis-of-sgd-for-linear","title":"A Simplified Analysis of SGD for Linear Regression with Weight Averaging","date":"2025-06-18","arxiv_id":"2506.15535","n_code_links":0,"syntology":null},{"paper":"/paper/sharpness-aware-machine-unlearning","slug":"sharpness-aware-machine-unlearning","title":"Sharpness-Aware Machine Unlearning","date":"2025-06-16","arxiv_id":"2506.13715","n_code_links":0,"syntology":{"ran":1,"of":1,"n_ran_checked":1,"n_instrument":0,"unverified":0,"pointer_only":0,"phrase":"1 ran (of which 1 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified; the one sample that ran constructed an object rather than computing a result","official":null}},{"paper":null,"slug":"is-your-batch-size-the-problem-revisiting-the","title":"Is your batch size the problem? Revisiting the Adam-SGD gap in language modeling","date":"2025-06-14","arxiv_id":"2506.12543","n_code_links":0,"syntology":null},{"paper":null,"slug":"learning-single-index-models-via-harmonic","title":"Learning single-index models via harmonic decomposition","date":"2025-06-11","arxiv_id":"2506.09887","n_code_links":0,"syntology":null},{"paper":null,"slug":"2506-08415","title":"Improved Scaling Laws in Linear Regression via Data Reuse","date":"2025-06-10","arxiv_id":"2506.08415","n_code_links":0,"syntology":null},{"paper":null,"slug":"2506-08419","title":"Online Learning-guided Learning Rate Adaptation via Gradient Alignment","date":"2025-06-10","arxiv_id":"2506.08419","n_code_links":0,"syntology":null},{"paper":"/paper/an-adaptive-method-stabilizing-activations","slug":"an-adaptive-method-stabilizing-activations","title":"An Adaptive Method Stabilizing Activations for Enhanced Generalization","date":"2025-06-10","arxiv_id":"2506.08353","n_code_links":1,"syntology":null},{"paper":"/paper/2506-04430","slug":"2506-04430","title":"Leveraging Coordinate Momentum in SignSGD and Muon: Memory-Optimized Zero-Order","date":"2025-06-04","arxiv_id":"2506.04430","n_code_links":1,"syntology":null},{"paper":null,"slug":"classifying-dental-care-providers-through","title":"Classifying Dental Care Providers Through Machine Learning with Features Ranking","date":"2025-06-04","arxiv_id":"2506.04474","n_code_links":0,"syntology":null},{"paper":null,"slug":"incremental-gradient-descent-with-small-epoch","title":"Incremental Gradient Descent with Small Epoch Counts is Surprisingly Slow on Ill-Conditioned Problems","date":"2025-06-04","arxiv_id":"2506.04126","n_code_links":0,"syntology":null},{"paper":null,"slug":"orthogonal-gradient-descent-improves-neural","title":"Orthogonal Gradient Descent Improves Neural Calibration","date":"2025-06-04","arxiv_id":"2506.04487","n_code_links":0,"syntology":null},{"paper":null,"slug":"replay-can-provably-increase-forgetting","title":"Replay Can Provably Increase Forgetting","date":"2025-06-04","arxiv_id":"2506.04377","n_code_links":0,"syntology":null},{"paper":null,"slug":"asymptotics-of-sgd-in-sequence-single-index","title":"Asymptotics of SGD in Sequence-Single Index Models and Single-Layer Attention Networks","date":"2025-06-03","arxiv_id":"2506.02651","n_code_links":0,"syntology":null},{"paper":null,"slug":"lightsam-parameter-agnostic-sharpness-aware","title":"LightSAM: Parameter-Agnostic Sharpness-Aware Minimization","date":"2025-05-30","arxiv_id":"2505.24399","n_code_links":0,"syntology":null},{"paper":"/paper/sgd-as-free-energy-minimization-a","slug":"sgd-as-free-energy-minimization-a","title":"SGD as Free Energy Minimization: A Thermodynamic View on Neural Network Training","date":"2025-05-29","arxiv_id":"2505.23489","n_code_links":1,"syntology":null},{"paper":null,"slug":"the-rich-and-the-simple-on-the-implicit-bias","title":"The Rich and the Simple: On the Implicit Bias of Adam and SGD","date":"2025-05-29","arxiv_id":"2505.24022","n_code_links":0,"syntology":null},{"paper":null,"slug":"des-loc-desynced-low-communication-adaptive","title":"DES-LOC: Desynced Low Communication Adaptive Optimizers for Training Foundation Models","date":"2025-05-28","arxiv_id":"2505.22549","n_code_links":0,"syntology":null},{"paper":null,"slug":"padam-parallel-averaged-adam-reduces-the","title":"PADAM: Parallel averaged Adam reduces the error for stochastic optimization in scientific machine learning","date":"2025-05-28","arxiv_id":"2505.22085","n_code_links":0,"syntology":null},{"paper":null,"slug":"autosgd-automatic-learning-rate-selection-for","title":"AutoSGD: Automatic Learning Rate Selection for Stochastic Gradient Descent","date":"2025-05-27","arxiv_id":"2505.21651","n_code_links":0,"syntology":null},{"paper":null,"slug":"convergence-of-clipped-sgd-for-convex-l-0-l-1","title":"Convergence of Clipped-SGD for Convex $(L_0,L_1)$-Smooth Optimization with Heavy-Tailed Noise","date":"2025-05-27","arxiv_id":"2505.20817","n_code_links":0,"syntology":null},{"paper":null,"slug":"decaf-decentralized-consensus-and","title":"DeCAF: Decentralized Consensus-And-Factorization for Low-Rank Adaptation of Foundation Models","date":"2025-05-27","arxiv_id":"2505.21382","n_code_links":0,"syntology":null},{"paper":null,"slug":"gradient-flow-matching-for-learning-update","title":"Gradient Flow Matching for Learning Update Dynamics in Neural Network Training","date":"2025-05-26","arxiv_id":"2505.20221","n_code_links":0,"syntology":null},{"paper":null,"slug":"convergence-sticking-and-escape-stochastic","title":"Convergence, Sticking and Escape: Stochastic Dynamics Near Critical Points in SGD","date":"2025-05-24","arxiv_id":"2505.18535","n_code_links":0,"syntology":null},{"paper":null,"slug":"accelerating-learned-image-compression","title":"Accelerating Learned Image Compression Through Modeling Neural Training Dynamics","date":"2025-05-23","arxiv_id":"2505.18107","n_code_links":0,"syntology":null},{"paper":"/paper/decomposition-of-water-demand-patterns-using","slug":"decomposition-of-water-demand-patterns-using","title":"Decomposition of Water Demand Patterns Using Skewed Gaussian Distributions for Behavioral Insights and Operational Planning","date":"2025-05-23","arxiv_id":"2505.18245","n_code_links":1,"syntology":null},{"paper":null,"slug":"emergence-of-hebbian-dynamics-in-regularized","title":"Emergence of Hebbian Dynamics in Regularized Non-Local Learners","date":"2025-05-23","arxiv_id":"2505.18069","n_code_links":0,"syntology":null},{"paper":"/paper/adams-momentum-itself-can-be-a-normalizer-for","slug":"adams-momentum-itself-can-be-a-normalizer-for","title":"AdamS: Momentum Itself Can Be A Normalizer for LLM Pretraining and Post-training","date":"2025-05-22","arxiv_id":"2505.16363","n_code_links":1,"syntology":{"ran":1,"of":1,"n_ran_checked":1,"n_instrument":0,"unverified":0,"pointer_only":0,"phrase":"1 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified","official":{"repos":["pku-huzhang/AdamS"],"state":"official: no sample here; runs from other or unrecorded repositories","n_ran":0,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":0,"ran_from_kinds":["found_in_text"]}}},{"paper":null,"slug":"dimension-adapted-momentum-outscales-sgd","title":"Dimension-adapted Momentum Outscales SGD","date":"2025-05-22","arxiv_id":"2505.16098","n_code_links":0,"syntology":null},{"paper":null,"slug":"smoothed-sgd-for-quantiles-bahadur","title":"Smoothed SGD for quantiles: Bahadur representation and Gaussian approximation","date":"2025-05-19","arxiv_id":"2505.13299","n_code_links":0,"syntology":null},{"paper":null,"slug":"accelerating-neural-network-training-along","title":"Accelerating Neural Network Training Along Sharp and Flat Directions","date":"2025-05-17","arxiv_id":"2505.11972","n_code_links":0,"syntology":null},{"paper":null,"slug":"on-the-o-frac-sqrt-d-k-1-4-convergence-rate","title":"On the $O(\\frac{\\sqrt{d}}{K^{1/4}})$ Convergence Rate of AdamW Measured by $\\ell_1$ Norm","date":"2025-05-17","arxiv_id":"2505.11840","n_code_links":0,"syntology":null},{"paper":null,"slug":"2505-11343","title":"Revisiting Stochastic Approximation and Stochastic Gradient Descent","date":"2025-05-16","arxiv_id":"2505.11343","n_code_links":0,"syntology":null},{"paper":null,"slug":"2505-11434","title":"Controlling the Flow: Stability and Convergence for Stochastic Gradient Descent with Decaying Regularization","date":"2025-05-16","arxiv_id":"2505.11434","n_code_links":0,"syntology":null},{"paper":null,"slug":"deepseqcoco-a-robust-mobile-friendly-deep","title":"DeepSeqCoco: A Robust Mobile Friendly Deep Learning Model for Detection of Diseases in Cocos nucifera","date":"2025-05-15","arxiv_id":"2505.10030","n_code_links":0,"syntology":null},{"paper":null,"slug":"birch-sgd-a-tree-graph-framework-for-local","title":"Birch SGD: A Tree Graph Framework for Local and Asynchronous SGD Methods","date":"2025-05-14","arxiv_id":"2505.09218","n_code_links":0,"syntology":null},{"paper":"/paper/differentiable-channel-selection-in-self","slug":"differentiable-channel-selection-in-self","title":"Differentiable Channel Selection in Self-Attention For Person Re-Identification","date":"2025-05-13","arxiv_id":"2505.08961","n_code_links":1,"syntology":null},{"paper":null,"slug":"rapid-overfitting-of-multi-pass-stochastic","title":"Rapid Overfitting of Multi-Pass Stochastic Gradient Descent in Stochastic Convex Optimization","date":"2025-05-13","arxiv_id":"2505.08306","n_code_links":0,"syntology":null},{"paper":null,"slug":"sharp-gaussian-approximations-for","title":"Sharp Gaussian approximations for Decentralized Federated Learning","date":"2025-05-12","arxiv_id":"2505.08125","n_code_links":0,"syntology":null},{"paper":null,"slug":"streaming-krylov-accelerated-stochastic","title":"Streaming Krylov-Accelerated Stochastic Gradient Descent","date":"2025-05-11","arxiv_id":"2505.07046","n_code_links":0,"syntology":null},{"paper":null,"slug":"complexity-lower-bounds-of-adaptive-gradient","title":"Complexity Lower Bounds of Adaptive Gradient Algorithms for Non-convex Stochastic Optimization under Relaxed Smoothness","date":"2025-05-07","arxiv_id":"2505.04599","n_code_links":0,"syntology":null},{"paper":null,"slug":"more-optimal-fractional-order-stochastic","title":"More Optimal Fractional-Order Stochastic Gradient Descent for Non-Convex Optimization Problems","date":"2025-05-05","arxiv_id":"2505.02985","n_code_links":0,"syntology":null},{"paper":null,"slug":"memory-efficient-llm-training-by-various","title":"Memory-Efficient LLM Training by Various-Grained Low-Rank Projection of Gradients","date":"2025-05-03","arxiv_id":"2505.01744","n_code_links":0,"syntology":null},{"paper":null,"slug":"emergence-and-scaling-laws-in-sgd-learning-of","title":"Emergence and scaling laws in SGD learning of shallow neural networks","date":"2025-04-28","arxiv_id":"2504.19983","n_code_links":0,"syntology":null},{"paper":null,"slug":"theoretical-framework-for-tempered-fractional","title":"Theoretical Framework for Tempered Fractional Gradient Descent: Application to Breast Cancer Classification","date":"2025-04-26","arxiv_id":"2504.18849","n_code_links":0,"syntology":null},{"paper":"/paper/towards-practical-second-order-optimizers-in","slug":"towards-practical-second-order-optimizers-in","title":"Towards Practical Second-Order Optimizers in Deep Learning: Insights from Fisher Information Analysis","date":"2025-04-26","arxiv_id":"2504.20096","n_code_links":1,"syntology":null},{"paper":null,"slug":"learning-operators-by-regularized-stochastic","title":"Learning Operators by Regularized Stochastic Gradient Descent with Operator-valued Kernels","date":"2025-04-25","arxiv_id":"2504.18184","n_code_links":0,"syntology":null},{"paper":null,"slug":"some-optimizers-are-more-equal-understanding","title":"Some Optimizers are More Equal: Understanding the Role of Optimizers in Group Fairness","date":"2025-04-21","arxiv_id":"2504.14882","n_code_links":0,"syntology":null},{"paper":null,"slug":"connecting-parameter-magnitudes-and-hessian","title":"Connecting Parameter Magnitudes and Hessian Eigenspaces at Scale using Sketched Methods","date":"2025-04-20","arxiv_id":"2504.14701","n_code_links":0,"syntology":null},{"paper":null,"slug":"first-and-second-order-approximations-to","title":"First and Second Order Approximations to Stochastic Gradient Descent Methods with Momentum Terms","date":"2025-04-18","arxiv_id":"2504.13992","n_code_links":0,"syntology":null},{"paper":null,"slug":"stochastic-gradient-descent-in-non-convex","title":"Stochastic Gradient Descent in Non-Convex Problems: Asymptotic Convergence with Relaxed Step-Size via Stopping Time Methods","date":"2025-04-17","arxiv_id":"2504.12601","n_code_links":0,"syntology":null},{"paper":"/paper/corner-gradient-descent","slug":"corner-gradient-descent","title":"Corner Gradient Descent","date":"2025-04-16","arxiv_id":"2504.12519","n_code_links":1,"syntology":null},{"paper":null,"slug":"a-piecewise-lyapunov-analysis-of-sub","title":"A Piecewise Lyapunov Analysis of Sub-quadratic SGD: Applications to Robust and Quantile Regression","date":"2025-04-11","arxiv_id":"2504.08178","n_code_links":0,"syntology":null},{"paper":null,"slug":"better-rates-for-random-task-orderings-in","title":"Better Rates for Random Task Orderings in Continual Linear Models","date":"2025-04-06","arxiv_id":"2504.04579","n_code_links":0,"syntology":null},{"paper":null,"slug":"architect-your-landscape-approach-ayla-for","title":"Architect Your Landscape Approach (AYLA) for Optimizations in Deep Learning","date":"2025-04-02","arxiv_id":"2504.01875","n_code_links":0,"syntology":null},{"paper":null,"slug":"feature-learning-from-non-gaussian-inputs-the","title":"Feature learning from non-Gaussian inputs: the case of Independent Component Analysis in high dimensions","date":"2025-03-31","arxiv_id":"2503.23896","n_code_links":0,"syntology":null},{"paper":null,"slug":"learning-a-single-index-model-from","title":"Learning a Single Index Model from Anisotropic Data with vanilla Stochastic Gradient Descent","date":"2025-03-31","arxiv_id":"2503.23642","n_code_links":0,"syntology":null},{"paper":null,"slug":"nested-stochastic-gradient-descent-for","title":"Nested Stochastic Gradient Descent for (Generalized) Sinkhorn Distance-Regularized Distributionally Robust Optimization","date":"2025-03-29","arxiv_id":"2503.22923","n_code_links":0,"syntology":null},{"paper":null,"slug":"almost-bayesian-the-fractal-dynamics-of","title":"Almost Bayesian: The Fractal Dynamics of Stochastic Gradient Descent","date":"2025-03-28","arxiv_id":"2503.22478","n_code_links":0,"syntology":null},{"paper":null,"slug":"streaming-federated-learning-with-markovian","title":"Streaming Federated Learning with Markovian Data","date":"2025-03-24","arxiv_id":"2503.18807","n_code_links":0,"syntology":null},{"paper":null,"slug":"finding-stable-subnetworks-at-initialization","title":"Finding Stable Subnetworks at Initialization with Dataset Distillation","date":"2025-03-23","arxiv_id":"2503.17905","n_code_links":0,"syntology":null},{"paper":null,"slug":"the-global-convergence-time-of-stochastic","title":"The global convergence time of stochastic gradient descent in non-convex landscapes: Sharp estimates via large deviations","date":"2025-03-20","arxiv_id":"2503.16398","n_code_links":0,"syntology":null},{"paper":null,"slug":"layer-wise-adaptive-gradient-norm-penalizing","title":"Layer-wise Adaptive Gradient Norm Penalizing Method for Efficient and Accurate Deep Learning","date":"2025-03-18","arxiv_id":"2503.14205","n_code_links":0,"syntology":null},{"paper":null,"slug":"effective-dimension-aware-fractional-order","title":"Effective Dimension Aware Fractional-Order Stochastic Gradient Descent for Convex Optimization Problems","date":"2025-03-17","arxiv_id":"2503.13764","n_code_links":0,"syntology":null},{"paper":null,"slug":"convergence-analysis-of-alpha-svrg-under","title":"Convergence Analysis of alpha-SVRG under Strong Convexity","date":"2025-03-16","arxiv_id":"2503.12454","n_code_links":0,"syntology":null},{"paper":null,"slug":"understanding-gradient-orthogonalization-for","title":"Understanding Gradient Orthogonalization for Deep Learning via Non-Euclidean Trust-Region Optimization","date":"2025-03-16","arxiv_id":"2503.12645","n_code_links":0,"syntology":null},{"paper":"/paper/zo2-scalable-zeroth-order-fine-tuning-for","slug":"zo2-scalable-zeroth-order-fine-tuning-for","title":"ZO2: Scalable Zeroth-Order Fine-Tuning for Extremely Large Language Models with Limited GPU Memory","date":"2025-03-16","arxiv_id":"2503.12668","n_code_links":1,"syntology":null},{"paper":null,"slug":"asynchronous-sharpness-aware-minimization-for","title":"Asynchronous Sharpness-Aware Minimization For Fast and Accurate Deep Learning","date":"2025-03-14","arxiv_id":"2503.11147","n_code_links":0,"syntology":null},{"paper":null,"slug":"global-convergence-and-rich-feature-learning","title":"Global Convergence and Rich Feature Learning in $L$-Layer Infinite-Width Neural Networks under $μ$P Parametrization","date":"2025-03-12","arxiv_id":"2503.09565","n_code_links":0,"syntology":null},{"paper":null,"slug":"water-quality-data-imputation-via-a-fast","title":"Water Quality Data Imputation via A Fast Latent Factorization of Tensors with PID-based Optimizer","date":"2025-03-10","arxiv_id":"2503.06997","n_code_links":0,"syntology":null},{"paper":null,"slug":"analyzing-the-role-of-permutation-invariance","title":"Analyzing the Role of Permutation Invariance in Linear Mode Connectivity","date":"2025-03-08","arxiv_id":"2503.06001","n_code_links":0,"syntology":null},{"paper":null,"slug":"efficiently-escaping-saddle-points-under","title":"Efficiently Escaping Saddle Points under Generalized Smoothness via Self-Bounding Regularity","date":"2025-03-06","arxiv_id":"2503.04712","n_code_links":0,"syntology":null},{"paper":null,"slug":"fine-tuning-florence2-for-enhanced-object","title":"Fine-Tuning Florence2 for Enhanced Object Detection in Un-constructed Environments: Vision-Language Model Approach","date":"2025-03-06","arxiv_id":"2503.04918","n_code_links":0,"syntology":null},{"paper":"/paper/frankenstein-optimizer-harnessing-the","slug":"frankenstein-optimizer-harnessing-the","title":"Frankenstein Optimizer: Harnessing the Potential by Revisiting Optimization Tricks","date":"2025-03-04","arxiv_id":"2503.02147","n_code_links":1,"syntology":null},{"paper":null,"slug":"non-convergence-to-the-optimal-risk-for-adam","title":"Non-convergence to the optimal risk for Adam and stochastic gradient descent optimization in the training of deep neural networks","date":"2025-03-03","arxiv_id":"2503.01660","n_code_links":0,"syntology":null},{"paper":null,"slug":"on-the-byzantine-fault-tolerance-of-signsgd","title":"On the Byzantine Fault Tolerance of signSGD with Majority Vote","date":"2025-02-26","arxiv_id":"2502.19170","n_code_links":0,"syntology":null},{"paper":null,"slug":"an-improved-privacy-and-utility-analysis-of","title":"An Improved Privacy and Utility Analysis of Differentially Private SGD with Bounded Domain and Smooth Losses","date":"2025-02-25","arxiv_id":"2502.17772","n_code_links":0,"syntology":null},{"paper":"/paper/sassha-sharpness-aware-adaptive-second-order","slug":"sassha-sharpness-aware-adaptive-second-order","title":"SASSHA: Sharpness-aware Adaptive Second-order Optimization with Stable Hessian Approximation","date":"2025-02-25","arxiv_id":"2502.18153","n_code_links":1,"syntology":{"ran":3,"of":3,"n_ran_checked":2,"n_instrument":1,"unverified":0,"pointer_only":3,"phrase":"3 ran (of which 1 constructed an object rather than computing a result; 2 with no instrument failure: 1 honoured, 0 violated, 1 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified","official":{"repos":["LOG-postech/Sassha"],"state":"official (archive's flag): 3 ran","n_ran":3,"n_constructed":1,"n_ran_no_instrument_failure":2,"n_unverified":0,"ran_from_kinds":["official"]}}},{"paper":null,"slug":"unbiased-and-sign-compression-in-distributed","title":"Unbiased and Sign Compression in Distributed Learning: Comparing Noise Resilience via SDEs","date":"2025-02-24","arxiv_id":"2502.17009","n_code_links":0,"syntology":null},{"paper":"/paper/a-stronger-mixture-of-low-rank-experts-for","slug":"a-stronger-mixture-of-low-rank-experts-for","title":"A Stronger Mixture of Low-Rank Experts for Fine-Tuning Foundation Models","date":"2025-02-20","arxiv_id":"2502.15828","n_code_links":1,"syntology":{"ran":3,"of":4,"n_ran_checked":0,"n_instrument":3,"unverified":1,"pointer_only":4,"phrase":"3 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 1 unverified","official":{"repos":["thudm/moelora_riemannian"],"state":"official (archive's flag): 3 ran","n_ran":3,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":1,"ran_from_kinds":["official"]}}},{"paper":null,"slug":"variance-reduction-methods-do-not-need-to","title":"Variance Reduction Methods Do Not Need to Compute Full Gradients: Improved Efficiency through Shuffling","date":"2025-02-20","arxiv_id":"2502.14648","n_code_links":0,"syntology":null},{"paper":null,"slug":"understanding-sgd-with-exponential-moving","title":"Understanding SGD with Exponential Moving Average: A Case Study in Linear Regression","date":"2025-02-19","arxiv_id":"2502.14123","n_code_links":0,"syntology":null},{"paper":null,"slug":"improving-multi-turn-task-completion-in-task","title":"Improving Multi-turn Task Completion in Task-Oriented Dialog Systems via Prompt Chaining and Fine-Grained Feedback","date":"2025-02-18","arxiv_id":"2502.13298","n_code_links":0,"syntology":null},{"paper":null,"slug":"liveval-time-aware-data-valuation-via","title":"LiveVal: Time-aware Data Valuation via Adaptive Reference Points","date":"2025-02-14","arxiv_id":"2502.10489","n_code_links":0,"syntology":null},{"paper":null,"slug":"sewa-selective-weight-average-via","title":"SeWA: Selective Weight Average via Probabilistic Masking","date":"2025-02-14","arxiv_id":"2502.10119","n_code_links":0,"syntology":null},{"paper":null,"slug":"linear-time-user-level-dp-sco-via-robust","title":"Linear-Time User-Level DP-SCO via Robust Statistics","date":"2025-02-13","arxiv_id":"2502.08889","n_code_links":0,"syntology":null},{"paper":null,"slug":"optimal-algorithms-in-linear-regression-under","title":"Optimal Algorithms in Linear Regression under Covariate Shift: On the Importance of Precondition","date":"2025-02-13","arxiv_id":"2502.09047","n_code_links":0,"syntology":null},{"paper":null,"slug":"scaling-law-for-stochastic-gradient-descent","title":"Scaling Law for Stochastic Gradient Descent in Quadratically Parameterized Linear Regression","date":"2025-02-13","arxiv_id":"2502.09106","n_code_links":0,"syntology":null},{"paper":null,"slug":"towards-efficient-optimizer-design-for-llm","title":"Towards Efficient Optimizer Design for LLM via Structured Fisher Approximation with a Low-Rank Extension","date":"2025-02-11","arxiv_id":"2502.07752","n_code_links":0,"syntology":null},{"paper":null,"slug":"analog-in-memory-training-on-general-non","title":"Analog In-memory Training on General Non-ideal Resistive Elements: The Impact of Response Functions","date":"2025-02-10","arxiv_id":"2502.06309","n_code_links":0,"syntology":null},{"paper":null,"slug":"gaussian-approximation-and-multiplier-1","title":"Gaussian Approximation and Multiplier Bootstrap for Stochastic Gradient Descent","date":"2025-02-10","arxiv_id":"2502.06719","n_code_links":0,"syntology":null},{"paper":null,"slug":"satellite-observations-guided-diffusion-model","title":"Satellite Observations Guided Diffusion Model for Accurate Meteorological States at Arbitrary Resolution","date":"2025-02-09","arxiv_id":"2502.07814","n_code_links":0,"syntology":null},{"paper":null,"slug":"the-late-stage-training-dynamics-of","title":"The late-stage training dynamics of (stochastic) subgradient descent on homogeneous neural networks","date":"2025-02-08","arxiv_id":"2502.05668","n_code_links":0,"syntology":null},{"paper":null,"slug":"data-parallel-neural-network-training-via","title":"Data-Parallel Neural Network Training via Nonlinearly Preconditioned Trust-Region Method","date":"2025-02-07","arxiv_id":"2502.05133","n_code_links":0,"syntology":null},{"paper":null,"slug":"bayesian-parameter-shift-rule-in-variational","title":"Bayesian Parameter Shift Rule in Variational Quantum Eigensolvers","date":"2025-02-04","arxiv_id":"2502.02625","n_code_links":0,"syntology":null},{"paper":null,"slug":"on-the-concurrence-of-layer-wise","title":"On The Concurrence of Layer-wise Preconditioning Methods and Provable Feature Learning","date":"2025-02-03","arxiv_id":"2502.01763","n_code_links":0,"syntology":null},{"paper":null,"slug":"training-in-reverse-how-iteration-order","title":"Training in reverse: How iteration order influences convergence and stability in deep learning","date":"2025-02-03","arxiv_id":"2502.01557","n_code_links":0,"syntology":null}],"papers_shown":100,"papers_order":"date (newest first), then slug","paper_counts":{"archive_papers_tagged":2021,"with_a_code_link":591,"where_syntology_ran_a_sample":192,"not_listed_spam_title":0,"listed":2021,"listed_where_code_ran":192,"where_syntology_ran_a_sample_split":{"with_a_run_with_no_instrument_failure":160,"every_run_a_failure_of_syntologys_instrument":32,"listed_with_a_run_with_no_instrument_failure":160,"listed_every_run_a_failure_of_syntologys_instrument":32,"filter":{"states":["a run with no instrument failure","any run, instrument failures included"],"default":"a run with no instrument failure","note":"on the 'only where code ran' pages the default hides, in the browser, the rows where every run was a failure of Syntology's instrument; the second state shows them again. Rows are hidden, never re-ordered; these twins list every row"}},"definition":"distinct papers the archive tags; 'where Syntology ran a sample' counts papers with at least one harvested sample that ran, which is not a correctness claim"},"papers_where_code_ran":192,"papers_where_code_ran_first_page":"/method/sgd/papers/ran/1","papers_where_code_ran_caption":"We ran code from the paper's repository; we did not isolate this method inside it.","pages":[{"order":"archive","page":2,"of":21,"url":"/method/sgd/papers/2","json":"/method/sgd/papers/2.json","rows":100},{"order":"archive","page":3,"of":21,"url":"/method/sgd/papers/3","json":"/method/sgd/papers/3.json","rows":100},{"order":"archive","page":4,"of":21,"url":"/method/sgd/papers/4","json":"/method/sgd/papers/4.json","rows":100},{"order":"archive","page":5,"of":21,"url":"/method/sgd/papers/5","json":"/method/sgd/papers/5.json","rows":100},{"order":"archive","page":6,"of":21,"url":"/method/sgd/papers/6","json":"/method/sgd/papers/6.json","rows":100},{"order":"archive","page":7,"of":21,"url":"/method/sgd/papers/7","json":"/method/sgd/papers/7.json","rows":100},{"order":"archive","page":8,"of":21,"url":"/method/sgd/papers/8","json":"/method/sgd/papers/8.json","rows":100},{"order":"archive","page":9,"of":21,"url":"/method/sgd/papers/9","json":"/method/sgd/papers/9.json","rows":100},{"order":"archive","page":10,"of":21,"url":"/method/sgd/papers/10","json":"/method/sgd/papers/10.json","rows":100},{"order":"archive","page":11,"of":21,"url":"/method/sgd/papers/11","json":"/method/sgd/papers/11.json","rows":100},{"order":"archive","page":12,"of":21,"url":"/method/sgd/papers/12","json":"/method/sgd/papers/12.json","rows":100},{"order":"archive","page":13,"of":21,"url":"/method/sgd/papers/13","json":"/method/sgd/papers/13.json","rows":100},{"order":"archive","page":14,"of":21,"url":"/method/sgd/papers/14","json":"/method/sgd/papers/14.json","rows":100},{"order":"archive","page":15,"of":21,"url":"/method/sgd/papers/15","json":"/method/sgd/papers/15.json","rows":100},{"order":"archive","page":16,"of":21,"url":"/method/sgd/papers/16","json":"/method/sgd/papers/16.json","rows":100},{"order":"archive","page":17,"of":21,"url":"/method/sgd/papers/17","json":"/method/sgd/papers/17.json","rows":100},{"order":"archive","page":18,"of":21,"url":"/method/sgd/papers/18","json":"/method/sgd/papers/18.json","rows":100},{"order":"archive","page":19,"of":21,"url":"/method/sgd/papers/19","json":"/method/sgd/papers/19.json","rows":100},{"order":"archive","page":20,"of":21,"url":"/method/sgd/papers/20","json":"/method/sgd/papers/20.json","rows":100},{"order":"archive","page":21,"of":21,"url":"/method/sgd/papers/21","json":"/method/sgd/papers/21.json","rows":21},{"order":"ran","page":1,"of":2,"url":"/method/sgd/papers/ran/1","json":"/method/sgd/papers/ran/1.json","rows":100},{"order":"ran","page":2,"of":2,"url":"/method/sgd/papers/ran/2","json":"/method/sgd/papers/ran/2.json","rows":92}],"tasks":[{"task":"/task/stochastic-optimization","name":"Stochastic Optimization","papers":149},{"task":"/task/image-classification","name":"Image Classification","papers":122},{"task":"/task/federated-learning","name":"Federated Learning","papers":110},{"task":"/task/image-classification","name":"image-classification","papers":102},{"task":"/task/machine-learning","name":"BIG-bench Machine Learning","papers":66},{"task":"/task/deep-learning","name":"Deep Learning","papers":66},{"task":"/task/regression-1","name":"regression","papers":59},{"task":"/task/quantization","name":"Quantization","papers":55},{"task":"/task/language-modelling","name":"Language Modelling","papers":52},{"task":null,"name":"GPU","papers":49},{"task":"/task/generalization-bounds","name":"Generalization Bounds","papers":45},{"task":"/task/language-modeling","name":"Language Modeling","papers":44},{"task":"/task/distributed-optimization","name":"Distributed Optimization","papers":39},{"task":"/task/classification","name":"General Classification","papers":39},{"task":"/task/computational-efficiency","name":"Computational Efficiency","papers":27},{"task":"/task/data-augmentation","name":"Data Augmentation","papers":27},{"task":"/task/second-order-methods","name":"Second-order methods","papers":25},{"task":"/task/privacy-preserving","name":"Privacy Preserving","papers":24},{"task":"/task/learning-theory","name":"Learning Theory","papers":23},{"task":"/task/speech-recognition","name":"Speech Recognition","papers":23}],"tasks_shown":20,"n_tasks":362,"usage_by_year":[{"year":"2011","papers":3},{"year":"2012","papers":2},{"year":"2013","papers":10},{"year":"2014","papers":10},{"year":"2015","papers":25},{"year":"2016","papers":41},{"year":"2017","papers":75},{"year":"2018","papers":152},{"year":"2019","papers":218},{"year":"2020","papers":291},{"year":"2021","papers":337},{"year":"2022","papers":226},{"year":"2023","papers":270},{"year":"2024","papers":244},{"year":"2025","papers":116}],"row_source":"methods_table","archive":{"source":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","archive_url":"https://paperswithcode.com/method/sgd"},"syntology_read_at":"2026-10-05T00:36:11+00:00","syntology":{"source":"Syntology's have check for this name (the free coverage lookup), read from Syntology's graph","have_url":"https://syntology.ai/api/graph/have?x=SGD","read_at":"2026-10-05T01:01:53+00:00","query":"SGD","kind":"paper","have":"unknown_name","tier":null,"verified_implementation":null,"level":null,"level_meaning":"the served verification level, per answer; never summed across methods","resolved_to":null,"resolution":null,"resolved_name_differs":false,"absence_asserted":false,"absence_unchecked":null,"no_code_is_about_this_method":false,"subject":null,"archive_paper_arxiv_id":null,"repositories":null,"retrieve_with":null,"name_competition":{"verdict":"unresolved_bearers","may_be_wrong_sense":true,"bearers_found":5,"served":null,"leading_other_bearer":{"arxiv_id":"1901.09401","title":"SGD: General Analysis and Improved Rates","citations":488,"named_by":["title_prefix"],"bearer_name":"SGD: General Analysis and Improved Rates","ask_with":"have('1901.09401')"},"n_other_bearers_listed":5,"note":"Nothing resolved to 'SGD' by name, but 5 thing(s) here DECLARE that name -- most cited: the paper 'SGD: General Analysis and Improved Rates' (arXiv:1901.09401, 488 citations). That is a RESOLUTION miss, not a coverage gap: the corpus holds the name under a spelling the lookup did not reach. Ask again with an entry of `also_named`."},"name_competition_text":"Nothing resolved to this name, but 5 entries in Syntology's graph carry it; the most cited is the paper “SGD: General Analysis and Improved Rates” (arXiv:1901.09401, 488 citations). The lookup reads this as a miss of its own name matching, not a gap in coverage: the graph holds the name under a spelling the lookup did not reach.","near_matches":["∞-SGD","SGD-KV","INO-SGD","FBi-SGD","badr-sgd","FO-DP-SGD","DP-SGD-RC","SMA-DP-SGD"],"code_nearby":[],"code_nearby_caption":null,"origin_paper":null,"line":"name not resolved (unknown_name). Other entries in Syntology's graph carry this name; the lookup's note below says what that means for this name.","level_legend":null},"syntology_have_state":"loaded","record_sha256":"5d6b81979a4b332cfaf57a0d7f37a35ba2d2126727d6edf9708d96d68a0e3002","record_changed_at":"2026-10-05","record_changed_at_basis":"changed"}