Browse State-of-the-Art › Second-order methods
Second-order methods
52 papers with code · 0 benchmarks · 0 datasets archive 2025-07-28
Use second-order statistics to process data.
Description from the archive archive 2025-07-28.
Benchmarks archive 2025-07-28
No benchmark for this task in the archive.
Libraries
Not in the archive: the export carries no per-task library table, so there is nothing to show at snapshot 2025-07-28.
Datasets archive 2025-07-28
No dataset record in the archive lists this task.
Subtasks archive 2025-07-28
No subtask under this task in the archive's task tree.
Papers archive 2025-07-28
archive papers tagged: 181 · with a code link: 52 · where Syntology ran a sample: 12 (11 with a run with no instrument failure, 1 where every run was a failure of Syntology's instrument) Syntology
Show: all tagged papersonly where code ran (12 of 181 tagged: 11 with a run with no instrument failure, 1 where every run was a failure of Syntology's instrument)
Page 1 of 2: papers 1 to 100 of 181, in archive order: by repositories listed in the archive (most first), then newest first, not by stars (the archive holds no stars, so PwC's “Social” and “Latest” sorts cannot be reproduced). Papers that list no repository come after every paper that lists one.
Papers without a page here are shown as plain text. A Syntology line reads “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the figure “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. When the archive marks a repository official for the paper, the line starts with that repository's state (the archive's flag, not a verdict on who wrote the code); hover it for the repositories the samples that ran came from. The first sentence of the abstract is shown for the first 30 papers; every abstract is on its paper's page.
-
1 Jun 2020 4 repositories listed Syntology official (archive's flag): 4 ran · 5 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 0 honoured, 0 violated, 4 with no contract checked; 1 where Syntology's instrument failed) · 6 unverified (of 11 harvested samples) · 1 pointer-only (licence)We introduce ADAHESSIAN, a second order stochastic optimization algorithm which dynamically incorporates the curvature of the loss function via ADAptive estimates of the HESSIAN.
-
12 Feb 2016 4 repositories listedFirst-order stochastic methods are the state-of-the-art in large-scale machine learning optimization owing to efficient per-iteration complexity.
-
5 Feb 2024 2 repositories listed Syntology official (archive's flag): 9 ran · 9 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 3 honoured, 1 violated, 1 with no contract checked; 4 where Syntology's instrument failed) · 1 unverified (of 10 harvested samples) · 9 pointer-only (licence)Adaptive gradient optimizers like Adam(W) are the default training algorithms for many deep learning architectures, such as transformers.
-
9 Dec 2023 2 repositories listed Syntology official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified (of 1 harvested sample) · 1 pointer-only (licence)Second-order methods such as KFAC can be useful for neural net training.
-
12 May 2022 2 repositories listedWe emphasize the relevance of OODD and its specific supervision requirements for the detection of a multimodal, diverse targets class among other similar radar targets and clutter in real-life critical systems.
-
7 Jul 2021 2 repositories listed Syntology official (archive's flag): 4 ran · 7 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 4 honoured, 0 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 7 unverified (of 14 harvested samples)We propose two new algorithms as part of a framework called M-FAC: the first algorithm is tailored towards network compression and can compute the IHVP for dimension d, if the Hessian is given as a sum of m rank-one…
-
3 Jun 2020 2 repositories listedThis enables researchers to further study and improve this promising optimization technique and hopefully reconsider stochastic second-order methods as competitive optimization techniques for training DNNs; we also hope…
-
7 Feb 2020 2 repositories listedIn this work we motivate the extension of Newton methods to the SA regime, and argue for the use of the scalable low rank saddle free Newton (LRSFN) method, which avoids forming the Hessian in favor of making a low rank…
-
15 Jan 2020 2 repositories listedNMC is similar to the Newton-Raphson update in optimization where the second order gradient is used to automatically scale the step size in each dimension.
-
10 Jun 2025 1 repository listedAdaptive gradient methods are computationally efficient and converge quickly, but they often suffer from poor generalization.
-
26 Apr 2025 1 repository listedAdaFisher aims to bridge the gap between the improved convergence and generalization of second-order methods and the computational efficiency needed for training DNNs.
-
25 Feb 2025 1 repository listed Syntology official (archive's flag): 3 ran · 3 ran (of which 1 constructed an object rather than computing a result; 2 with no instrument failure: 1 honoured, 0 violated, 1 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 3 harvested samples) · 3 pointer-only (licence)Approximate second-order optimization methods often exhibit poorer generalization compared to first-order approaches.
-
10 Feb 2025 1 repository listedGiven the ubiquity of streaming data, online algorithms have been widely used for parameter estimation, with second-order methods particularly standing out for their efficiency and robustness.
-
12 Nov 2024 1 repository listed Syntology official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 0 violated, 3 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified (of 3 harvested samples)We exploit this novel setting to study the training and generalization properties of the GN optimizer.
-
24 Jul 2024 1 repository listedThis paper presents a novel hybrid algorithm for minimizing the sum of a continuously differentiable loss function and a nonsmooth, possibly nonconvex, sparse regularization function.
-
5 Jun 2024 1 repository listedSecond-order information is valuable for many applications but challenging to compute.
-
10 Apr 2024 1 repository listedIt modifies the Gauss-Newton method to approximate the min-max Hessian and uses the Sherman-Morrison inversion formula to calculate the inverse.
-
5 Mar 2024 1 repository listedCompared with first-order optimizers, it adopts a certain amount of information from the Hessian matrix to assist optimization, while compared with the existing second-order optimizers, it keeps the good generalization…
-
6 Dec 2023 1 repository listedWhen training large models, such as neural networks, the full derivatives of order 2 and beyond are usually inaccessible, due to their computational cost.
-
30 Oct 2023 1 repository listedWe introduce AdaSub, a stochastic optimization algorithm that computes a search direction based on second-order information in a low-dimensional subspace that is defined adaptively based on available current and past…
-
23 Oct 2023 1 repository listed Syntology official (archive's flag): 7 ran · 7 ran (of which 0 constructed an object rather than computing a result; 7 with no instrument failure: 0 honoured, 0 violated, 7 with no contract checked; 0 where Syntology's instrument failed) · 9 unverified (of 16 harvested samples) · 16 pointer-only (licence)Research into optimisation for deep learning is characterised by a tension between the computational efficiency of first-order, gradient-based methods (such as SGD and Adam) and the theoretical efficiency of…
-
9 Jun 2023 1 repository listed Syntology official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 1 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 1 unverified (of 3 harvested samples)Experiments on deep neural networks show that this approach can compress full-matrix preconditioners to up to 99% sparsity without accuracy loss, effectively removing the memory overhead of full-matrix preconditioners…
-
2 Jun 2023 1 repository listed Syntology official (archive's flag): 9 ran · 9 ran (of which 2 constructed an object rather than computing a result; 6 with no instrument failure: 0 honoured, 0 violated, 6 with no contract checked; 3 where Syntology's instrument failed) · 1 unverified (of 10 harvested samples) · 10 pointer-only (licence)This work proposes a Momentum-Enabled Kronecker-Factor-Based Optimizer Using Rank-1 updates, called MKOR, that improves the training time and convergence properties of deep neural networks (DNNs).
-
26 May 2023 1 repository listedSpecifically, they exhibit a superlinear rate with O(d²) cost in contrast to the linear rate of first-order methods with O(d) cost and the quadratic rate of second-order methods with O(d³) cost.
-
1 May 2023 1 repository listed Syntology official: harvested, nothing ran · 0 ran · 1 unverified (of 1 harvested sample)We present ISAAC (Input-baSed ApproximAte Curvature), a novel method that conditions the gradient using selected second-order information and has an asymptotically vanishing computational overhead, assuming a batch size…
-
11 Apr 2023 1 repository listed Syntology official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 1 harvested sample) · 1 pointer-only (licence)Automatic gradient descent trains both fully-connected and convolutional networks out-of-the-box and at ImageNet scale.
-
16 Feb 2023 1 repository listed Syntology official: harvested, nothing ran · 0 ran · 1 unverified (of 1 harvested sample)Popular machine learning approaches forgo second-order information due to the difficulty of computing curvature in high dimensions.
-
24 Nov 2022 1 repository listedWe find that linear regression is much more effective than logistic regression from both privacy and computational aspects, especially at stricter epsilon values (ϵ< 1).
-
8 Nov 2022 1 repository listedAlso, for the general m-dimensional case, a compact recursive analytical expression is established, and its algorithmic implementation is discussed.
-
6 Jun 2022 1 repository listedYet, comparatively little is known about the benefits of using variance reduction to accelerate popular stochastic second-order methods such as Subsampled Newton.
-
27 May 2022 1 repository listed
-
21 May 2022 1 repository listed Syntology official (archive's flag): 5 ran · 5 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 2 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified (of 5 harvested samples) · 5 pointer-only (licence)
-
19 Feb 2022 1 repository listed
-
24 Jan 2022 1 repository listed
-
29 Sep 2021 1 repository listed Syntology official: harvested, nothing ran · 0 ran · 2 unverified (of 2 harvested samples)
-
13 Sep 2021 1 repository listed
-
11 Jun 2021 1 repository listed
-
5 Jun 2021 1 repository listed Syntology official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 1 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 1 unverified (of 3 harvested samples) · 3 pointer-only (licence)
-
16 Jun 2020 1 repository listed
-
9 Jun 2020 1 repository listed
-
4 Jun 2020 1 repository listed
-
27 Mar 2020 1 repository listed
-
3 Dec 2019 1 repository listed
-
11 Oct 2019 1 repository listed
-
16 Jul 2019 1 repository listed
-
29 May 2019 1 repository listed
-
20 Apr 2019 1 repository listed
-
31 Jan 2019 1 repository listed
-
26 Dec 2018 1 repository listed
-
2 Oct 2018 1 repository listed
-
26 Mar 2018 1 repository listed
-
30 Jun 2017 1 repository listed
-
KOALA++: Efficient Kalman-Based Optimization of Neural Networks with Gradient-Covariance Products4 Jun 2025 0 repositories listed
-
Accelerated Training of Federated Learning via Second-Order Methods29 May 2025 0 repositories listed
-
Representation Meets Optimization: Training PINNs and PIKANs for Gray-Box Discovery in Systems Pharmacology10 Apr 2025 0 repositories listed
-
FedOSAA: Improving Federated Learning with One-Step Anderson Acceleration14 Mar 2025 0 repositories listed
-
A Survey of Optimization Methods for Training DL Models: Theoretical Perspective on Convergence and Generalization24 Jan 2025 0 repositories listed
-
Distributed Quasi-Newton Method for Fair and Fast Federated Learning18 Jan 2025 0 repositories listed
-
Preconditioners for the Stochastic Training of Neural Fields1 Jan 2025 0 repositories listed
-
GP-FL: Model-Based Hessian Estimation for Second-Order Over-the-Air Federated Learning5 Dec 2024 0 repositories listed
-
A Learn-to-Optimize Approach for Coordinate-Wise Step Sizes for Quasi-Newton Methods25 Nov 2024 0 repositories listed
-
Gradient Norm Regularization Second-Order Algorithms for Solving Nonconvex-Strongly Concave Minimax Problems24 Nov 2024 0 repositories listed
-
Don't Be So Positive: Negative Step Sizes in Second-Order Methods18 Nov 2024 0 repositories listed
-
Improving Stochastic Cubic Newton with Momentum25 Oct 2024 0 repositories listed
-
Second-Order Min-Max Optimization with Lazy Hessians12 Oct 2024 0 repositories listed
-
Unlocking FedNL: Self-Contained Compute-Optimized Implementation11 Oct 2024 0 repositories listed
-
Adversarial Vulnerability as a Consequence of On-Manifold Inseparibility9 Oct 2024 0 repositories listed
-
FLeNS: Federated Learning with Enhanced Nesterov-Newton Sketch23 Sep 2024 0 repositories listed
-
Fed-Sophia: A Communication-Efficient Second-Order Federated Learning Algorithm10 Jun 2024 0 repositories listed
-
Adaptive and Optimal Second-order Optimistic Methods for Minimax Optimization4 Jun 2024 0 repositories listed
-
Stochastic Newton Proximal Extragradient Method3 Jun 2024 0 repositories listed
-
Kronecker-Factored Approximate Curvature for Physics-Informed Neural Networks24 May 2024 0 repositories listed
-
Inverse-Free Fast Natural Gradient Descent Method for Deep Learning6 Mar 2024 0 repositories listed
-
Second Order Methods for Bandit Optimization and Control14 Feb 2024 0 repositories listed
-
The Challenges of the Nonlinear Regime for Physics-Informed Neural Networks6 Feb 2024 0 repositories listed
-
On The Temporal Domain of Differential Equation Inspired Graph Neural Networks20 Jan 2024 0 repositories listed
-
Krylov Cubic Regularized Newton: A Subspace Second-Order Method with Dimension-Free Convergence Rate5 Jan 2024 0 repositories listed
-
A Computationally Efficient Sparsified Online Newton Method16 Nov 2023 0 repositories listed
-
Adaptive Optimization Algorithms for Machine Learning16 Nov 2023 0 repositories listed
-
Jorge: Approximate Preconditioning for GPU-efficient Second-order Optimization18 Oct 2023 0 repositories listed
-
Stochastic Optimization for Non-convex Problem with Inexact Hessian Matrix, Gradient, and Function18 Oct 2023 0 repositories listed
-
A Homogenization Approach for Gradient-Dominated Stochastic Optimization21 Aug 2023 0 repositories listed
-
Decentralized Riemannian Conjugate Gradient Method on the Stiefel Manifold21 Aug 2023 0 repositories listed
-
Convolutions and More as Einsum: A Tensor Network Perspective with Advances for Second-Order Methods5 Jul 2023 0 repositories listed
-
Near-Optimal Nonconvex-Strongly-Convex Bilevel Optimization with Fully First-Order Oracles26 Jun 2023 0 repositories listed
-
Minibatching Offers Improved Generalization Performance for Second Order Optimizers26 May 2023 0 repositories listed
-
Faster Differentially Private Convex Optimization via Second-Order Methods22 May 2023 0 repositories listed
-
Adaptive Consensus Optimization Method for GANs20 Apr 2023 0 repositories listed
-
Gradient-Boosted Based Structured and Unstructured Learning28 Feb 2023 0 repositories listed
-
Stochastic Dimension-reduced Second-order Methods for Policy Optimization28 Jan 2023 0 repositories listed
-
On backpropagating Hessians through ODEs19 Jan 2023 0 repositories listed
-
A survey of deep learning optimizers -- first and second order methods28 Nov 2022 0 repositories listed
-
Extra-Newton: A First Approach to Noise-Adaptive Accelerated Second-Order Methods3 Nov 2022 0 repositories listed
-
Explicit Second-Order Min-Max Optimization Methods with Optimal Convergence Guarantee23 Oct 2022 0 repositories listed
-
Component-Wise Natural Gradient Descent -- An Efficient Neural Network Optimization11 Oct 2022 0 repositories listed
-
Mirror Prox Algorithm for Large-Scale Cell-Free Massive MIMO Uplink Power Control20 Sep 2022 0 repositories listed
-
Adaptive Second Order Coresets for Data-efficient Machine Learning28 Jul 2022 0 repositories listed
-
SP2: A Second Order Stochastic Polyak Method17 Jul 2022 0 repositories listed
-
A Newton-CG based barrier method for finding a second-order stationary point of nonconvex conic optimization with complexity guarantees12 Jul 2022 0 repositories listed
-
Stochastic Second-Order Methods Improve Best-Known Sample Complexity of SGD for Gradient-Dominated Function25 May 2022 0 repositories listed
Syntology lines on 15 of the papers shown; no Syntology record for the others (a paper without an arXiv id cannot be joined to the graph, and absence from the graph layer is not a recorded non-run). “Ran” means the sample executed on a synthesized fixture, not that the paper's result was reproduced; each line links to that paper's sample list. Syntology's record for this page last changed ; when this build read Syntology's graph is in the build record. For agents: get_harvested_code_for_paper(arxiv_id) lists each paper's samples; how to connect.
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections