Methods › General › Data Parallel Methods › Gradient Sparsification
Gradient Sparsification
Introduced by Jianqiao Wangni et al. in Gradient Sparsification for Communication-Efficient Distributed Optimization
archive 2025-07-28 Description, source and code snippet are the archive's method entry.
Gradient Sparsification is a technique for distributed training that sparsifies stochastic gradients to reduce the communication cost, with minor increase in the number of iterations. The key idea behind our sparsification technique is to drop some coordinates of the stochastic gradient and appropriately amplify the remaining coordinates to ensure the unbiasedness of the sparsified stochastic gradient. The sparsification approach can significantly reduce the coding length of the stochastic gradient and only slightly increase the variance of the stochastic gradient.
In Syntology name not resolved (unknown_name). The lookup matched no method, paper or venue by this name; that asserts nothing about whether code exists.
Near matches the lookup lists, each a different name: gradient EM; Gradient Cuff; Gradient Mirage; pathway gradient; Gradient Surgery; Gradient Sealing; Gradient Surgery; phantom gradient.
Syntology's have check for this name (free, it answers coverage only), read 2026-10-05; ask it again live. A level is the served verification level of that one answer; nothing here is summed across methods.
Papers archive 2025-07-28
archive papers tagged: 38 · with a code link: 10 · where Syntology ran a sample: 1 (1 with a run with no instrument failure, 0 where every run was a failure of Syntology's instrument) Syntology
Show: all tagged papersonly where code ran (1 of 38 tagged: 1 with a run with no instrument failure, 0 where every run was a failure of Syntology's instrument)
Page 1 of 1: papers 1 to 38 of 38, newest first by the archive's date (ties by slug), in archive order.
Repository counts are the archive's code-links table. A Syntology line states what Syntology ran from that paper's harvested code, as “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the instrument figure counts failures of Syntology's instrument, not of the code. It is per sample and not a correctness claim. When the archive marks a repository official for the paper, the line starts with that repository's state (the archive's flag, not a verdict on who wrote the code; “community repositories only” when every sample that ran came from a community repository, “official: no sample here; runs from other or unrecorded repositories” when some came from a repository the paper names or has in its text, or from none recorded); hover it for the repositories the samples that ran came from.
-
Mobility-Aware Asynchronous Federated Learning with Dynamic Sparsification 8 Jun 2025 · 0 repositories · arXiv:2506.07328
-
Dynamic Gradient Sparsification Training for Few-Shot Fine-tuning of CT Lymph Node Segmentation Foundation Model 2 Mar 2025 · 1 repository · arXiv:2503.00748
-
Sparse Incremental Aggregation in Satellite Federated Learning 20 Jan 2025 · 0 repositories · arXiv:2501.11385
-
Regularized Top-k: A Bayesian Framework for Gradient Sparsification 10 Jan 2025 · 0 repositories · arXiv:2501.05633
-
DQRM: Deep Quantized Recommendation Models 26 Oct 2024 · 1 repository · arXiv:2410.20046
-
Age-of-Gradient Updates for Federated Learning over Random Access Channels 15 Oct 2024 · 0 repositories · arXiv:2410.11986
-
Novel Gradient Sparsification Algorithm via Bayesian Inference 23 Sep 2024 · 0 repositories · arXiv:2409.14893
-
Preserving Near-Optimal Gradient Sparsification Cost for Scalable Distributed Deep Learning 21 Feb 2024 · 1 repository · arXiv:2402.13781
-
JointSQ: Joint Sparsification-Quantization for Distributed Learning 1 Jan 2024 · 1 repository
-
RS-DGC: Exploring Neighborhood Statistics for Dynamic Gradient Compression on Remote Sensing Image Interpretation 29 Dec 2023 · 0 repositories · arXiv:2312.17530
-
MiCRO: Near-Zero Cost Gradient Sparsification for Scaling and Accelerating Distributed DNN Training 2 Oct 2023 · 1 repository · arXiv:2310.00967
-
Gradient Sparsification For Masked Fine-Tuning of Transformers 19 Jul 2023 · 0 repositories · arXiv:2307.10098
-
DEFT: Exploiting Gradient Norm Difference between Model Layers for Scalable Gradient Sparsification 7 Jul 2023 · 1 repository · arXiv:2307.03500
-
Gradient Sparsification for Efficient Wireless Federated Learning with Differential Privacy 9 Apr 2023 · 0 repositories · arXiv:2304.04164
-
Efficient and Secure Federated Learning for Financial Applications 15 Mar 2023 · 0 repositories · arXiv:2303.08355
-
On the Interaction Between Differential Privacy and Gradient Compression in Deep Learning 1 Nov 2022 · 0 repositories · arXiv:2211.00734
-
Downlink Compression Improves TopK Sparsification 30 Sep 2022 · 0 repositories · arXiv:2209.15203
-
Empirical Analysis on Top-k Gradient Sparsification for Distributed Deep Learning in a Supercomputing Environment 18 Sep 2022 · 0 repositories · arXiv:2209.08497
-
Near-Optimal Sparse Allreduce for Distributed Deep Learning 19 Jan 2022 · 1 repository · arXiv:2201.07598
-
Sparsified Secure Aggregation for Privacy-Preserving Federated Learning 23 Dec 2021 · 0 repositories · arXiv:2112.12872
-
Communication-Efficient Federated Learning via Quantized Compressed Sensing 30 Nov 2021 · 0 repositories · arXiv:2111.15071
-
Federated Dynamic Neural Network for Deep MIMO Detection 24 Nov 2021 · 0 repositories · arXiv:2111.12260
-
Gradient Sparsification For Masked Fine-Tuning of Transformers 16 Nov 2021 · 0 repositories
-
Linear Convergence in Federated Learning: Tackling Client Heterogeneity and Sparse Gradients 14 Feb 2021 · 0 repositories · arXiv:2102.07053
-
Large-Scale Training System for 100-Million Classification at Alibaba 9 Feb 2021 · 0 repositories · arXiv:2102.06025
-
Gradient Sparsification Can Improve Performance of Differentially-Private Convex Machine Learning 30 Nov 2020 · 0 repositories · arXiv:2011.14572
-
A Reputation Mechanism Is All You Need: Collaborative Fairness and Adversarial Robustness in Federated Learning 20 Nov 2020 · 2 repositories · arXiv:2011.10464
-
FLAME: Differentially Private Federated Learning in the Shuffle Model 17 Sep 2020 · 1 repository · arXiv:2009.08063Syntology 1 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified (of 1 harvested sample)
-
rTop-k: A Statistical Estimation Approach to Distributed SGD 21 May 2020 · 0 repositories · arXiv:2005.10761
-
Adaptive Gradient Sparsification for Efficient Federated Learning: An Online Learning Approach 14 Jan 2020 · 0 repositories · arXiv:2001.04756
-
Layer-wise Adaptive Gradient Sparsification for Distributed Deep Learning with Convergence Guarantees 20 Nov 2019 · 0 repositories · arXiv:1911.08727
-
Understanding Top-k Sparsification in Distributed Deep Learning 20 Nov 2019 · 1 repository · arXiv:1911.08772
-
Gradient Sparification for Asynchronous Distributed Training 24 Oct 2019 · 0 repositories · arXiv:1910.10929
-
Hierarchical Federated Learning Across Heterogeneous Cellular Networks 5 Sep 2019 · 0 repositories · arXiv:1909.02362
-
A Linear Speedup Analysis of Distributed Deep Learning with Sparse and Quantized Communication 1 Dec 2018 · 0 repositories
-
Double Quantization for Communication-Efficient Distributed Optimization 25 May 2018 · 0 repositories · arXiv:1805.10111
-
Sparse Binary Compression: Towards Distributed Deep Learning with minimal Communication 22 May 2018 · 0 repositories · arXiv:1805.08768
-
Gradient Sparsification for Communication-Efficient Distributed Optimization 26 Oct 2017 · 0 repositories · arXiv:1710.09854
Tasks archive 2025-07-28
20 shown of 28 tasks the archive attaches to papers tagged with this method, by distinct papers. A task without a page in the catalog is plain text.
Usage over time archive 2025-07-28
Components: the archive holds no method-to-method composition, so PwC's Components table cannot be rebuilt; the Papers list carries no Results column for the same reason (the archive does not join its leaderboard rows to method tags).
Categories archive 2025-07-28
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections