{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/method/distributed-shampoo","slug":"distributed-shampoo","name":"Distributed Shampoo","full_name":"Distributed Shampoo","full_name_withheld":false,"description_markdown":"A scalable second order optimization algorithm for deep learning.\r\n\r\nOptimization in machine learning, both theoretical and applied, is presently dominated by first-order gradient methods such as stochastic gradient descent. Second-order optimization methods, that involve second derivatives and/or second order statistics of the data, are far less prevalent despite strong theoretical properties, due to their prohibitive computation, memory and communication costs. In an attempt to bridge this gap between theoretical and practical optimization, we present a scalable implementation of a second-order preconditioned method (concretely, a variant of full-matrix Adagrad), that along with several critical algorithmic and numerical improvements, provides significant convergence and wall-clock time improvements compared to conventional first-order methods on state-of-the-art deep models. Our novel design effectively utilizes the prevalent heterogeneous hardware architecture for training deep models, consisting of a multicore CPU coupled with multiple accelerator units. We demonstrate superior performance compared to state-of-the-art on very large learning tasks such as machine translation with Transformers, language modeling with BERT, click-through rate prediction on Criteo, and image classification on ImageNet with ResNet-50.","description_state":"present","introduced_year":null,"introduced_by":{"title":"Towards Practical Second Order Optimization for Deep Learning","paper":"/paper/towards-practical-second-order-optimization","first_author":"Rohan Anil","n_authors":5,"url_abs":null,"archive_paper_url":"https://paperswithcode.com/paper/towards-practical-second-order-optimization"},"source":{"url":"https://openreview.net/forum?id=Sc8cY4Jpi3s","title":"Towards Practical Second Order Optimization for Deep Learning","url_on_a_paper_host":true},"code_snippet_url":"https://github.com/google-research/google-research/tree/master/scalable_shampoo","code_snippet_url_on_a_code_host":true,"categories":[{"area":"General","area_id":"general","collection":"Large Batch Optimization","url":"/methods/category/large-batch-optimization","pwc_aliases":[]},{"area":"General","area_id":"general","collection":"Stochastic Optimization","url":"/methods/category/stochastic-optimization","pwc_aliases":[]}],"n_papers_tagged":5,"archive_num_papers":5,"papers_newest_first":[{"paper":"/paper/accelerating-neural-network-training-an","slug":"accelerating-neural-network-training-an","title":"Accelerating Neural Network Training: An Analysis of the AlgoPerf Competition","date":"2025-02-20","arxiv_id":"2502.15015","n_code_links":2,"syntology":{"ran":16,"of":19,"n_ran_checked":16,"n_instrument":0,"unverified":3,"pointer_only":0,"phrase":"16 ran (of which 0 constructed an object rather than computing a result; 16 with no instrument failure: 0 honoured, 0 violated, 16 with no contract checked; 0 where Syntology's instrument failed) · 3 unverified","official":null}},{"paper":"/paper/knowledge-distillation-a-good-teacher-is","slug":"knowledge-distillation-a-good-teacher-is","title":"Knowledge distillation: A good teacher is patient and consistent","date":"2021-06-09","arxiv_id":"2106.05237","n_code_links":10,"syntology":null},{"paper":"/paper/tensor-normal-training-for-deep-learning","slug":"tensor-normal-training-for-deep-learning","title":"Tensor Normal Training for Deep Learning Models","date":"2021-06-05","arxiv_id":"2106.02925","n_code_links":1,"syntology":{"ran":2,"of":3,"n_ran_checked":1,"n_instrument":1,"unverified":1,"pointer_only":3,"phrase":"2 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 1 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 1 unverified","official":{"repos":["renyiryry/tnt_neurips_2021"],"state":"official (archive's flag): 2 ran","n_ran":2,"n_constructed":0,"n_ran_no_instrument_failure":1,"n_unverified":1,"ran_from_kinds":["official"]}}},{"paper":"/paper/towards-practical-second-order-optimization","slug":"towards-practical-second-order-optimization","title":"Towards Practical Second Order Optimization for Deep Learning","date":"2021-01-01","arxiv_id":null,"n_code_links":0,"syntology":null},{"paper":"/paper/second-order-optimization-made-practical","slug":"second-order-optimization-made-practical","title":"Scalable Second Order Optimization for Deep Learning","date":"2020-02-20","arxiv_id":"2002.09018","n_code_links":2,"syntology":null}],"papers_shown":5,"papers_order":"date (newest first), then slug","paper_counts":{"archive_papers_tagged":5,"with_a_code_link":4,"where_syntology_ran_a_sample":2,"not_listed_spam_title":0,"listed":5,"listed_where_code_ran":2,"where_syntology_ran_a_sample_split":{"with_a_run_with_no_instrument_failure":2,"every_run_a_failure_of_syntologys_instrument":0,"listed_with_a_run_with_no_instrument_failure":2,"listed_every_run_a_failure_of_syntologys_instrument":0,"filter":{"states":["a run with no instrument failure","any run, instrument failures included"],"default":"a run with no instrument failure","note":"on the 'only where code ran' pages the default hides, in the browser, the rows where every run was a failure of Syntology's instrument; the second state shows them again. Rows are hidden, never re-ordered; these twins list every row"}},"definition":"distinct papers the archive tags; 'where Syntology ran a sample' counts papers with at least one harvested sample that ran, which is not a correctness claim"},"papers_where_code_ran":2,"papers_where_code_ran_first_page":"/method/distributed-shampoo/papers/ran/1","papers_where_code_ran_caption":"We ran code from the paper's repository; we did not isolate this method inside it.","pages":[{"order":"ran","page":1,"of":1,"url":"/method/distributed-shampoo/papers/ran/1","json":"/method/distributed-shampoo/papers/ran/1.json","rows":2}],"tasks":[{"task":"/task/deep-learning","name":"Deep Learning","papers":3},{"task":"/task/image-classification","name":"Image Classification","papers":3},{"task":null,"name":"CPU","papers":2},{"task":"/task/language-modeling","name":"Language Modeling","papers":2},{"task":"/task/language-modelling","name":"Language Modelling","papers":2},{"task":"/task/machine-translation","name":"Machine Translation","papers":2},{"task":"/task/translation","name":"Translation","papers":2},{"task":"/task/image-classification","name":"image-classification","papers":2},{"task":"/task/click-through-rate-prediction","name":"Click-Through Rate Prediction","papers":1},{"task":"/task/knowledge-distillation","name":"Knowledge Distillation","papers":1},{"task":"/task/second-order-methods","name":"Second-order methods","papers":1},{"task":"/task/stochastic-optimization","name":"Stochastic Optimization","papers":1}],"tasks_shown":12,"n_tasks":12,"usage_by_year":[{"year":"2020","papers":1},{"year":"2021","papers":3},{"year":"2025","papers":1}],"row_source":"methods_table","archive":{"source":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","archive_url":"https://paperswithcode.com/method/distributed-shampoo"},"syntology_read_at":"2026-10-05T00:36:11+00:00","syntology":{"source":"Syntology's have check for this name (the free coverage lookup), read from Syntology's graph","have_url":"https://syntology.ai/api/graph/have?x=Distributed%20Shampoo","read_at":"2026-10-05T00:55:10+00:00","query":"Distributed Shampoo","kind":"method","have":"unknown_name","tier":null,"verified_implementation":null,"level":null,"level_meaning":"the served verification level, per answer; never summed across methods","resolved_to":null,"resolution":null,"resolved_name_differs":false,"absence_asserted":false,"absence_unchecked":null,"no_code_is_about_this_method":false,"subject":null,"archive_paper_arxiv_id":null,"repositories":null,"retrieve_with":null,"name_competition":{"verdict":"none","may_be_wrong_sense":false,"bearers_found":0,"served":null,"leading_other_bearer":null,"n_other_bearers_listed":0,"note":"Nothing in this corpus is named 'Distributed Shampoo' -- no method node, no paper announcing it in its title, no expanded form declaring it as an acronym."},"name_competition_text":null,"near_matches":["4-bit Shampoo","Distributed Sentinel","distributed prompt caching","Distributed Matrix Mechanism","Unapologetically Distributed","Distributed Game-Theoretic RL","distributed sparse embeddings","layer-wise distributed optimizer"],"code_nearby":[],"code_nearby_caption":null,"origin_paper":null,"line":"name not resolved (unknown_name). The lookup matched no method, paper or venue by this name; that asserts nothing about whether code exists.","level_legend":null},"syntology_have_state":"loaded","record_sha256":"9a5a65a2f1cb990504c61df0baea0085035f0e1277b2b2158baf146f4bebfa29","record_changed_at":"2026-10-05","record_changed_at_basis":"changed"}