{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/method/sm3","slug":"sm3","name":"SM3","full_name":"SM3","full_name_withheld":false,"description_markdown":"# Memory-Efficient Adaptive Optimization\r\n\r\nSource: https://arxiv.org/abs/1901.11150\r\n\r\nAdaptive gradient-based optimizers such as [AdaGrad](https://paperswithcode.com/method/adagrad) and [Adam](https://paperswithcode.com/method/adam) are among the\r\ndefacto methods of choice in modern machine learning.These methods tune the learning rate for each parameter during the optimization process using cumulative second-order statistics. These methods provide superior convergence properties and are very attractive in large scale applications due to their moderate time and space requirements which are linear in the number of parameters.\r\n\r\n\r\nHowever, the recent advances in natural language processing such as [BERT](https://paperswithcode.com/method/bert) and GPT2 show that models with 10<sup>8</sup> to 10<sup>10</sup> parameters, trained with adaptive optimization methods, achieve state-of-the-art results. In such cases, the memory overhead of the optimizer can restrict the size of the model that can be used as well as the batch size, both of which can have a dramatic effect on the quality of the final model.\r\n\r\n\r\nHere we construct a new adaptive optimization method that retains most of the benefits of standard per-parameter adaptivity while significantly reducing memory overhead.\r\n\r\n\r\nWe observe that in standard neural networks that certain entries of the stochastic gradients have (on average) similar values, and exhibit what we refer to as an activation pattern. For example, in gradients of embedding layers of deep networks, an entire row (or column) is either zero or non-zero. Similarly, in intermediate layers we often observe that gradients associated with the same unit are of similar order of magnitude. In these cases, a similar phenomenon is observed in the second-order statistics maintained by adaptive methods. With this key observation, to reduce the memory overhead of the optimizer our method takes in a cover set of the parameters. Cover sets are typically selected in practice such that parameters in each of the sets have second order statistics of similar magnitude. Our method is general enough that it can easily be extended to arbitrary cover sets. For parameters of deep networks that are organized as a collection of tensors, we form a cover consisting of slices of codimension one for each tensor. Thus, for an m x n parameter matrix, the cover consists of rows and columns of the matrix. The memory requirements therefore drop from mxn to merely m+n. For a parameter tensor of rank p, with dimensions n<sub>1</sub>  ...   n<sub>p</sub>, the reduction in memory consumption is even more pronounced, dropping from product of all the dimensions to the sum of all dimensions. This virtually eliminates the memory overhead associated with maintaining the adaptive learning rates!\r\n\r\nAnother practical aspect worthy of note is that our method does not require an external hand engineered learning rate decay schedule but instead relies on the per parameter adaptivity that is natural to its update rule which makes it easier to tune. We provide details in the supplementary section of the paper.\r\n\r\n## Advice on using SM3 on your model\r\n\r\n### Learning rate warm-up:\r\n\r\n```python\r\nlearning_rate = lr_constant * tf.minimum(1.0, (warm_up_step / global_step) ** p)\r\n```\r\n\r\n* p = 1, linear ramp up of learning rate.\r\n* p = 2, quadratic ramp up of learning rate [preferred].\r\n\r\nWe typically set `warm_up_step` as 5% of overall steps. Initially, the norm of the preconditioned gradient is much larger than norm of the weights. Learning rate warmup allows us to heuristically fix this scale mismatch.\r\n\r\n### Learning rate decay:\r\n\r\nWe make use accumulated gradient squares for the decay. This means that each coordinate gets its own natural decay based on the scales of the gradients over time. Hence, users need not put in an external learning rate decay schedule. Moreover, we found in our experiments with translation and language models that this approach is superior to a hand-tuned learning rate decay schedules which is typically combined with exponential moving averages of the gradient squares.\r\n\r\nHaving said that if users want to add exponential moving averages instead of the standard accumulated gradient squares - It's easy to modify the optimizer implementation to switch to exponential moving averages.\r\n\r\nFor rank > 1:\r\n\r\n|            from                     |                  to                 |\r\n|-------------------------------------|-------------------------------------|\r\n|  current_accumulator += grad * grad |  current_accumulator = beta * current_accumulator + (1-beta) * grad * grad |\r\n\r\n\r\nFor rank <= 1:\r\n\r\n\r\n|            from                     |                  to                 |\r\n|-------------------------------------|-------------------------------------|\r\n|  current_accumulator = tf.assign_add(accumulator, grad * grad) |   current_accumulator = tf.assign(accumulator, beta * accumulator + (1-beta) * (grad * grad)) |\r\n\r\n\r\n### [Polyak averaging](https://paperswithcode.com/method/polyak-averaging) of parameters: \r\nIt's useful to run [polyak averaging](https://www.tensorflow.org/api_docs/python/tf/train/ExponentialMovingAverage) of the parameters. These parameters are then used in inference / serving. Using the averaged parameters instead of the last iterate typically improves the overall performance of the model.\r\n\r\nAn **alternative** to polyak averaging which does not make use of extra memory is to decay the learning rate from the constant to zero for the last 10% of the steps of your entire training run, we term the phase a **cool-down** phase of the model. As training makes smaller and smaller steps the final iterate can be thought of as an average iterate.","description_state":"present","introduced_year":null,"introduced_by":{"title":null,"paper":null,"first_author":null,"n_authors":0,"url_abs":null,"archive_paper_url":null},"source":{"url":"http://papers.nips.cc/paper/9168-memory-efficient-adaptive-optimization","title":"Memory Efficient Adaptive Optimization","url_on_a_paper_host":true},"code_snippet_url":null,"code_snippet_url_on_a_code_host":false,"categories":[{"area":"General","area_id":"general","collection":"Stochastic Optimization","url":"/methods/category/stochastic-optimization","pwc_aliases":[]}],"n_papers_tagged":7,"archive_num_papers":null,"papers_newest_first":[{"paper":"/paper/smmf-square-matricized-momentum-factorization","slug":"smmf-square-matricized-momentum-factorization","title":"SMMF: Square-Matricized Momentum Factorization for Memory-Efficient Optimization","date":"2024-12-12","arxiv_id":"2412.08894","n_code_links":1,"syntology":null},{"paper":null,"slug":"deep-learning-based-detection-of-bacterial","title":"Deep Learning-based Detection of Bacterial Swarm Motion Using a Single Image","date":"2024-10-19","arxiv_id":"2410.15229","n_code_links":0,"syntology":null},{"paper":"/paper/scaling-sparse-fine-tuning-to-large-language","slug":"scaling-sparse-fine-tuning-to-large-language","title":"Scaling Sparse Fine-Tuning to Large Language Models","date":"2024-01-29","arxiv_id":"2401.16405","n_code_links":2,"syntology":{"ran":13,"of":15,"n_ran_checked":4,"n_instrument":9,"unverified":2,"pointer_only":5,"phrase":"13 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 2 honoured, 0 violated, 2 with no contract checked; 9 where Syntology's instrument failed) · 2 unverified","official":{"repos":["alanansell/peft","ducdauge/sft-llm"],"state":"official (archive's flag): 12 ran","n_ran":12,"n_constructed":0,"n_ran_no_instrument_failure":3,"n_unverified":2,"ran_from_kinds":["official","unlocated"]}}},{"paper":null,"slug":"png-bert-augmented-bert-on-phonemes-and","title":"PnG BERT: Augmented BERT on Phonemes and Graphemes for Neural TTS","date":"2021-03-28","arxiv_id":"2103.15060","n_code_links":0,"syntology":null},{"paper":"/paper/memory-efficient-adaptive-optimization","slug":"memory-efficient-adaptive-optimization","title":"Memory Efficient Adaptive Optimization","date":"2019-12-01","arxiv_id":null,"n_code_links":1,"syntology":null},{"paper":null,"slug":"big-bidirectional-insertion-representations","title":"Big Bidirectional Insertion Representations for Documents","date":"2019-10-29","arxiv_id":"1910.13034","n_code_links":0,"syntology":null},{"paper":"/paper/memory-efficient-adaptive-optimization-for","slug":"memory-efficient-adaptive-optimization-for","title":"Memory-Efficient Adaptive Optimization","date":"2019-01-30","arxiv_id":"1901.11150","n_code_links":4,"syntology":null}],"papers_shown":7,"papers_order":"date (newest first), then slug","paper_counts":{"archive_papers_tagged":7,"with_a_code_link":4,"where_syntology_ran_a_sample":1,"not_listed_spam_title":0,"listed":7,"listed_where_code_ran":1,"where_syntology_ran_a_sample_split":{"with_a_run_with_no_instrument_failure":1,"every_run_a_failure_of_syntologys_instrument":0,"listed_with_a_run_with_no_instrument_failure":1,"listed_every_run_a_failure_of_syntologys_instrument":0,"filter":{"states":["a run with no instrument failure","any run, instrument failures included"],"default":"a run with no instrument failure","note":"on the 'only where code ran' pages the default hides, in the browser, the rows where every run was a failure of Syntology's instrument; the second state shows them again. Rows are hidden, never re-ordered; these twins list every row"}},"definition":"distinct papers the archive tags; 'where Syntology ran a sample' counts papers with at least one harvested sample that ran, which is not a correctness claim"},"papers_where_code_ran":1,"papers_where_code_ran_first_page":"/method/sm3/papers/ran/1","papers_where_code_ran_caption":"We ran code from the paper's repository; we did not isolate this method inside it.","pages":[{"order":"ran","page":1,"of":1,"url":"/method/sm3/papers/ran/1","json":"/method/sm3/papers/ran/1.json","rows":1}],"tasks":[{"task":"/task/translation","name":"Translation","papers":3},{"task":"/task/language-modeling","name":"Language Modeling","papers":2},{"task":"/task/language-modelling","name":"Language Modelling","papers":2},{"task":"/task/machine-translation","name":"Machine Translation","papers":2},{"task":"/task/diagnostic","name":"Diagnostic","papers":1},{"task":"/task/form","name":"Form","papers":1},{"task":"/task/quantization","name":"Quantization","papers":1},{"task":"/task/representation-learning","name":"Representation Learning","papers":1},{"task":"/task/sensitivity","name":"Sensitivity","papers":1},{"task":"/task/sentence","name":"Sentence","papers":1},{"task":"/task/specificity","name":"Specificity","papers":1},{"task":"/task/text-generation","name":"Text Generation","papers":1},{"task":"/task/text-to-speech-synthesis","name":"Text-To-Speech Synthesis","papers":1},{"task":"/task/parameter-efficient-fine-tuning","name":"parameter-efficient fine-tuning","papers":1}],"tasks_shown":14,"n_tasks":14,"usage_by_year":[{"year":"2019","papers":3},{"year":"2021","papers":1},{"year":"2024","papers":3}],"row_source":"embedded","archive":{"source":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","archive_url":"https://paperswithcode.com/method/sm3"},"syntology_read_at":"2026-10-05T00:36:11+00:00","syntology":{"source":"Syntology's have check for this name (the free coverage lookup), read from Syntology's graph","have_url":"https://syntology.ai/api/graph/have?x=SM3","read_at":"2026-10-05T01:02:09+00:00","query":"SM3","kind":"method","have":"unknown_name","tier":null,"verified_implementation":null,"level":null,"level_meaning":"the served verification level, per answer; never summed across methods","resolved_to":null,"resolution":null,"resolved_name_differs":false,"absence_asserted":false,"absence_unchecked":null,"no_code_is_about_this_method":false,"subject":null,"archive_paper_arxiv_id":null,"repositories":null,"retrieve_with":null,"name_competition":{"verdict":"none","may_be_wrong_sense":false,"bearers_found":0,"served":null,"leading_other_bearer":null,"n_other_bearers_listed":0,"note":"Nothing in this corpus is named 'SM3' -- no method node, no paper announcing it in its title, no expanded form declaring it as an acronym."},"name_competition_text":null,"near_matches":[],"code_nearby":[],"code_nearby_caption":null,"origin_paper":null,"line":"name not resolved (unknown_name). The lookup matched no method, paper or venue by this name; that asserts nothing about whether code exists.","level_legend":null},"syntology_have_state":"loaded","record_sha256":"0ff8206c2bb1a762fccf83684c138b59dcae47420135cc810209beee4710efd7","record_changed_at":"2026-10-05","record_changed_at_basis":"changed"}