{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/method/weight-standardization","slug":"weight-standardization","name":"Weight Standardization","full_name":"Weight Standardization","full_name_withheld":false,"description_markdown":"**Weight Standardization** is a normalization technique that smooths the loss landscape by standardizing the weights in convolutional layers. Different from the previous normalization methods that focus on *activations*, WS considers the smoothing effects of *weights* more than just length-direction decoupling. Theoretically, WS reduces the Lipschitz constants of the loss and the gradients.\r\nHence, WS smooths the loss landscape and improves training.\r\n\r\nIn Weight Standardization, instead of directly optimizing the loss $\\mathcal{L}$ on the original weights $\\hat{W}$, we reparameterize the weights $\\hat{W}$ as a function of $W$, i.e. $\\hat{W}=\\text{WS}(W)$, and optimize the loss $\\mathcal{L}$ on $W$ by [SGD](https://paperswithcode.com/method/sgd):\r\n\r\n$$\r\n    \\hat{W} = \\Big[ \\hat{W}\\_{i,j}~\\big|~ \\hat{W}\\_{i,j} = \\dfrac{W\\_{i,j} - \\mu\\_{W\\_{i,\\cdot}}}{\\sigma\\_{W\\_{i,\\cdot}+\\epsilon}}\\Big]\r\n$$\r\n\r\n$$\r\n    y = \\hat{W}*x\r\n$$\r\n\r\nwhere\r\n\r\n$$\r\n    \\mu_{W\\_{i,\\cdot}} = \\dfrac{1}{I}\\sum\\_{j=1}^{I}W\\_{i, j},~~\\sigma\\_{W\\_{i,\\cdot}}=\\sqrt{\\dfrac{1}{I}\\sum\\_{i=1}^I(W\\_{i,j} - \\mu\\_{W\\_{i,\\cdot}})^2}\r\n$$\r\n\r\nSimilar to [Batch Normalization](https://paperswithcode.com/method/batch-normalization), WS controls the first and second moments of the weights of each output channel individually in convolutional layers. Note that many initialization methods also initialize the weights in some similar ways. Different from those methods, WS standardizes the weights in a differentiable way which aims to normalize gradients during back-propagation. Note that we do not have any affine transformation on $\\hat{W}$. This is because we assume that normalization layers such as BN or [GN](https://paperswithcode.com/method/group-normalization) will normalize this convolutional layer again.","description_state":"present","introduced_year":null,"introduced_by":{"title":"Micro-Batch Training with Batch-Channel Normalization and Weight Standardization","paper":"/paper/weight-standardization","first_author":"Siyuan Qiao","n_authors":5,"url_abs":null,"archive_paper_url":"https://paperswithcode.com/paper/weight-standardization"},"source":{"url":"https://arxiv.org/abs/1903.10520v2","title":"Micro-Batch Training with Batch-Channel Normalization and Weight Standardization","url_on_a_paper_host":true},"code_snippet_url":null,"code_snippet_url_on_a_code_host":false,"categories":[{"area":"General","area_id":"general","collection":"Normalization","url":"/methods/category/normalization","pwc_aliases":[]}],"n_papers_tagged":12,"archive_num_papers":12,"papers_newest_first":[{"paper":null,"slug":"addressing-data-heterogeneity-in-federated-1","title":"Addressing Data Heterogeneity in Federated Learning with Adaptive Normalization-Free Feature Recalibration","date":"2024-10-02","arxiv_id":"2410.02006","n_code_links":0,"syntology":null},{"paper":null,"slug":"an-adaptive-gradient-regularization-method","title":"Adaptive Gradient Regularization: A Faster and Generalizable Optimization Technique for Deep Neural Networks","date":"2024-07-24","arxiv_id":"2407.16944","n_code_links":0,"syntology":null},{"paper":"/paper/exploring-loss-functions-for-time-based","slug":"exploring-loss-functions-for-time-based","title":"Exploring Loss Functions for Time-based Training Strategy in Spiking Neural Networks","date":"2023-09-21","arxiv_id":null,"n_code_links":1,"syntology":null},{"paper":"/paper/rotational-optimizers-simple-robust-dnn","slug":"rotational-optimizers-simple-robust-dnn","title":"Rotational Equilibrium: How Weight Decay Balances Learning Across Neural Networks","date":"2023-05-26","arxiv_id":"2305.17212","n_code_links":2,"syntology":{"ran":5,"of":6,"n_ran_checked":5,"n_instrument":0,"unverified":1,"pointer_only":0,"phrase":"5 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 0 honoured, 0 violated, 5 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified","official":{"repos":["epfml/req","epfml/rotational-optimizers"],"state":"official (archive's flag): 5 ran","n_ran":5,"n_constructed":0,"n_ran_no_instrument_failure":5,"n_unverified":1,"ran_from_kinds":["official"]}}},{"paper":null,"slug":"domain-adaptation-and-active-learning-for","title":"Domain Adaptation and Active Learning for Fine-Grained Recognition in the Field of Biodiversity","date":"2021-10-22","arxiv_id":"2110.11778","n_code_links":0,"syntology":null},{"paper":"/paper/effect-of-large-scale-pre-training-on-full","slug":"effect-of-large-scale-pre-training-on-full","title":"Effect of Pre-Training Scale on Intra- and Inter-Domain Full and Few-Shot Transfer Learning for Natural and Medical X-Ray Chest Images","date":"2021-05-31","arxiv_id":"2106.00116","n_code_links":1,"syntology":null},{"paper":"/paper/recu-reviving-the-dead-weights-in-binary","slug":"recu-reviving-the-dead-weights-in-binary","title":"ReCU: Reviving the Dead Weights in Binary Neural Networks","date":"2021-03-23","arxiv_id":"2103.12369","n_code_links":3,"syntology":{"ran":2,"of":3,"n_ran_checked":1,"n_instrument":1,"unverified":1,"pointer_only":3,"phrase":"2 ran (of which 1 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 1 where Syntology's instrument failed) · 1 unverified","official":{"repos":["z-hXu/ReCU"],"state":"community repositories only","n_ran":0,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":0,"ran_from_kinds":["listed"]}}},{"paper":"/paper/characterizing-signal-propagation-to-close-1","slug":"characterizing-signal-propagation-to-close-1","title":"Characterizing signal propagation to close the performance gap in unnormalized ResNets","date":"2021-01-21","arxiv_id":"2101.08692","n_code_links":4,"syntology":{"ran":1,"of":1,"n_ran_checked":0,"n_instrument":1,"unverified":0,"pointer_only":0,"phrase":"1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified","official":{"repos":["deepmind/deepmind-research"],"state":"official (archive's flag): 1 ran","n_ran":1,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":0,"ran_from_kinds":["official"]}}},{"paper":"/paper/monocular-depth-estimation-using-laplacian","slug":"monocular-depth-estimation-using-laplacian","title":"Monocular Depth Estimation Using Laplacian Pyramid-Based Depth Residuals","date":"2021-01-08","arxiv_id":null,"n_code_links":1,"syntology":null},{"paper":"/paper/gradient-centralization-a-new-optimization","slug":"gradient-centralization-a-new-optimization","title":"Gradient Centralization: A New Optimization Technique for Deep Neural Networks","date":"2020-04-03","arxiv_id":"2004.01461","n_code_links":8,"syntology":{"ran":18,"of":29,"n_ran_checked":17,"n_instrument":1,"unverified":11,"pointer_only":3,"phrase":"18 ran (of which 0 constructed an object rather than computing a result; 17 with no instrument failure: 1 honoured, 0 violated, 16 with no contract checked; 1 where Syntology's instrument failed) · 11 unverified","official":{"repos":["Yonghongwei/Gradient-Centralization"],"state":"community repositories only","n_ran":0,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":0,"ran_from_kinds":["listed"]}}},{"paper":"/paper/large-scale-learning-of-general-visual","slug":"large-scale-learning-of-general-visual","title":"Big Transfer (BiT): General Visual Representation Learning","date":"2019-12-24","arxiv_id":"1912.11370","n_code_links":9,"syntology":{"ran":9,"of":10,"n_ran_checked":7,"n_instrument":2,"unverified":1,"pointer_only":0,"phrase":"9 ran (of which 0 constructed an object rather than computing a result; 7 with no instrument failure: 0 honoured, 1 violated, 6 with no contract checked; 2 where Syntology's instrument failed) · 1 unverified","official":{"repos":["google-research/big_transfer"],"state":"community repositories only","n_ran":0,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":0,"ran_from_kinds":["listed"]}}},{"paper":"/paper/weight-standardization","slug":"weight-standardization","title":"Micro-Batch Training with Batch-Channel Normalization and Weight Standardization","date":"2019-03-25","arxiv_id":"1903.10520","n_code_links":7,"syntology":null}],"papers_shown":12,"papers_order":"date (newest first), then slug","paper_counts":{"archive_papers_tagged":12,"with_a_code_link":9,"where_syntology_ran_a_sample":5,"not_listed_spam_title":0,"listed":12,"listed_where_code_ran":5,"where_syntology_ran_a_sample_split":{"with_a_run_with_no_instrument_failure":4,"every_run_a_failure_of_syntologys_instrument":1,"listed_with_a_run_with_no_instrument_failure":4,"listed_every_run_a_failure_of_syntologys_instrument":1,"filter":{"states":["a run with no instrument failure","any run, instrument failures included"],"default":"a run with no instrument failure","note":"on the 'only where code ran' pages the default hides, in the browser, the rows where every run was a failure of Syntology's instrument; the second state shows them again. Rows are hidden, never re-ordered; these twins list every row"}},"definition":"distinct papers the archive tags; 'where Syntology ran a sample' counts papers with at least one harvested sample that ran, which is not a correctness claim"},"papers_where_code_ran":5,"papers_where_code_ran_first_page":"/method/weight-standardization/papers/ran/1","papers_where_code_ran_caption":"We ran code from the paper's repository; we did not isolate this method inside it.","pages":[{"order":"ran","page":1,"of":1,"url":"/method/weight-standardization/papers/ran/1","json":"/method/weight-standardization/papers/ran/1.json","rows":5}],"tasks":[{"task":"/task/image-classification","name":"Image Classification","papers":5},{"task":"/task/image-classification","name":"image-classification","papers":3},{"task":"/task/few-shot-learning","name":"Few-Shot Learning","papers":2},{"task":"/task/fine-grained-image-classification","name":"Fine-Grained Image Classification","papers":2},{"task":"/task/transfer-learning","name":"Transfer Learning","papers":2},{"task":"/task/active-learning","name":"Active Learning","papers":1},{"task":"/task/binarization","name":"Binarization","papers":1},{"task":"/task/decoder","name":"Decoder","papers":1},{"task":"/task/depth-estimation","name":"Depth Estimation","papers":1},{"task":"/task/domain-adaptation","name":"Domain Adaptation","papers":1},{"task":"/task/federated-learning","name":"Federated Learning","papers":1},{"task":null,"name":"GPU","papers":1},{"task":"/task/classification","name":"General Classification","papers":1},{"task":"/task/image-generation","name":"Image Generation","papers":1},{"task":"/task/instance-segmentation","name":"Instance Segmentation","papers":1},{"task":"/task/l2-regularization","name":"L2 Regularization","papers":1},{"task":"/task/language-modeling","name":"Language Modeling","papers":1},{"task":"/task/language-modelling","name":"Language Modelling","papers":1},{"task":"/task/monocular-depth-estimation","name":"Monocular Depth Estimation","papers":1},{"task":"/task/object-detection","name":"Object Detection","papers":1}],"tasks_shown":20,"n_tasks":30,"usage_by_year":[{"year":"2019","papers":2},{"year":"2020","papers":1},{"year":"2021","papers":5},{"year":"2023","papers":2},{"year":"2024","papers":2}],"row_source":"methods_table","archive":{"source":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","archive_url":"https://paperswithcode.com/method/weight-standardization"},"syntology_read_at":"2026-10-05T00:36:11+00:00","syntology":{"source":"Syntology's have check for this name (the free coverage lookup), read from Syntology's graph","have_url":"https://syntology.ai/api/graph/have?x=Weight%20Standardization","read_at":"2026-10-05T01:03:54+00:00","query":"Weight Standardization","kind":"method","have":"unknown_name","tier":null,"verified_implementation":null,"level":null,"level_meaning":"the served verification level, per answer; never summed across methods","resolved_to":null,"resolution":null,"resolved_name_differs":false,"absence_asserted":false,"absence_unchecked":null,"no_code_is_about_this_method":false,"subject":null,"archive_paper_arxiv_id":"1903.10520","repositories":null,"retrieve_with":null,"name_competition":{"verdict":"none","may_be_wrong_sense":false,"bearers_found":0,"served":null,"leading_other_bearer":null,"n_other_bearers_listed":0,"note":"Nothing in this corpus is named 'Weight Standardization' -- no method node, no paper announcing it in its title, no expanded form declaring it as an acronym."},"name_competition_text":null,"near_matches":["Parametric Weights Standardization (PWS)","Weight Mixup","weight scaling","Weight Weaving","Weight Recycling","weight operators","Weight Balancing","Filter-then-Weight"],"code_nearby":[],"code_nearby_caption":null,"origin_paper":null,"line":"name not resolved (unknown_name). The lookup matched no method, paper or venue by this name; that asserts nothing about whether code exists.","level_legend":null},"syntology_have_state":"loaded","record_sha256":"90cc8052328430d9df87e6bcdf11c3c238d0dc41a36cdf2fd89c087bc9e864ee","record_changed_at":"2026-10-05","record_changed_at_basis":"changed"}