{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/method/layerscale","slug":"layerscale","name":"LayerScale","full_name":"LayerScale","full_name_withheld":false,"description_markdown":"**LayerScale** is a method used for [vision transformer](https://paperswithcode.com/methods/category/vision-transformer) architectures to help improve training dynamics. It adds a learnable diagonal matrix on output of each residual block, initialized close to (but not at) 0. Adding this simple layer after each residual block improves the training dynamic, allowing for the training of deeper high-capacity image transformers that benefit from depth.\r\n\r\nSpecifically, LayerScale is a per-channel multiplication of the vector produced by each residual block, as opposed to a single scalar, see Figure (d). The objective is to group the updates of the weights associated with the same output channel. Formally, LayerScale is a multiplication by a diagonal matrix on output of each residual block. In other words:\r\n\r\n$$\r\nx\\_{l}^{\\prime} =x\\_{l}+\\operatorname{diag}\\left(\\lambda\\_{l, 1}, \\ldots, \\lambda\\_{l, d}\\right) \\times \\operatorname{SA}\\left(\\eta\\left(x\\_{l}\\right)\\right) \r\n$$\r\n\r\n$$\r\nx\\_{l+1} =x\\_{l}^{\\prime}+\\operatorname{diag}\\left(\\lambda\\_{l, 1}^{\\prime}, \\ldots, \\lambda\\_{l, d}^{\\prime}\\right) \\times \\operatorname{FFN}\\left(\\eta\\left(x\\_{l}^{\\prime}\\right)\\right)\r\n$$\r\n\r\nwhere the parameters $\\lambda\\_{l, i}$ and $\\lambda\\_{l, i}^{\\prime}$ are learnable weights. The diagonal values are all initialized to a fixed small value $\\varepsilon:$ we set it to $\\varepsilon=0.1$ until depth 18 , $\\varepsilon=10^{-5}$ for depth 24 and $\\varepsilon=10^{-6}$ for deeper networks. \r\n\r\nThis formula is akin to other [normalization](https://paperswithcode.com/methods/category/normalization) strategies [ActNorm](https://paperswithcode.com/method/activation-normalization) or [LayerNorm](https://paperswithcode.com/method/layer-normalization) but executed on output of the residual block. Yet LayerScale seeks a different effect: [ActNorm](https://paperswithcode.com/method/activation-normalization) is a data-dependent initialization that calibrates activations so that they have zero-mean and unit variance, like [BatchNorm](https://paperswithcode.com/method/batch-normalization). In contrast, in LayerScale, we initialize the diagonal with small values so that the initial contribution of the residual branches to the function implemented by the transformer is small. In that respect the motivation is therefore closer to that of [ReZero](https://paperswithcode.com/method/rezero), [SkipInit](https://paperswithcode.com/method/skipinit), [Fixup](https://paperswithcode.com/method/fixup-initialization) and [T-Fixup](https://paperswithcode.com/method/t-fixup): to train closer to the identity function and let the network integrate the additional parameters progressively during the training. LayerScale offers more diversity in the optimization than just adjusting the whole layer by a single learnable scalar as in [ReZero](https://paperswithcode.com/method/rezero)/[SkipInit](https://paperswithcode.com/method/skipinit), [Fixup](https://paperswithcode.com/method/fixup-initialization) and [T-Fixup](https://paperswithcode.com/method/t-fixup).","description_state":"present","introduced_year":null,"introduced_by":{"title":"Going deeper with Image Transformers","paper":"/paper/going-deeper-with-image-transformers","first_author":"Hugo Touvron","n_authors":5,"url_abs":null,"archive_paper_url":"https://paperswithcode.com/paper/going-deeper-with-image-transformers"},"source":{"url":"https://arxiv.org/abs/2103.17239v2","title":"Going deeper with Image Transformers","url_on_a_paper_host":true},"code_snippet_url":null,"code_snippet_url_on_a_code_host":false,"categories":[{"area":"General","area_id":"general","collection":"Normalization","url":"/methods/category/normalization","pwc_aliases":[]},{"area":"General","area_id":"general","collection":"Regularization","url":"/methods/category/regularization","pwc_aliases":[]}],"n_papers_tagged":12,"archive_num_papers":12,"papers_newest_first":[{"paper":"/paper/guiding-the-experts-semantic-priors-for","slug":"guiding-the-experts-semantic-priors-for","title":"Guiding the Experts: Semantic Priors for Efficient and Focused MoE Routing","date":"2025-05-24","arxiv_id":"2505.18586","n_code_links":1,"syntology":null},{"paper":"/paper/spectralkd-understanding-and-optimizing","slug":"spectralkd-understanding-and-optimizing","title":"SpectralKD: A Unified Framework for Interpreting and Distilling Vision Transformers via Spectral Analysis","date":"2024-12-26","arxiv_id":"2412.19055","n_code_links":1,"syntology":null},{"paper":null,"slug":"detecting-severity-of-diabetic-retinopathy","title":"Detecting Severity of Diabetic Retinopathy from Fundus Images: A Transformer Network-based Review","date":"2023-01-03","arxiv_id":"2301.00973","n_code_links":0,"syntology":null},{"paper":null,"slug":"mait-leverage-attention-masks-for-more","title":"MaiT: Leverage Attention Masks for More Efficient Image Transformers","date":"2022-07-06","arxiv_id":"2207.03006","n_code_links":0,"syntology":null},{"paper":"/paper/deit-iii-revenge-of-the-vit","slug":"deit-iii-revenge-of-the-vit","title":"DeiT III: Revenge of the ViT","date":"2022-04-14","arxiv_id":"2204.07118","n_code_links":12,"syntology":null},{"paper":"/paper/three-things-everyone-should-know-about","slug":"three-things-everyone-should-know-about","title":"Three things everyone should know about Vision Transformers","date":"2022-03-18","arxiv_id":"2203.09795","n_code_links":8,"syntology":{"ran":1,"of":1,"n_ran_checked":1,"n_instrument":0,"unverified":0,"pointer_only":0,"phrase":"1 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 1 violated, 0 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified","official":{"repos":["facebookresearch/deit"],"state":"community repositories only","n_ran":0,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":0,"ran_from_kinds":["listed"]}}},{"paper":"/paper/a-convnet-for-the-2020s","slug":"a-convnet-for-the-2020s","title":"A ConvNet for the 2020s","date":"2022-01-10","arxiv_id":"2201.03545","n_code_links":54,"syntology":{"ran":57,"of":80,"n_ran_checked":49,"n_instrument":8,"unverified":23,"pointer_only":12,"phrase":"57 ran (of which 39 constructed an object rather than computing a result; 49 with no instrument failure: 2 honoured, 0 violated, 47 with no contract checked; 8 where Syntology's instrument failed) · 23 unverified","official":{"repos":["facebookresearch/ConvNeXt"],"state":"community repositories only","n_ran":0,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":0,"ran_from_kinds":["listed"]}}},{"paper":"/paper/augmenting-convolutional-networks-with","slug":"augmenting-convolutional-networks-with","title":"Augmenting Convolutional networks with attention-based aggregation","date":"2021-12-27","arxiv_id":"2112.13692","n_code_links":5,"syntology":{"ran":2,"of":2,"n_ran_checked":1,"n_instrument":1,"unverified":0,"pointer_only":0,"phrase":"2 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified","official":{"repos":["facebookresearch/deit"],"state":"official (archive's flag): 1 ran","n_ran":1,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":0,"ran_from_kinds":["listed","official"]}}},{"paper":null,"slug":"poformer-a-simple-pooling-transformer-for","title":"Poformer: A simple pooling transformer for speaker verification","date":"2021-10-10","arxiv_id":"2110.04692","n_code_links":0,"syntology":null},{"paper":"/paper/mait-integrating-spatial-locality-into-image","slug":"mait-integrating-spatial-locality-into-image","title":"MaiT: integrating spatial locality into image transformers with attention masks","date":"2021-09-29","arxiv_id":null,"n_code_links":1,"syntology":null},{"paper":"/paper/resmlp-feedforward-networks-for-image","slug":"resmlp-feedforward-networks-for-image","title":"ResMLP: Feedforward networks for image classification with data-efficient training","date":"2021-05-07","arxiv_id":"2105.03404","n_code_links":19,"syntology":{"ran":5,"of":7,"n_ran_checked":5,"n_instrument":0,"unverified":2,"pointer_only":0,"phrase":"5 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 1 honoured, 1 violated, 3 with no contract checked; 0 where Syntology's instrument failed) · 2 unverified","official":{"repos":["facebookresearch/deit"],"state":"community repositories only","n_ran":0,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":0,"ran_from_kinds":["listed"]}}},{"paper":"/paper/going-deeper-with-image-transformers","slug":"going-deeper-with-image-transformers","title":"Going deeper with Image Transformers","date":"2021-03-31","arxiv_id":"2103.17239","n_code_links":21,"syntology":{"ran":9,"of":11,"n_ran_checked":6,"n_instrument":3,"unverified":2,"pointer_only":2,"phrase":"9 ran (of which 0 constructed an object rather than computing a result; 6 with no instrument failure: 0 honoured, 2 violated, 4 with no contract checked; 3 where Syntology's instrument failed) · 2 unverified","official":{"repos":["rwightman/pytorch-image-models","facebookresearch/deit"],"state":"official: no sample here; runs from other or unrecorded repositories","n_ran":0,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":0,"ran_from_kinds":["listed","unlocated"]}}}],"papers_shown":12,"papers_order":"date (newest first), then slug","paper_counts":{"archive_papers_tagged":12,"with_a_code_link":9,"where_syntology_ran_a_sample":5,"not_listed_spam_title":0,"listed":12,"listed_where_code_ran":5,"where_syntology_ran_a_sample_split":{"with_a_run_with_no_instrument_failure":5,"every_run_a_failure_of_syntologys_instrument":0,"listed_with_a_run_with_no_instrument_failure":5,"listed_every_run_a_failure_of_syntologys_instrument":0,"filter":{"states":["a run with no instrument failure","any run, instrument failures included"],"default":"a run with no instrument failure","note":"on the 'only where code ran' pages the default hides, in the browser, the rows where every run was a failure of Syntology's instrument; the second state shows them again. Rows are hidden, never re-ordered; these twins list every row"}},"definition":"distinct papers the archive tags; 'where Syntology ran a sample' counts papers with at least one harvested sample that ran, which is not a correctness claim"},"papers_where_code_ran":5,"papers_where_code_ran_first_page":"/method/layerscale/papers/ran/1","papers_where_code_ran_caption":"We ran code from the paper's repository; we did not isolate this method inside it.","pages":[{"order":"ran","page":1,"of":1,"url":"/method/layerscale/papers/ran/1","json":"/method/layerscale/papers/ran/1.json","rows":5}],"tasks":[{"task":"/task/image-classification","name":"Image Classification","papers":6},{"task":"/task/image-classification","name":"image-classification","papers":5},{"task":"/task/semantic-segmentation","name":"Semantic Segmentation","papers":3},{"task":"/task/transfer-learning","name":"Transfer Learning","papers":3},{"task":"/task/classification-1","name":"Classification","papers":2},{"task":"/task/data-augmentation","name":"Data Augmentation","papers":2},{"task":"/task/fine-grained-image-classification","name":"Fine-Grained Image Classification","papers":2},{"task":"/task/object-detection","name":"Object Detection","papers":2},{"task":"/task/domain-generalization","name":"Domain Generalization","papers":1},{"task":"/task/classification","name":"General Classification","papers":1},{"task":"/task/image-segmentation","name":"Image Segmentation","papers":1},{"task":"/task/knowledge-distillation","name":"Knowledge Distillation","papers":1},{"task":"/task/machine-translation","name":"Machine Translation","papers":1},{"task":"/task/mixture-of-experts","name":"Mixture-of-Experts","papers":1},{"task":"/task/real-time-object-detection","name":"Real-Time Object Detection","papers":1},{"task":"/task/self-supervised-image-classification","name":"Self-Supervised Image Classification","papers":1},{"task":"/task/self-supervised-learning","name":"Self-Supervised Learning","papers":1},{"task":"/task/speaker-verification","name":"Speaker Verification","papers":1},{"task":"/task/translation","name":"Translation","papers":1}],"tasks_shown":19,"n_tasks":19,"usage_by_year":[{"year":"2021","papers":5},{"year":"2022","papers":4},{"year":"2023","papers":1},{"year":"2024","papers":1},{"year":"2025","papers":1}],"row_source":"methods_table","archive":{"source":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","archive_url":"https://paperswithcode.com/method/layerscale"},"syntology_read_at":"2026-10-05T00:36:11+00:00","syntology":{"source":"Syntology's have check for this name (the free coverage lookup), read from Syntology's graph","have_url":"https://syntology.ai/api/graph/have?x=LayerScale","read_at":"2026-10-05T00:58:27+00:00","query":"LayerScale","kind":"method","have":"unknown_name","tier":null,"verified_implementation":null,"level":null,"level_meaning":"the served verification level, per answer; never summed across methods","resolved_to":null,"resolution":null,"resolved_name_differs":false,"absence_asserted":false,"absence_unchecked":null,"no_code_is_about_this_method":false,"subject":null,"archive_paper_arxiv_id":"2103.17239","repositories":null,"retrieve_with":null,"name_competition":{"verdict":"none","may_be_wrong_sense":false,"bearers_found":0,"served":null,"leading_other_bearer":null,"n_other_bearers_listed":0,"note":"Nothing in this corpus is named 'LayerScale' -- no method node, no paper announcing it in its title, no expanded form declaring it as an acronym."},"name_competition_text":null,"near_matches":[],"code_nearby":[],"code_nearby_caption":null,"origin_paper":null,"line":"name not resolved (unknown_name). The lookup matched no method, paper or venue by this name; that asserts nothing about whether code exists.","level_legend":null},"syntology_have_state":"loaded","record_sha256":"46e141777785f81753caa4d46dea8d032d8b8e3feca74e596a0994360dd83d48","record_changed_at":"2026-10-05","record_changed_at_basis":"changed"}