{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/method/ca","slug":"ca","name":"Class Attention","full_name":"Class Attention","full_name_withheld":false,"description_markdown":"A **Class Attention** layer, or **CA Layer**, is an [attention mechanism](https://paperswithcode.com/methods/category/attention-mechanisms-1) for [vision transformers](https://paperswithcode.com/methods/category/vision-transformer) used in [CaiT](https://paperswithcode.com/method/cait) that aims to extract information from a set of processed patches. It is identical to a [self-attention layer](https://paperswithcode.com/method/scaled), except that it relies on the attention between (i) the class embedding $x_{\\text {class }}$ (initialized at CLS in the first CA) and (ii) itself plus the set of frozen patch embeddings $x_{\\text {patches }} .$ \r\n\r\nConsidering a network with $h$ heads and $p$ patches, and denoting by $d$ the embedding size, the multi-head class-attention is parameterized with several projection matrices, $W_{q}, W_{k}, W_{v}, W_{o} \\in \\mathbf{R}^{d \\times d}$, and the corresponding biases $b_{q}, b_{k}, b_{v}, b_{o} \\in \\mathbf{R}^{d} .$ With this notation, the computation of the CA residual block proceeds as follows. We first augment the patch embeddings (in matrix form) as $z=\\left[x_{\\text {class }}, x_{\\text {patches }}\\right]$. We then perform the projections:\r\n\r\n$$Q=W\\_{q} x\\_{\\text {class }}+b\\_{q}$$\r\n\r\n$$K=W\\_{k} z+b\\_{k}$$\r\n\r\n$$V=W\\_{v} z+b\\_{v}$$\r\n\r\nThe class-attention weights are given by\r\n\r\n$$\r\nA=\\operatorname{Softmax}\\left(Q . K^{T} / \\sqrt{d / h}\\right)\r\n$$\r\n\r\nwhere $Q . K^{T} \\in \\mathbf{R}^{h \\times 1 \\times p}$. This attention is involved in the weighted sum $A \\times V$ to produce the residual output vector\r\n\r\n$$\r\n\\operatorname{out}\\_{\\mathrm{CA}}=W\\_{o} A V+b\\_{o}\r\n$$\r\n\r\nwhich is in turn added to $x\\_{\\text {class }}$ for subsequent processing.","description_state":"present","introduced_year":null,"introduced_by":{"title":"Going deeper with Image Transformers","paper":"/paper/going-deeper-with-image-transformers","first_author":"Hugo Touvron","n_authors":5,"url_abs":null,"archive_paper_url":"https://paperswithcode.com/paper/going-deeper-with-image-transformers"},"source":{"url":"https://arxiv.org/abs/2103.17239v2","title":"Going deeper with Image Transformers","url_on_a_paper_host":true},"code_snippet_url":null,"code_snippet_url_on_a_code_host":false,"categories":[{"area":"General","area_id":"general","collection":"Attention Mechanisms","url":"/methods/category/attention-mechanisms","pwc_aliases":["attention-mechanisms-1"]},{"area":"General","area_id":"general","collection":"Attention","url":"/methods/category/attention","pwc_aliases":[]}],"n_papers_tagged":36,"archive_num_papers":36,"papers_newest_first":[{"paper":"/paper/prompt-cam-a-simpler-interpretable","slug":"prompt-cam-a-simpler-interpretable","title":"Prompt-CAM: A Simpler Interpretable Transformer for Fine-Grained Analysis","date":"2025-01-16","arxiv_id":"2501.09333","n_code_links":1,"syntology":{"ran":1,"of":1,"n_ran_checked":1,"n_instrument":0,"unverified":0,"pointer_only":0,"phrase":"1 ran (of which 1 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified; the one sample that ran constructed an object rather than computing a result","official":null}},{"paper":"/paper/prompt-cam-making-vision-transformers","slug":"prompt-cam-making-vision-transformers","title":"Prompt-CAM: Making Vision Transformers Interpretable for Fine-Grained Analysis","date":"2025-01-01","arxiv_id":null,"n_code_links":1,"syntology":null},{"paper":"/paper/spectralkd-understanding-and-optimizing","slug":"spectralkd-understanding-and-optimizing","title":"SpectralKD: A Unified Framework for Interpreting and Distilling Vision Transformers via Spectral Analysis","date":"2024-12-26","arxiv_id":"2412.19055","n_code_links":1,"syntology":null},{"paper":"/paper/an-attention-based-representation","slug":"an-attention-based-representation","title":"An Attention-based Representation Distillation Baseline for Multi-Label Continual Learning","date":"2024-07-19","arxiv_id":"2407.14249","n_code_links":1,"syntology":null},{"paper":"/paper/lf-vit-reducing-spatial-redundancy-in-vision","slug":"lf-vit-reducing-spatial-redundancy-in-vision","title":"LF-ViT: Reducing Spatial Redundancy in Vision Transformer for Efficient Image Recognition","date":"2024-01-08","arxiv_id":"2402.00033","n_code_links":1,"syntology":{"ran":11,"of":18,"n_ran_checked":6,"n_instrument":5,"unverified":7,"pointer_only":18,"phrase":"11 ran (of which 0 constructed an object rather than computing a result; 6 with no instrument failure: 1 honoured, 0 violated, 5 with no contract checked; 5 where Syntology's instrument failed) · 7 unverified","official":{"repos":["edgeai1/lf-vit"],"state":"official (archive's flag): 11 ran","n_ran":11,"n_constructed":0,"n_ran_no_instrument_failure":6,"n_unverified":7,"ran_from_kinds":["official"]}}},{"paper":null,"slug":"self-distilled-masked-attention-guided-masked","title":"Self-distilled Masked Attention guided masked image modeling with noise Regularized Teacher (SMART) for medical image analysis","date":"2023-10-02","arxiv_id":"2310.01209","n_code_links":0,"syntology":null},{"paper":"/paper/class-attention-transfer-based-knowledge","slug":"class-attention-transfer-based-knowledge","title":"Class Attention Transfer Based Knowledge Distillation","date":"2023-04-25","arxiv_id":"2304.12777","n_code_links":1,"syntology":{"ran":4,"of":4,"n_ran_checked":1,"n_instrument":3,"unverified":0,"pointer_only":4,"phrase":"4 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified","official":{"repos":["gzyaftermath/cat-kd"],"state":"official (archive's flag): 4 ran","n_ran":4,"n_constructed":0,"n_ran_no_instrument_failure":1,"n_unverified":0,"ran_from_kinds":["official"]}}},{"paper":"/paper/sacanet-scene-aware-class-attention-network","slug":"sacanet-scene-aware-class-attention-network","title":"SACANet: scene-aware class attention network for semantic segmentation of remote sensing images","date":"2023-04-22","arxiv_id":"2304.11424","n_code_links":1,"syntology":null},{"paper":null,"slug":"detecting-severity-of-diabetic-retinopathy","title":"Detecting Severity of Diabetic Retinopathy from Fundus Images: A Transformer Network-based Review","date":"2023-01-03","arxiv_id":"2301.00973","n_code_links":0,"syntology":null},{"paper":null,"slug":"bidirectional-representations-for-low","title":"Bidirectional Representations for Low Resource Spoken Language Understanding","date":"2022-11-24","arxiv_id":"2211.14320","n_code_links":0,"syntology":null},{"paper":"/paper/adaptive-sparse-vit-towards-learnable","slug":"adaptive-sparse-vit-towards-learnable","title":"Adaptive Sparse ViT: Towards Learnable Adaptive Token Pruning by Fully Exploiting Self-Attention","date":"2022-09-28","arxiv_id":"2209.13802","n_code_links":1,"syntology":{"ran":1,"of":1,"n_ran_checked":0,"n_instrument":1,"unverified":0,"pointer_only":1,"phrase":"1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified","official":{"repos":["cydia2018/as-vit"],"state":"official: no sample here; runs from other or unrecorded repositories","n_ran":0,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":0,"ran_from_kinds":["unlocated"]}}},{"paper":"/paper/class-attention-video-transformer-for","slug":"class-attention-video-transformer-for","title":"Class-attention Video Transformer for Engagement Intensity Prediction","date":"2022-08-12","arxiv_id":"2208.07216","n_code_links":1,"syntology":null},{"paper":null,"slug":"mait-leverage-attention-masks-for-more","title":"MaiT: Leverage Attention Masks for More Efficient Image Transformers","date":"2022-07-06","arxiv_id":"2207.03006","n_code_links":0,"syntology":null},{"paper":"/paper/augmenting-convolutional-networks-with","slug":"augmenting-convolutional-networks-with","title":"Augmenting Convolutional networks with attention-based aggregation","date":"2021-12-27","arxiv_id":"2112.13692","n_code_links":5,"syntology":{"ran":2,"of":2,"n_ran_checked":1,"n_instrument":1,"unverified":0,"pointer_only":0,"phrase":"2 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified","official":{"repos":["facebookresearch/deit"],"state":"official (archive's flag): 1 ran","n_ran":1,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":0,"ran_from_kinds":["listed","official"]}}},{"paper":null,"slug":"ssa-semantic-structure-aware-inference-for","title":"SSA: Semantic Structure Aware Inference for Weakly Pixel-Wise Dense Predictions without Cost","date":"2021-11-05","arxiv_id":"2111.03392","n_code_links":0,"syntology":null},{"paper":"/paper/mait-integrating-spatial-locality-into-image","slug":"mait-integrating-spatial-locality-into-image","title":"MaiT: integrating spatial locality into image transformers with attention masks","date":"2021-09-29","arxiv_id":null,"n_code_links":1,"syntology":null},{"paper":null,"slug":"is-cell-segregation-just-like-oil-and-water","title":"Is cell segregation like oil and water: asymptotic versus transitory regime","date":"2021-09-01","arxiv_id":"2109.00364","n_code_links":0,"syntology":null},{"paper":"/paper/dynamic-relevance-learning-for-few-shot","slug":"dynamic-relevance-learning-for-few-shot","title":"Dynamic Relevance Learning for Few-Shot Object Detection","date":"2021-08-04","arxiv_id":"2108.02235","n_code_links":1,"syntology":null},{"paper":"/paper/evo-vit-slow-fast-token-evolution-for-dynamic","slug":"evo-vit-slow-fast-token-evolution-for-dynamic","title":"Evo-ViT: Slow-Fast Token Evolution for Dynamic Vision Transformer","date":"2021-08-03","arxiv_id":"2108.01390","n_code_links":1,"syntology":{"ran":7,"of":9,"n_ran_checked":4,"n_instrument":3,"unverified":2,"pointer_only":1,"phrase":"7 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 3 honoured, 0 violated, 1 with no contract checked; 3 where Syntology's instrument failed) · 2 unverified","official":{"repos":["YifanXu74/Evo-ViT"],"state":"official (archive's flag): 7 ran","n_ran":7,"n_constructed":0,"n_ran_no_instrument_failure":4,"n_unverified":2,"ran_from_kinds":["official"]}}},{"paper":null,"slug":"a-comparison-of-latent-semantic-analysis-and","title":"A comparison of latent semantic analysis and correspondence analysis of document-term matrices","date":"2021-07-25","arxiv_id":"2108.06197","n_code_links":0,"syntology":null},{"paper":null,"slug":"otimizacao-de-redes-neurais-atraves-de","title":"Otimizacao de Redes Neurais atraves de Algoritmos Geneticos Celulares","date":"2021-07-18","arxiv_id":"2107.08326","n_code_links":0,"syntology":null},{"paper":null,"slug":"scientia-potentia-est-on-the-role-of","title":"Scientia Potentia Est -- On the Role of Knowledge in Computational Argumentation","date":"2021-07-01","arxiv_id":"2107.00281","n_code_links":0,"syntology":null},{"paper":null,"slug":"detecting-cattle-and-elk-in-the-wild-from","title":"Detecting Cattle and Elk in the Wild from Space","date":"2021-06-29","arxiv_id":"2106.15448","n_code_links":0,"syntology":null},{"paper":"/paper/towards-self-organized-control-using-neural","slug":"towards-self-organized-control-using-neural","title":"Towards self-organized control: Using neural cellular automata to robustly control a cart-pole agent","date":"2021-06-29","arxiv_id":"2106.15240","n_code_links":1,"syntology":null},{"paper":null,"slug":"contrastive-attention-for-automatic-chest-x","title":"Contrastive Attention for Automatic Chest X-ray Report Generation","date":"2021-06-13","arxiv_id":"2106.06965","n_code_links":0,"syntology":null},{"paper":"/paper/a-reversible-system-based-on-hybrid-toggle","slug":"a-reversible-system-based-on-hybrid-toggle","title":"A reversible system based on hybrid toggle radius-4 cellular automata and its application as a block cipher","date":"2021-06-09","arxiv_id":"2106.04777","n_code_links":1,"syntology":null},{"paper":"/paper/cherry-picking-gradients-learning-low-rank","slug":"cherry-picking-gradients-learning-low-rank","title":"Cherry-Picking Gradients: Learning Low-Rank Embeddings of Visual Data via Differentiable Cross-Approximation","date":"2021-05-29","arxiv_id":"2105.14250","n_code_links":1,"syntology":null},{"paper":"/paper/evolutionary-algorithms-for-designing","slug":"evolutionary-algorithms-for-designing","title":"Evolutionary Algorithms for Designing Reversible Cellular Automata","date":"2021-05-25","arxiv_id":"2105.12039","n_code_links":1,"syntology":null},{"paper":null,"slug":"designing-ai-based-conversational-agent-for","title":"Designing AI-based Conversational Agent for Diabetes Care in a Multilingual Context","date":"2021-05-20","arxiv_id":"2105.09490","n_code_links":0,"syntology":null},{"paper":null,"slug":"ebm-fold-fully-differentiable-protein-folding","title":"EBM-Fold: Fully-Differentiable Protein Folding Powered by Energy-based Models","date":"2021-05-11","arxiv_id":"2105.04771","n_code_links":0,"syntology":null},{"paper":null,"slug":"elementary-intracellular-ca-signals-are","title":"Elementary Intracellular Ca Signals are Initiated by a Transition of Release Channel System from a Metastable State","date":"2021-05-04","arxiv_id":"2105.01340","n_code_links":0,"syntology":null},{"paper":null,"slug":"virdoc-statistical-and-machine-learning-by-a","title":"VIRDOCD: a VIRtual DOCtor to Predict Dengue Fatality","date":"2021-04-29","arxiv_id":"2104.14282","n_code_links":0,"syntology":null},{"paper":null,"slug":"modelling-the-spread-of-an-epidemic-in","title":"Modelling the Spread of an Epidemic in Presence of Vaccination using Cellular Automata","date":"2021-04-21","arxiv_id":"2104.10456","n_code_links":0,"syntology":null},{"paper":null,"slug":"time-multiplexed-coded-aperture-imaging","title":"Time-Multiplexed Coded Aperture Imaging: Learned Coded Aperture and Pixel Exposures for Compressive Imaging Systems","date":"2021-04-06","arxiv_id":"2104.02820","n_code_links":0,"syntology":null},{"paper":null,"slug":"a-conversational-agent-system-for-dietary","title":"A Conversational Agent System for Dietary Supplements Use","date":"2021-04-04","arxiv_id":"2104.01543","n_code_links":0,"syntology":null},{"paper":"/paper/going-deeper-with-image-transformers","slug":"going-deeper-with-image-transformers","title":"Going deeper with Image Transformers","date":"2021-03-31","arxiv_id":"2103.17239","n_code_links":21,"syntology":{"ran":9,"of":11,"n_ran_checked":6,"n_instrument":3,"unverified":2,"pointer_only":2,"phrase":"9 ran (of which 0 constructed an object rather than computing a result; 6 with no instrument failure: 0 honoured, 2 violated, 4 with no contract checked; 3 where Syntology's instrument failed) · 2 unverified","official":{"repos":["rwightman/pytorch-image-models","facebookresearch/deit"],"state":"official: no sample here; runs from other or unrecorded repositories","n_ran":0,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":0,"ran_from_kinds":["listed","unlocated"]}}}],"papers_shown":36,"papers_order":"date (newest first), then slug","paper_counts":{"archive_papers_tagged":36,"with_a_code_link":18,"where_syntology_ran_a_sample":7,"not_listed_spam_title":0,"listed":36,"listed_where_code_ran":7,"where_syntology_ran_a_sample_split":{"with_a_run_with_no_instrument_failure":6,"every_run_a_failure_of_syntologys_instrument":1,"listed_with_a_run_with_no_instrument_failure":6,"listed_every_run_a_failure_of_syntologys_instrument":1,"filter":{"states":["a run with no instrument failure","any run, instrument failures included"],"default":"a run with no instrument failure","note":"on the 'only where code ran' pages the default hides, in the browser, the rows where every run was a failure of Syntology's instrument; the second state shows them again. Rows are hidden, never re-ordered; these twins list every row"}},"definition":"distinct papers the archive tags; 'where Syntology ran a sample' counts papers with at least one harvested sample that ran, which is not a correctness claim"},"papers_where_code_ran":7,"papers_where_code_ran_first_page":"/method/ca/papers/ran/1","papers_where_code_ran_caption":"We ran code from the paper's repository; we did not isolate this method inside it.","pages":[{"order":"ran","page":1,"of":1,"url":"/method/ca/papers/ran/1","json":"/method/ca/papers/ran/1.json","rows":7}],"tasks":[{"task":"/task/image-classification","name":"Image Classification","papers":3},{"task":"/task/semantic-segmentation","name":"Semantic Segmentation","papers":3},{"task":"/task/transfer-learning","name":"Transfer Learning","papers":3},{"task":"/task/efficient-vits","name":"Efficient ViTs","papers":2},{"task":"/task/knowledge-distillation","name":"Knowledge Distillation","papers":2},{"task":"/task/natural-language-understanding","name":"Natural Language Understanding","papers":2},{"task":"/task/object-detection","name":"Object Detection","papers":2},{"task":"/task/visual-prompt-tuning","name":"Visual Prompt Tuning","papers":2},{"task":"/task/image-classification","name":"image-classification","papers":2},{"task":"/task/authorship-attribution","name":"Authorship Attribution","papers":1},{"task":"/task/automatic-speech-recognition-2","name":"Automatic Speech Recognition","papers":1},{"task":"/task/automatic-speech-recognition","name":"Automatic Speech Recognition (ASR)","papers":1},{"task":"/task/machine-learning","name":"BIG-bench Machine Learning","papers":1},{"task":"/task/classification-1","name":"Classification","papers":1},{"task":"/task/common-sense-reasoning","name":"Common Sense Reasoning","papers":1},{"task":"/task/computed-tomography-ct","name":"Computed Tomography (CT)","papers":1},{"task":"/task/continual-learning","name":"Continual Learning","papers":1},{"task":"/task/decision-making","name":"Decision Making","papers":1},{"task":"/task/denoising","name":"Denoising","papers":1},{"task":"/task/dimensionality-reduction","name":"Dimensionality Reduction","papers":1}],"tasks_shown":20,"n_tasks":49,"usage_by_year":[{"year":"2021","papers":23},{"year":"2022","papers":4},{"year":"2023","papers":4},{"year":"2024","papers":3},{"year":"2025","papers":2}],"row_source":"methods_table","archive":{"source":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","archive_url":"https://paperswithcode.com/method/ca"},"syntology_read_at":"2026-10-05T00:36:11+00:00","syntology":{"source":"Syntology's have check for this name (the free coverage lookup), read from Syntology's graph","have_url":"https://syntology.ai/api/graph/have?x=Class%20Attention","read_at":"2026-10-05T00:53:18+00:00","query":"Class Attention","kind":"method","have":"unknown_name","tier":null,"verified_implementation":null,"level":null,"level_meaning":"the served verification level, per answer; never summed across methods","resolved_to":null,"resolution":null,"resolved_name_differs":false,"absence_asserted":false,"absence_unchecked":null,"no_code_is_about_this_method":false,"subject":null,"archive_paper_arxiv_id":"2103.17239","repositories":null,"retrieve_with":null,"name_competition":{"verdict":"none","may_be_wrong_sense":false,"bearers_found":0,"served":null,"leading_other_bearer":null,"n_other_bearers_listed":0,"note":"Nothing in this corpus is named 'Class Attention' -- no method node, no paper announcing it in its title, no expanded form declaring it as an acronym."},"name_competition_text":null,"near_matches":["attention-based probing classifier","bi-directional attention classifier","Class Semantics-based Attention (CSA)","hybrid quantum-classical attention model","Class-Specific Residual Attention (CSRA)","Class-Knowledge-Task Multi-Head Attention (CKT-MHA)","Class-Weighted Sparse-Attention Fusion Network (SAFN)","Stratified Class-Specific Attention Based Transformer Network"],"code_nearby":[],"code_nearby_caption":null,"origin_paper":null,"line":"name not resolved (unknown_name). The lookup matched no method, paper or venue by this name; that asserts nothing about whether code exists.","level_legend":null},"syntology_have_state":"loaded","record_sha256":"18d6394bec51ec89dc2a8c3bff09150a0266858a3ff63f78bee1a3fdebdbe6f9","record_changed_at":"2026-10-05","record_changed_at_basis":"changed"}