Methods › General › Attention Mechanisms › Class Attention
Class Attention
Introduced by Hugo Touvron et al. in Going deeper with Image Transformers
archive 2025-07-28 Description, source and code snippet are the archive's method entry.
A Class Attention layer, or CA Layer, is an attention mechanism for vision transformers used in CaiT that aims to extract information from a set of processed patches. It is identical to a self-attention layer, except that it relies on the attention between (i) the class embedding x_(class) (initialized at CLS in the first CA) and (ii) itself plus the set of frozen patch embeddings x_(patches) .
Considering a network with h heads and p patches, and denoting by d the embedding size, the multi-head class-attention is parameterized with several projection matrices, W_q, Wₖ, Wᵥ, Wₒ ∈R^(d ×d), and the corresponding biases b_q, bₖ, bᵥ, bₒ ∈Rᵈ . With this notation, the computation of the CA residual block proceeds as follows. We first augment the patch embeddings (in matrix form) as z=[x_(class), x_(patches)]. We then perform the projections:
Q=W_q x_(class)+b_q
K=Wₖ z+bₖ
V=Wᵥ z+bᵥ
The class-attention weights are given by
A=Softmax(Q . Kᵀ / √(d / h))
where Q . Kᵀ ∈R^(h ×1 ×p). This attention is involved in the weighted sum A ×V to produce the residual output vector
out_(CA)=Wₒ A V+bₒ
which is in turn added to x_(class) for subsequent processing.
In Syntology name not resolved (unknown_name). The lookup matched no method, paper or venue by this name; that asserts nothing about whether code exists.
Near matches the lookup lists, each a different name: attention-based probing classifier; bi-directional attention classifier; Class Semantics-based Attention (CSA); hybrid quantum-classical attention model; Class-Specific Residual Attention (CSRA); Class-Knowledge-Task Multi-Head Attention (CKT-MHA); Class-Weighted Sparse-Attention Fusion Network (SAFN); Stratified Class-Specific Attention Based Transformer Network.
Syntology's have check for this name (free, it answers coverage only), read 2026-10-05; ask it again live. A level is the served verification level of that one answer; nothing here is summed across methods.
Papers archive 2025-07-28
archive papers tagged: 36 · with a code link: 18 · where Syntology ran a sample: 7 (6 with a run with no instrument failure, 1 where every run was a failure of Syntology's instrument) Syntology
Show: all tagged papersonly where code ran (7 of 36 tagged: 6 with a run with no instrument failure, 1 where every run was a failure of Syntology's instrument)
Page 1 of 1: papers 1 to 36 of 36, newest first by the archive's date (ties by slug), in archive order.
Repository counts are the archive's code-links table. A Syntology line states what Syntology ran from that paper's harvested code, as “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the instrument figure counts failures of Syntology's instrument, not of the code. It is per sample and not a correctness claim. When the archive marks a repository official for the paper, the line starts with that repository's state (the archive's flag, not a verdict on who wrote the code; “community repositories only” when every sample that ran came from a community repository, “official: no sample here; runs from other or unrecorded repositories” when some came from a repository the paper names or has in its text, or from none recorded); hover it for the repositories the samples that ran came from.
-
Prompt-CAM: A Simpler Interpretable Transformer for Fine-Grained Analysis 16 Jan 2025 · 1 repository · arXiv:2501.09333Syntology 1 ran (of which 1 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified; the one sample that ran constructed an object rather than computing a result (of 1 harvested sample)
-
Prompt-CAM: Making Vision Transformers Interpretable for Fine-Grained Analysis 1 Jan 2025 · 1 repository
-
SpectralKD: A Unified Framework for Interpreting and Distilling Vision Transformers via Spectral Analysis 26 Dec 2024 · 1 repository · arXiv:2412.19055
-
An Attention-based Representation Distillation Baseline for Multi-Label Continual Learning 19 Jul 2024 · 1 repository · arXiv:2407.14249
-
LF-ViT: Reducing Spatial Redundancy in Vision Transformer for Efficient Image Recognition 8 Jan 2024 · 1 repository · arXiv:2402.00033Syntology official (archive's flag): 11 ran · 11 ran (of which 0 constructed an object rather than computing a result; 6 with no instrument failure: 1 honoured, 0 violated, 5 with no contract checked; 5 where Syntology's instrument failed) · 7 unverified (of 18 harvested samples) · 18 pointer-only (licence)
-
Self-distilled Masked Attention guided masked image modeling with noise Regularized Teacher (SMART) for medical image analysis 2 Oct 2023 · 0 repositories · arXiv:2310.01209
-
Class Attention Transfer Based Knowledge Distillation 25 Apr 2023 · 1 repository · arXiv:2304.12777Syntology official (archive's flag): 4 ran · 4 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified (of 4 harvested samples) · 4 pointer-only (licence)
-
SACANet: scene-aware class attention network for semantic segmentation of remote sensing images 22 Apr 2023 · 1 repository · arXiv:2304.11424
-
Detecting Severity of Diabetic Retinopathy from Fundus Images: A Transformer Network-based Review 3 Jan 2023 · 0 repositories · arXiv:2301.00973
-
Bidirectional Representations for Low Resource Spoken Language Understanding 24 Nov 2022 · 0 repositories · arXiv:2211.14320
-
Adaptive Sparse ViT: Towards Learnable Adaptive Token Pruning by Fully Exploiting Self-Attention 28 Sep 2022 · 1 repository · arXiv:2209.13802Syntology official: no sample here; runs from other or unrecorded repositories · 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 1 harvested sample) · 1 pointer-only (licence)
-
Class-attention Video Transformer for Engagement Intensity Prediction 12 Aug 2022 · 1 repository · arXiv:2208.07216
-
MaiT: Leverage Attention Masks for More Efficient Image Transformers 6 Jul 2022 · 0 repositories · arXiv:2207.03006
-
Augmenting Convolutional networks with attention-based aggregation 27 Dec 2021 · 5 repositories · arXiv:2112.13692Syntology official (archive's flag): 1 ran · 2 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 2 harvested samples)
-
SSA: Semantic Structure Aware Inference for Weakly Pixel-Wise Dense Predictions without Cost 5 Nov 2021 · 0 repositories · arXiv:2111.03392
-
MaiT: integrating spatial locality into image transformers with attention masks 29 Sep 2021 · 1 repository
-
Is cell segregation like oil and water: asymptotic versus transitory regime 1 Sep 2021 · 0 repositories · arXiv:2109.00364
-
Dynamic Relevance Learning for Few-Shot Object Detection 4 Aug 2021 · 1 repository · arXiv:2108.02235
-
Evo-ViT: Slow-Fast Token Evolution for Dynamic Vision Transformer 3 Aug 2021 · 1 repository · arXiv:2108.01390Syntology official (archive's flag): 7 ran · 7 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 3 honoured, 0 violated, 1 with no contract checked; 3 where Syntology's instrument failed) · 2 unverified (of 9 harvested samples) · 1 pointer-only (licence)
-
A comparison of latent semantic analysis and correspondence analysis of document-term matrices 25 Jul 2021 · 0 repositories · arXiv:2108.06197
-
Otimizacao de Redes Neurais atraves de Algoritmos Geneticos Celulares 18 Jul 2021 · 0 repositories · arXiv:2107.08326
-
Scientia Potentia Est -- On the Role of Knowledge in Computational Argumentation 1 Jul 2021 · 0 repositories · arXiv:2107.00281
-
Detecting Cattle and Elk in the Wild from Space 29 Jun 2021 · 0 repositories · arXiv:2106.15448
-
Towards self-organized control: Using neural cellular automata to robustly control a cart-pole agent 29 Jun 2021 · 1 repository · arXiv:2106.15240
-
Contrastive Attention for Automatic Chest X-ray Report Generation 13 Jun 2021 · 0 repositories · arXiv:2106.06965
-
A reversible system based on hybrid toggle radius-4 cellular automata and its application as a block cipher 9 Jun 2021 · 1 repository · arXiv:2106.04777
-
Cherry-Picking Gradients: Learning Low-Rank Embeddings of Visual Data via Differentiable Cross-Approximation 29 May 2021 · 1 repository · arXiv:2105.14250
-
Evolutionary Algorithms for Designing Reversible Cellular Automata 25 May 2021 · 1 repository · arXiv:2105.12039
-
Designing AI-based Conversational Agent for Diabetes Care in a Multilingual Context 20 May 2021 · 0 repositories · arXiv:2105.09490
-
EBM-Fold: Fully-Differentiable Protein Folding Powered by Energy-based Models 11 May 2021 · 0 repositories · arXiv:2105.04771
-
Elementary Intracellular Ca Signals are Initiated by a Transition of Release Channel System from a Metastable State 4 May 2021 · 0 repositories · arXiv:2105.01340
-
VIRDOCD: a VIRtual DOCtor to Predict Dengue Fatality 29 Apr 2021 · 0 repositories · arXiv:2104.14282
-
Modelling the Spread of an Epidemic in Presence of Vaccination using Cellular Automata 21 Apr 2021 · 0 repositories · arXiv:2104.10456
-
Time-Multiplexed Coded Aperture Imaging: Learned Coded Aperture and Pixel Exposures for Compressive Imaging Systems 6 Apr 2021 · 0 repositories · arXiv:2104.02820
-
A Conversational Agent System for Dietary Supplements Use 4 Apr 2021 · 0 repositories · arXiv:2104.01543
-
Going deeper with Image Transformers 31 Mar 2021 · 21 repositories · arXiv:2103.17239Syntology official: no sample here; runs from other or unrecorded repositories · 9 ran (of which 0 constructed an object rather than computing a result; 6 with no instrument failure: 0 honoured, 2 violated, 4 with no contract checked; 3 where Syntology's instrument failed) · 2 unverified (of 11 harvested samples) · 2 pointer-only (licence)
Tasks archive 2025-07-28
20 shown of 49 tasks the archive attaches to papers tagged with this method, by distinct papers. A task without a page in the catalog is plain text.
Usage over time archive 2025-07-28
Components: the archive holds no method-to-method composition, so PwC's Components table cannot be rebuilt; the Papers list carries no Results column for the same reason (the archive does not join its leaderboard rows to method tags).
Categories archive 2025-07-28
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections