{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/method/cvt","slug":"cvt","name":"CvT","full_name":"Convolutional Vision Transformer","full_name_withheld":false,"description_markdown":"The **Convolutional vision Transformer (CvT)** is an architecture which incorporates convolutions into the [Transformer](https://paperswithcode.com/method/transformer). The CvT design introduces convolutions to two core sections of the ViT architecture.\r\n\r\nFirst, the Transformers are partitioned into multiple stages that form a hierarchical structure of Transformers. The beginning of each stage consists of a convolutional token embedding that performs an overlapping [convolution](https://paperswithcode.com/method/convolution) operation with stride on a 2D-reshaped token map (i.e., reshaping flattened token sequences back to the spatial grid), followed by [layer normalization](https://paperswithcode.com/method/layer-normalization). This allows the model to not only capture local information, but also progressively decrease the sequence length while simultaneously increasing the dimension of token features across stages, achieving spatial downsampling while concurrently increasing the number of feature maps, as is performed in CNNs. \r\n\r\nSecond, the linear projection prior to every self-attention block in the Transformer module is replaced with a proposed convolutional projection, which employs a s × s depth-wise separable convolution operation on an 2D-reshaped token map. This allows the model to further capture local spatial context and reduce semantic ambiguity in the attention mechanism. It also permits management of computational complexity, as the stride of convolution can be used to subsample the key and value matrices to improve efficiency by 4× or more, with minimal degradation of performance.","description_state":"present","introduced_year":null,"introduced_by":{"title":"CvT: Introducing Convolutions to Vision Transformers","paper":"/paper/cvt-introducing-convolutions-to-vision","first_author":"Haiping Wu","n_authors":7,"url_abs":null,"archive_paper_url":"https://paperswithcode.com/paper/cvt-introducing-convolutions-to-vision"},"source":{"url":"https://arxiv.org/abs/2103.15808v1","title":"CvT: Introducing Convolutions to Vision Transformers","url_on_a_paper_host":true},"code_snippet_url":null,"code_snippet_url_on_a_code_host":false,"categories":[{"area":"Computer Vision","area_id":"computer-vision","collection":"Image Models","url":"/methods/category/image-models","pwc_aliases":[]},{"area":"Computer Vision","area_id":"computer-vision","collection":"Vision Transformers","url":"/methods/category/vision-transformers","pwc_aliases":["vision-transformer"]}],"n_papers_tagged":12,"archive_num_papers":12,"papers_newest_first":[{"paper":"/paper/residual-transformer-fusion-network-for-salt-1","slug":"residual-transformer-fusion-network-for-salt-1","title":"Residual Transformer Fusion Network for Salt and Pepper Image Denoising","date":"2025-02-13","arxiv_id":"2502.09000","n_code_links":0,"syntology":null},{"paper":"/paper/spectral-convolutional-transformer","slug":"spectral-convolutional-transformer","title":"Heracles: A Hybrid SSM-Transformer Model for High-Resolution Image and Time-Series Analysis","date":"2024-03-26","arxiv_id":"2403.18063","n_code_links":2,"syntology":{"ran":4,"of":4,"n_ran_checked":4,"n_instrument":0,"unverified":0,"pointer_only":4,"phrase":"4 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 1 honoured, 0 violated, 3 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified","official":{"repos":["badripatro/heracles","badripatro/sct"],"state":"official (archive's flag): 4 ran","n_ran":4,"n_constructed":0,"n_ran_no_instrument_failure":4,"n_unverified":0,"ran_from_kinds":["official"]}}},{"paper":null,"slug":"a-comparison-of-deep-learning-models-for","title":"A Comparison of Deep Learning Models for Proton Background Rejection with the AMS Electromagnetic Calorimeter","date":"2024-02-26","arxiv_id":"2402.16285","n_code_links":0,"syntology":null},{"paper":null,"slug":"on-convolutional-vision-transformers-for","title":"On Convolutional Vision Transformers for Yield Prediction","date":"2024-02-08","arxiv_id":"2402.05557","n_code_links":0,"syntology":null},{"paper":null,"slug":"making-vision-transformers-truly-shift","title":"Making Vision Transformers Truly Shift-Equivariant","date":"2023-05-25","arxiv_id":"2305.16316","n_code_links":0,"syntology":null},{"paper":null,"slug":"a-novel-framework-for-decentralized-dynamic","title":"A Novel Framework for Decentralized Dynamic Resource Allocation Using Voronoi Tessellations","date":"2022-11-30","arxiv_id":"2212.00140","n_code_links":0,"syntology":null},{"paper":null,"slug":"online-continual-learning-with-contrastive","title":"Online Continual Learning with Contrastive Vision Transformer","date":"2022-07-24","arxiv_id":"2207.13516","n_code_links":0,"syntology":null},{"paper":"/paper/sima-simple-softmax-free-attention-for-vision","slug":"sima-simple-softmax-free-attention-for-vision","title":"SimA: Simple Softmax-free Attention for Vision Transformers","date":"2022-06-17","arxiv_id":"2206.08898","n_code_links":1,"syntology":{"ran":2,"of":3,"n_ran_checked":2,"n_instrument":0,"unverified":1,"pointer_only":3,"phrase":"2 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified","official":{"repos":["ucdvision/sima"],"state":"official (archive's flag): 2 ran","n_ran":2,"n_constructed":0,"n_ran_no_instrument_failure":2,"n_unverified":1,"ran_from_kinds":["official"]}}},{"paper":"/paper/convolutional-xformers-for-vision","slug":"convolutional-xformers-for-vision","title":"Convolutional Xformers for Vision","date":"2022-01-25","arxiv_id":"2201.10271","n_code_links":1,"syntology":null},{"paper":null,"slug":"energy-optimal-design-and-control-of-electric-1","title":"Energy-optimal Design and Control of Electric Powertrains under Motor Thermal Constraints","date":"2021-11-15","arxiv_id":"2111.07711","n_code_links":0,"syntology":null},{"paper":null,"slug":"convolutional-gated-mlp-combining","title":"Convolutional Gated MLP: Combining Convolutions & gMLP","date":"2021-11-06","arxiv_id":"2111.03940","n_code_links":0,"syntology":null},{"paper":"/paper/cvt-introducing-convolutions-to-vision","slug":"cvt-introducing-convolutions-to-vision","title":"CvT: Introducing Convolutions to Vision Transformers","date":"2021-03-29","arxiv_id":"2103.15808","n_code_links":16,"syntology":{"ran":39,"of":47,"n_ran_checked":36,"n_instrument":3,"unverified":8,"pointer_only":8,"phrase":"39 ran (of which 19 constructed an object rather than computing a result; 36 with no instrument failure: 2 honoured, 0 violated, 34 with no contract checked; 3 where Syntology's instrument failed) · 8 unverified","official":{"repos":["microsoft/CvT"],"state":"official (archive's flag): 10 ran","n_ran":10,"n_constructed":0,"n_ran_no_instrument_failure":10,"n_unverified":4,"ran_from_kinds":["listed","named_in_paper","official","unlocated"]}}}],"papers_shown":12,"papers_order":"date (newest first), then slug","paper_counts":{"archive_papers_tagged":12,"with_a_code_link":4,"where_syntology_ran_a_sample":3,"not_listed_spam_title":0,"listed":12,"listed_where_code_ran":3,"where_syntology_ran_a_sample_split":{"with_a_run_with_no_instrument_failure":3,"every_run_a_failure_of_syntologys_instrument":0,"listed_with_a_run_with_no_instrument_failure":3,"listed_every_run_a_failure_of_syntologys_instrument":0,"filter":{"states":["a run with no instrument failure","any run, instrument failures included"],"default":"a run with no instrument failure","note":"on the 'only where code ran' pages the default hides, in the browser, the rows where every run was a failure of Syntology's instrument; the second state shows them again. Rows are hidden, never re-ordered; these twins list every row"}},"definition":"distinct papers the archive tags; 'where Syntology ran a sample' counts papers with at least one harvested sample that ran, which is not a correctness claim"},"papers_where_code_ran":3,"papers_where_code_ran_first_page":"/method/cvt/papers/ran/1","papers_where_code_ran_caption":"We ran code from the paper's repository; we did not isolate this method inside it.","pages":[{"order":"ran","page":1,"of":1,"url":"/method/cvt/papers/ran/1","json":"/method/cvt/papers/ran/1.json","rows":3}],"tasks":[{"task":"/task/image-classification","name":"Image Classification","papers":3},{"task":"/task/semantic-segmentation","name":"Semantic Segmentation","papers":2},{"task":"/task/transfer-learning","name":"Transfer Learning","papers":2},{"task":"/task/image-classification","name":"image-classification","papers":2},{"task":"/task/continual-learning","name":"Continual Learning","papers":1},{"task":"/task/contrastive-learning","name":"Contrastive Learning","papers":1},{"task":"/task/deep-learning","name":"Deep Learning","papers":1},{"task":"/task/denoising","name":"Denoising","papers":1},{"task":null,"name":"GPU","papers":1},{"task":"/task/image-denoising","name":"Image Denoising","papers":1},{"task":"/task/image-reconstruction","name":"Image Reconstruction","papers":1},{"task":"/task/inductive-bias","name":"Inductive Bias","papers":1},{"task":"/task/instance-segmentation","name":"Instance Segmentation","papers":1},{"task":"/task/mamba","name":"Mamba","papers":1},{"task":"/task/prediction","name":"Prediction","papers":1},{"task":"/task/state-space-models","name":"State Space Models","papers":1},{"task":"/task/time-series-1","name":"Time Series","papers":1},{"task":"/task/time-series","name":"Time Series Analysis","papers":1}],"tasks_shown":18,"n_tasks":18,"usage_by_year":[{"year":"2021","papers":3},{"year":"2022","papers":4},{"year":"2023","papers":1},{"year":"2024","papers":3},{"year":"2025","papers":1}],"row_source":"methods_table","archive":{"source":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","archive_url":"https://paperswithcode.com/method/cvt"},"syntology_read_at":"2026-10-05T00:36:11+00:00","syntology":{"source":"Syntology's have check for this name (the free coverage lookup), read from Syntology's graph","have_url":"https://syntology.ai/api/graph/have?x=CvT","read_at":"2026-10-05T00:54:23+00:00","query":"CvT","kind":"method","have":"declared_code","tier":"repo_known","verified_implementation":false,"level":null,"level_meaning":"the served verification level, per answer; never summed across methods","resolved_to":{"method":"CvT","origin_arxiv_id":"2103.15808"},"resolution":"exact","resolved_name_differs":false,"absence_asserted":false,"absence_unchecked":null,"no_code_is_about_this_method":false,"subject":"same_paper","archive_paper_arxiv_id":"2103.15808","repositories":{"listed":5,"in_graph":16,"list_truncated":true,"strongest_link":"official"},"retrieve_with":null,"name_competition":{"verdict":"contested","may_be_wrong_sense":false,"bearers_found":4,"served":{"arxiv_id":"2103.15808","citations":2519},"leading_other_bearer":{"arxiv_id":"2207.13516","title":"Online Continual Learning with Contrastive Vision Transformer","citations":47,"named_by":["acronym_expansion"],"bearer_name":"Contrastive Vision Transformer (CVT)","ask_with":"have('2207.13516')"},"n_other_bearers_listed":3,"note":"3 other thing(s) here are also called 'CvT', and none of them is cited more than the paper this answer is about. They are listed in `also_named` -- check `served.arxiv_id` is the paper you meant."},"name_competition_text":null,"near_matches":null,"code_nearby":[],"code_nearby_caption":null,"origin_paper":{"arxiv_id":"2103.15808","page":"/paper/cvt-introducing-convolutions-to-vision","syntology_url":"https://syntology.ai/paper/2103.15808","in_graph_layer":true,"samples":47,"ran":39,"n_ran_checked":36,"n_instrument":3,"unverified":8,"phrase":"39 ran (of which 19 constructed an object rather than computing a result; 36 with no instrument failure: 2 honoured, 0 violated, 34 with no contract checked; 3 where Syntology's instrument failed) · 8 unverified","instrument_means":"failures of Syntology's instrument, not of the code"},"line":"repository known, no verified implementation (declared_code; repositories: 5 listed of 16 in the graph, strongest link: official). This answer is about arXiv:2103.15808, the paper the archive credits with this method.","level_legend":null},"syntology_have_state":"loaded","record_sha256":"7b0935cca18c9795acc5754c1625fac205bedbf68cd470c8020af8c0f773b14b","record_changed_at":"2026-10-05","record_changed_at_basis":"changed"}