{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/contextual-transformer-networks-for-visual","title":"Contextual Transformer Networks for Visual Recognition","arxiv_id":"2107.12292","date":"2021-07-26","proceeding":null,"authors":["Yehao Li","Ting Yao","Yingwei Pan","Tao Mei"],"abstract":"Transformer with self-attention has led to the revolutionizing of natural language processing field, and recently inspires the emergence of Transformer-style architecture design with competitive results in numerous computer vision tasks. Nevertheless, most of existing designs directly employ self-attention over a 2D feature map to obtain the attention matrix based on pairs of isolated queries and keys at each spatial location, but leave the rich contexts among neighbor keys under-exploited. In this work, we design a novel Transformer-style module, i.e., Contextual Transformer (CoT) block, for visual recognition. Such design fully capitalizes on the contextual information among input keys to guide the learning of dynamic attention matrix and thus strengthens the capacity of visual representation. Technically, CoT block first contextually encodes input keys via a $3\\times3$ convolution, leading to a static contextual representation of inputs. We further concatenate the encoded keys with input queries to learn the dynamic multi-head attention matrix through two consecutive $1\\times1$ convolutions. The learnt attention matrix is multiplied by input values to achieve the dynamic contextual representation of inputs. The fusion of the static and dynamic contextual representations are finally taken as outputs. Our CoT block is appealing in the view that it can readily replace each $3\\times3$ convolution in ResNet architectures, yielding a Transformer-style backbone named as Contextual Transformer Networks (CoTNet). Through extensive experiments over a wide range of applications (e.g., image recognition, object detection and instance segmentation), we validate the superiority of CoTNet as a stronger backbone. Source code is available at \\url{https://github.com/JDAI-CV/CoTNet}.","url_abs":"https://arxiv.org/abs/2107.12292v1","url_pdf":"https://arxiv.org/pdf/2107.12292v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"contextual-transformer-networks-for-visual","repo_url":"https://github.com/yehli/imagenetmodel","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"contextual-transformer-networks-for-visual","repo_url":"https://github.com/JDAI-CV/CoTNet","is_official":0,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}},{"paper_slug":"contextual-transformer-networks-for-visual","repo_url":"https://github.com/JDAI-CV/CoTNet-ObjectDetection-InstanceSegmentation","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}},{"paper_slug":"contextual-transformer-networks-for-visual","repo_url":"https://github.com/xmu-xiaoma666/External-Attention-pytorch","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"contextual-transformer-networks-for-visual","repo_url":"https://github.com/zhaozsq/airway_segmentation","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"contextual-transformer-networks-for-visual","repo_url":"https://github.com/leondgarse/keras_cv_attention_models/tree/main/keras_cv_attention_models/cotnet","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"tf","reach":null},{"paper_slug":"contextual-transformer-networks-for-visual","repo_url":"https://github.com/mindspore-courses/External-Attention-MindSpore/blob/main/model/attention/CoTAttention.py","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":null}],"tasks":[{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"instance-segmentation","task_name":"Instance Segmentation"},{"task_slug":"object-detection","task_name":"Object Detection"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"},{"task_slug":"object-detection-1","task_name":"object-detection"}],"methods":[{"method_slug":"1x1-convolution","method_name":"1x1 Convolution"},{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"average-pooling","method_name":"Average Pooling"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"batch-normalization","method_name":"Batch Normalization"},{"method_slug":"bottleneck-residual-block","method_name":"Bottleneck Residual Block"},{"method_slug":"convolution","method_name":"Convolution"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"global-average-pooling","method_name":"Global Average Pooling"},{"method_slug":"kaiming-initialization","method_name":"Kaiming Initialization"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"max-pooling","method_name":"Max Pooling"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"relu","method_name":"ReLU"},{"method_slug":"residual-block","method_name":"Residual Block"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/image-classification-on-imagenet","task":"Image Classification","dataset":"ImageNet","model":"SE-CoTNetD-152","rank_in_archive_order":304,"of":1060,"metrics":{"GFLOPs":"26.5","Number of params":"55.8M","Top 1 Accuracy":"84.6%"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-imagenet","task":"Image Classification","dataset":"ImageNet","model":"SE-CoTNetD-101","rank_in_archive_order":455,"of":1060,"metrics":{"GFLOPs":"8.5","Number of params":"40.9M","Top 1 Accuracy":"83.2%"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-imagenet","task":"Image Classification","dataset":"ImageNet","model":"SE-CoTNetD-50","rank_in_archive_order":626,"of":1060,"metrics":{"GFLOPs":"4.1","Number of params":"23.1M","Top 1 Accuracy":"81.6%"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2107.12292","atlas_url":"https://app.syntology.ai/?focus=2107.12292","mcp":null,"developers":"https://syntology.ai/developers","held":true,"reason":"arxiv_id_in_atlas","checked_against":{"atlas_ids":549449,"atlas_ids_sha256":"afd72399dafd9906e9062927f70655952385a3d75488558e1f0484e950ab09b7","read_at":"2026-10-05T01:46:21Z"},"agent_calls":[{"tool":"get_citation_path","arguments":{"paper_1":"2107.12292"},"arguments_in":{"paper_2":"another paper's arXiv id or title"},"call":"get_citation_path(paper_1=\"2107.12292\", paper_2=\"…\")"},{"tool":"get_concepts_for_paper","arguments":{"arxiv_id":"2107.12292"},"call":"get_concepts_for_paper(arxiv_id=\"2107.12292\")"}],"by_repo":[{"repo":"yehli/imagenetmodel","kind":"official","samples":0,"ran":0,"instrument":0,"unverified":0,"harvested_for_other_papers":1,"harvested_for":["2207.04978"],"harvested_for_calls":[{"arxiv_id":"2207.04978","tool":"get_harvested_code_for_paper","arguments":{"arxiv_id":"2207.04978"},"call":"get_harvested_code_for_paper(arxiv_id=\"2207.04978\")"}]},{"repo":"JDAI-CV/CoTNet","kind":"named_in_paper","samples":0,"ran":0,"instrument":0,"unverified":0},{"repo":"JDAI-CV/CoTNet-ObjectDetection-InstanceSegmentation","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0},{"repo":"xmu-xiaoma666/External-Attention-pytorch","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0,"harvested_for_other_papers":7,"harvested_for":["1706.03762","2101.03697","2103.06255","2103.12731","2105.01601","2105.08050","2110.02178"],"harvested_for_calls":[{"arxiv_id":"1706.03762","tool":"get_harvested_code_for_paper","arguments":{"arxiv_id":"1706.03762"},"call":"get_harvested_code_for_paper(arxiv_id=\"1706.03762\")"},{"arxiv_id":"2101.03697","tool":"get_harvested_code_for_paper","arguments":{"arxiv_id":"2101.03697"},"call":"get_harvested_code_for_paper(arxiv_id=\"2101.03697\")"},{"arxiv_id":"2103.06255","tool":"get_harvested_code_for_paper","arguments":{"arxiv_id":"2103.06255"},"call":"get_harvested_code_for_paper(arxiv_id=\"2103.06255\")"},{"arxiv_id":"2103.12731","tool":"get_harvested_code_for_paper","arguments":{"arxiv_id":"2103.12731"},"call":"get_harvested_code_for_paper(arxiv_id=\"2103.12731\")"},{"arxiv_id":"2105.01601","tool":"get_harvested_code_for_paper","arguments":{"arxiv_id":"2105.01601"},"call":"get_harvested_code_for_paper(arxiv_id=\"2105.01601\")"},{"arxiv_id":"2105.08050","tool":"get_harvested_code_for_paper","arguments":{"arxiv_id":"2105.08050"},"call":"get_harvested_code_for_paper(arxiv_id=\"2105.08050\")"},{"arxiv_id":"2110.02178","tool":"get_harvested_code_for_paper","arguments":{"arxiv_id":"2110.02178"},"call":"get_harvested_code_for_paper(arxiv_id=\"2110.02178\")"}]},{"repo":"zhaozsq/airway_segmentation","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0},{"repo":"leondgarse/keras_cv_attention_models","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0,"harvested_for_other_papers":1,"harvested_for":["2307.09283"],"harvested_for_calls":[{"arxiv_id":"2307.09283","tool":"get_harvested_code_for_paper","arguments":{"arxiv_id":"2307.09283"},"call":"get_harvested_code_for_paper(arxiv_id=\"2307.09283\")"}]},{"repo":"mindspore-courses/External-Attention-MindSpore","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0}],"by_repo_harvested_for_is":"harvested_for: the arXiv ids (as Syntology's graph holds them) of every paper whose samples Syntology's graph links to this repository, in arXiv-id order, complete; the page names the first 3 and says how many more. A link only: it does not say the papers are the same work, that one copied the other, or that the code implements this paper's method; harvested_for_calls gives the get_harvested_code_for_paper call for each arXiv-shaped id among them","by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":null,"arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"dc79ee9da830f89bff578a54bc95a251da126e55950a31ff733ba6850e25b1cb","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}