{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/self-supervised-learning-with-swin","title":"Self-Supervised Learning with Swin Transformers","arxiv_id":"2105.04553","date":"2021-05-10","proceeding":null,"authors":["Zhenda Xie","Yutong Lin","Zhuliang Yao","Zheng Zhang","Qi Dai","Yue Cao","Han Hu"],"abstract":"We are witnessing a modeling shift from CNN to Transformers in computer vision. In this work, we present a self-supervised learning approach called MoBY, with Vision Transformers as its backbone architecture. The approach basically has no new inventions, which is combined from MoCo v2 and BYOL and tuned to achieve reasonably high accuracy on ImageNet-1K linear evaluation: 72.8% and 75.0% top-1 accuracy using DeiT-S and Swin-T, respectively, by 300-epoch training. The performance is slightly better than recent works of MoCo v3 and DINO which adopt DeiT as the backbone, but with much lighter tricks. More importantly, the general-purpose Swin Transformer backbone enables us to also evaluate the learnt representations on downstream tasks such as object detection and semantic segmentation, in contrast to a few recent approaches built on ViT/DeiT which only report linear evaluation results on ImageNet-1K due to ViT/DeiT not tamed for these dense prediction tasks. We hope our results can facilitate more comprehensive evaluation of self-supervised learning methods designed for Transformer architectures. Our code and models are available at https://github.com/SwinTransformer/Transformer-SSL, which will be continually enriched.","url_abs":"https://arxiv.org/abs/2105.04553v2","url_pdf":"https://arxiv.org/pdf/2105.04553v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"self-supervised-learning-with-swin","repo_url":"https://github.com/SwinTransformer/Transformer-SSL","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"self-supervised-learning-with-swin","repo_url":"https://github.com/2023-MindSpore-4/Code10/tree/main/SwinTransformer","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":null},{"paper_slug":"self-supervised-learning-with-swin","repo_url":"https://github.com/2023-MindSpore-4/Code9/tree/main/SwinTransformer","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":null},{"paper_slug":"self-supervised-learning-with-swin","repo_url":"https://github.com/2024-MindSpore-1/Code4/tree/main/SwinTransformer","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":null},{"paper_slug":"self-supervised-learning-with-swin","repo_url":"https://github.com/alibaba/EasyCV","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":null},{"paper_slug":"self-supervised-learning-with-swin","repo_url":"https://github.com/microsoft/Swin-Transformer","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"linear-evaluation","task_name":"Linear evaluation"},{"task_slug":"object-detection","task_name":"Object Detection"},{"task_slug":"self-supervised-image-classification","task_name":"Self-Supervised Image Classification"},{"task_slug":"self-supervised-learning","task_name":"Self-Supervised Learning"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"},{"task_slug":"object-detection-1","task_name":"object-detection"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"adamw","method_name":"AdamW"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"byol","method_name":"BYOL"},{"method_slug":"batch-normalization","method_name":"Batch Normalization"},{"method_slug":"deit","method_name":"DeiT"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"droppath","method_name":"DropPath"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"feedforward-network","method_name":"Feedforward Network"},{"method_slug":"infonce","method_name":"InfoNCE"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"moby","method_name":"MoBY"},{"method_slug":"moco","method_name":"MoCo"},{"method_slug":"moco-v2","method_name":"MoCo v2"},{"method_slug":"moco-v3","method_name":"MoCo v3"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"random-gaussian-blur","method_name":"Random Gaussian Blur"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"stochastic-depth","method_name":"Stochastic Depth"},{"method_slug":"swin-transformer","method_name":"Swin Transformer"},{"method_slug":"transformer","method_name":"Transformer"},{"method_slug":"vision-transformer","method_name":"Vision Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/self-supervised-image-classification-on","task":"Self-Supervised Image Classification","dataset":"ImageNet","model":"MoBY (Swin-T)","rank_in_archive_order":79,"of":144,"metrics":{"Number of Params":"29M","Top 1 Accuracy":"75%"},"uses_additional_data":false},{"leaderboard":"/sota/self-supervised-image-classification-on","task":"Self-Supervised Image Classification","dataset":"ImageNet","model":"MoBY (DeiT-S)","rank_in_archive_order":94,"of":144,"metrics":{"Number of Params":"22M","Top 1 Accuracy":"72.8%"},"uses_additional_data":false},{"leaderboard":"/sota/semantic-segmentation-on-ade20k","task":"Semantic Segmentation","dataset":"ADE20K","model":"Swin-T (UPerNet) MoBY","rank_in_archive_order":189,"of":235,"metrics":{"Validation mIoU":"45.58"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2105.04553","atlas_url":"https://app.syntology.ai/?focus=2105.04553","mcp":null,"developers":"https://syntology.ai/developers","held":true,"reason":"arxiv_id_in_atlas","checked_against":{"atlas_ids":549449,"atlas_ids_sha256":"afd72399dafd9906e9062927f70655952385a3d75488558e1f0484e950ab09b7","read_at":"2026-10-05T01:46:21Z"},"agent_calls":[{"tool":"get_citation_path","arguments":{"paper_1":"2105.04553"},"arguments_in":{"paper_2":"another paper's arXiv id or title"},"call":"get_citation_path(paper_1=\"2105.04553\", paper_2=\"…\")"},{"tool":"get_concepts_for_paper","arguments":{"arxiv_id":"2105.04553"},"call":"get_concepts_for_paper(arxiv_id=\"2105.04553\")"}],"by_repo":[{"repo":"SwinTransformer/Transformer-SSL","kind":"official","samples":0,"ran":0,"instrument":0,"unverified":0,"harvested_for_other_papers":1,"harvested_for":["2103.14030"],"harvested_for_calls":[{"arxiv_id":"2103.14030","tool":"get_harvested_code_for_paper","arguments":{"arxiv_id":"2103.14030"},"call":"get_harvested_code_for_paper(arxiv_id=\"2103.14030\")"}]},{"repo":"2023-MindSpore-4/Code10","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0},{"repo":"2023-MindSpore-4/Code9","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0},{"repo":"2024-MindSpore-1/Code4","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0,"harvested_for_other_papers":1,"harvested_for":["2305.05773"],"harvested_for_calls":[{"arxiv_id":"2305.05773","tool":"get_harvested_code_for_paper","arguments":{"arxiv_id":"2305.05773"},"call":"get_harvested_code_for_paper(arxiv_id=\"2305.05773\")"}]},{"repo":"alibaba/EasyCV","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0,"harvested_for_other_papers":4,"harvested_for":["2005.12872","2012.12877","2111.06377","2203.01305"],"harvested_for_calls":[{"arxiv_id":"2005.12872","tool":"get_harvested_code_for_paper","arguments":{"arxiv_id":"2005.12872"},"call":"get_harvested_code_for_paper(arxiv_id=\"2005.12872\")"},{"arxiv_id":"2012.12877","tool":"get_harvested_code_for_paper","arguments":{"arxiv_id":"2012.12877"},"call":"get_harvested_code_for_paper(arxiv_id=\"2012.12877\")"},{"arxiv_id":"2111.06377","tool":"get_harvested_code_for_paper","arguments":{"arxiv_id":"2111.06377"},"call":"get_harvested_code_for_paper(arxiv_id=\"2111.06377\")"},{"arxiv_id":"2203.01305","tool":"get_harvested_code_for_paper","arguments":{"arxiv_id":"2203.01305"},"call":"get_harvested_code_for_paper(arxiv_id=\"2203.01305\")"}]},{"repo":"microsoft/Swin-Transformer","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0,"harvested_for_other_papers":1,"harvested_for":["2103.14030"],"harvested_for_calls":[{"arxiv_id":"2103.14030","tool":"get_harvested_code_for_paper","arguments":{"arxiv_id":"2103.14030"},"call":"get_harvested_code_for_paper(arxiv_id=\"2103.14030\")"}]}],"by_repo_harvested_for_is":"harvested_for: the arXiv ids (as Syntology's graph holds them) of every paper whose samples Syntology's graph links to this repository, in arXiv-id order, complete; the page names the first 3 and says how many more. A link only: it does not say the papers are the same work, that one copied the other, or that the code implements this paper's method; harvested_for_calls gives the get_harvested_code_for_paper call for each arXiv-shaped id among them","by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":null,"arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"460a42f5b4f7dbfaacc8749b94fbd4f8cd3a417ff2cbc1e094d79b775c59b372","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}