{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/scaling-vision-transformers-to-22-billion","title":"Scaling Vision Transformers to 22 Billion Parameters","arxiv_id":"2302.05442","date":"2023-02-10","proceeding":null,"authors":["Mostafa Dehghani","Josip Djolonga","Basil Mustafa","Piotr Padlewski","Jonathan Heek","Justin Gilmer","Andreas Steiner","Mathilde Caron","Robert Geirhos","Ibrahim Alabdulmohsin","Rodolphe Jenatton","Lucas Beyer","Michael Tschannen","Anurag Arnab","Xiao Wang","Carlos Riquelme","Matthias Minderer","Joan Puigcerver","Utku Evci","Manoj Kumar","Sjoerd van Steenkiste","Gamaleldin F. Elsayed","Aravindh Mahendran","Fisher Yu","Avital Oliver","Fantine Huot","Jasmijn Bastings","Mark Patrick Collier","Alexey Gritsenko","Vighnesh Birodkar","Cristina Vasconcelos","Yi Tay","Thomas Mensink","Alexander Kolesnikov","Filip Pavetić","Dustin Tran","Thomas Kipf","Mario Lučić","Xiaohua Zhai","Daniel Keysers","Jeremiah Harmsen","Neil Houlsby"],"abstract":"The scaling of Transformers has driven breakthrough capabilities for language models. At present, the largest large language models (LLMs) contain upwards of 100B parameters. Vision Transformers (ViT) have introduced the same architecture to image and video modelling, but these have not yet been successfully scaled to nearly the same degree; the largest dense ViT contains 4B parameters (Chen et al., 2022). We present a recipe for highly efficient and stable training of a 22B-parameter ViT (ViT-22B) and perform a wide variety of experiments on the resulting model. When evaluated on downstream tasks (often with a lightweight linear model on frozen features), ViT-22B demonstrates increasing performance with scale. We further observe other interesting benefits of scale, including an improved tradeoff between fairness and performance, state-of-the-art alignment to human visual perception in terms of shape/texture bias, and improved robustness. ViT-22B demonstrates the potential for \"LLM-like\" scaling in vision, and provides key steps towards getting there.","url_abs":"https://arxiv.org/abs/2302.05442v1","url_pdf":"https://arxiv.org/pdf/2302.05442v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"scaling-vision-transformers-to-22-billion","repo_url":"https://github.com/lucidrains/flash-cosine-sim-attention","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"action-classification","task_name":"Action Classification"},{"task_slug":"fairness","task_name":"Fairness"},{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"linear-probe-classification","task_name":"Linear-Probe Classification"},{"task_slug":"object-recognition","task_name":"Object Recognition"},{"task_slug":"zero-shot-transfer-image-classification","task_name":"Zero-Shot Transfer Image Classification"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/action-classification-on-kinetics-400","task":"Action Classification","dataset":"Kinetics-400","model":"ViT-22B","rank_in_archive_order":29,"of":207,"metrics":{"Acc@1":"88.0"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-imagenet","task":"Image Classification","dataset":"ImageNet","model":"ViT-L/16 (384res, distilled from ViT-22B)","rank_in_archive_order":15,"of":1060,"metrics":{"Number of params":"307M","Top 1 Accuracy":"89.6%"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-imagenet","task":"Image Classification","dataset":"ImageNet","model":"ViT-B/16","rank_in_archive_order":36,"of":1060,"metrics":{"Number of params":"86M","Top 1 Accuracy":"88.6%"},"uses_additional_data":false},{"leaderboard":"/sota/object-recognition-on-shape-bias","task":"Object Recognition","dataset":"shape bias","model":"ViT-22B-384","rank_in_archive_order":4,"of":18,"metrics":{"shape bias":"86.4"},"uses_additional_data":false},{"leaderboard":"/sota/object-recognition-on-shape-bias","task":"Object Recognition","dataset":"shape bias","model":"ViT-22B-560","rank_in_archive_order":5,"of":18,"metrics":{"shape bias":"83.8"},"uses_additional_data":false},{"leaderboard":"/sota/object-recognition-on-shape-bias","task":"Object Recognition","dataset":"shape bias","model":"ViT-22B-224","rank_in_archive_order":7,"of":18,"metrics":{"shape bias":"78.0"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-transfer-image-classification-on-1","task":"Zero-Shot Transfer Image Classification","dataset":"ImageNet","model":"LiT-22B","rank_in_archive_order":4,"of":23,"metrics":{"Accuracy (Private)":"85.9"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-transfer-image-classification-on-3","task":"Zero-Shot Transfer Image Classification","dataset":"ImageNet V2","model":"LiT-22B","rank_in_archive_order":2,"of":13,"metrics":{"Accuracy (Private)":"80.9"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-transfer-image-classification-on-5","task":"Zero-Shot Transfer Image Classification","dataset":"ImageNet-A","model":"LiT-22B","rank_in_archive_order":2,"of":13,"metrics":{"Accuracy (Private)":"90.1"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-transfer-image-classification-on-4","task":"Zero-Shot Transfer Image Classification","dataset":"ImageNet-R","model":"LiT-22B","rank_in_archive_order":4,"of":12,"metrics":{"Accuracy":"96.0"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-transfer-image-classification-on-6","task":"Zero-Shot Transfer Image Classification","dataset":"ObjectNet","model":"LiT-22B","rank_in_archive_order":1,"of":9,"metrics":{"Accuracy (Private)":"87.6"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2302.05442","atlas_url":"https://app.syntology.ai/?focus=2302.05442","mcp":null,"developers":"https://syntology.ai/developers","held":true,"reason":"arxiv_id_in_atlas","checked_against":{"atlas_ids":549449,"atlas_ids_sha256":"afd72399dafd9906e9062927f70655952385a3d75488558e1f0484e950ab09b7","read_at":"2026-10-05T01:46:21Z"},"agent_calls":[{"tool":"get_citation_path","arguments":{"paper_1":"2302.05442"},"arguments_in":{"paper_2":"another paper's arXiv id or title"},"call":"get_citation_path(paper_1=\"2302.05442\", paper_2=\"…\")"},{"tool":"get_concepts_for_paper","arguments":{"arxiv_id":"2302.05442"},"call":"get_concepts_for_paper(arxiv_id=\"2302.05442\")"}],"by_repo":[{"repo":"lucidrains/flash-cosine-sim-attention","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0,"harvested_for_other_papers":2,"harvested_for":["2112.05682","2205.14135"],"harvested_for_calls":[{"arxiv_id":"2112.05682","tool":"get_harvested_code_for_paper","arguments":{"arxiv_id":"2112.05682"},"call":"get_harvested_code_for_paper(arxiv_id=\"2112.05682\")"},{"arxiv_id":"2205.14135","tool":"get_harvested_code_for_paper","arguments":{"arxiv_id":"2205.14135"},"call":"get_harvested_code_for_paper(arxiv_id=\"2205.14135\")"}]}],"by_repo_harvested_for_is":"harvested_for: the arXiv ids (as Syntology's graph holds them) of every paper whose samples Syntology's graph links to this repository, in arXiv-id order, complete; the page names the first 3 and says how many more. A link only: it does not say the papers are the same work, that one copied the other, or that the code implements this paper's method; harvested_for_calls gives the get_harvested_code_for_paper call for each arXiv-shaped id among them","by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":null,"arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"483d01f9987d5346f62c72e249eede0c15c7409a0e1cc70533414cd1d8f5e2ca","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}