{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/nasvit-neural-architecture-search-for","title":"NASViT: Neural Architecture Search for Efficient Vision Transformers with Gradient Conflict aware Supernet Training","arxiv_id":null,"date":"2021-09-29","proceeding":"ICLR 2022 4","authors":["Chengyue Gong","Dilin Wang","Meng Li","Xinlei Chen","Zhicheng Yan","Yuandong Tian","Qiang Liu","Vikas Chandra"],"abstract":"Designing accurate and efficient vision transformers (ViTs) is a highly important but challenging task. Supernet-based one-shot neural architecture search (NAS) enables fast architecture optimization and has achieved state-of-the-art (SOTA) results on convolutional neural networks (CNNs). However, directly applying the supernet-based NAS to optimize ViTs leads to poor performance - even worse compared to training single ViTs. In this work, we observe that the poor performance is due to a gradient conflict issue: the gradients of different sub-networks conflict with that of the supernet more severely in ViTs than CNNs, which leads to early saturation in training and inferior convergence. To alleviate this issue, we propose a series of techniques, including a gradient projection algorithm, a switchable layer scaling design, and a simplified data augmentation and regularization training recipe. The proposed techniques significantly improve the convergence and the performance of all sub-networks. Our discovered hybrid ViT model family, dubbed NASViT, achieves top-1 accuracy from 78.2% to 81.8% on ImageNet from 200M to 800M FLOPs, and outperforms all the prior art CNNs and ViTs, including AlphaNet and LeViT, etc. When transferred to semantic segmentation tasks, NASViTs also outperform previous backbones on both Cityscape and ADE20K datasets, achieving 73.2% and 37.9% mIoU with only 5G FLOPs, respectively.\n","url_abs":"https://openreview.net/forum?id=Qaw16njk6L","url_pdf":"https://openreview.net/pdf?id=Qaw16njk6L","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"nasvit-neural-architecture-search-for","repo_url":"https://github.com/facebookresearch/NASViT","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"data-augmentation","task_name":"Data Augmentation"},{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"architecture-search","task_name":"Neural Architecture Search"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/image-classification-on-imagenet","task":"Image Classification","dataset":"ImageNet","model":"NASViT (supernet)","rank_in_archive_order":483,"of":1060,"metrics":{"GFLOPs":"1.881","Top 1 Accuracy":"82.9%"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-imagenet","task":"Image Classification","dataset":"ImageNet","model":"NASViT-A5","rank_in_archive_order":604,"of":1060,"metrics":{"GFLOPs":"0.757","Top 1 Accuracy":"81.8%"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-imagenet","task":"Image Classification","dataset":"ImageNet","model":"NASViT-A4","rank_in_archive_order":640,"of":1060,"metrics":{"GFLOPs":"0.591","Top 1 Accuracy":"81.4%"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-imagenet","task":"Image Classification","dataset":"ImageNet","model":"NASViT-A3","rank_in_archive_order":669,"of":1060,"metrics":{"GFLOPs":"0.528","Top 1 Accuracy":"81.0%"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-imagenet","task":"Image Classification","dataset":"ImageNet","model":"NASViT-A2","rank_in_archive_order":695,"of":1060,"metrics":{"GFLOPs":"0.421","Top 1 Accuracy":"80.5%"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-imagenet","task":"Image Classification","dataset":"ImageNet","model":"NASViT-A1","rank_in_archive_order":744,"of":1060,"metrics":{"GFLOPs":"0.309","Top 1 Accuracy":"79.7%"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-imagenet","task":"Image Classification","dataset":"ImageNet","model":"NASViT-A0","rank_in_archive_order":844,"of":1060,"metrics":{"GFLOPs":"0.208","Top 1 Accuracy":"78.2%"},"uses_additional_data":false},{"leaderboard":"/sota/neural-architecture-search-on-imagenet","task":"Neural Architecture Search","dataset":"ImageNet","model":"NASViT-A5","rank_in_archive_order":7,"of":135,"metrics":{"Accuracy":"81.8","FLOPs":"757M","Top-1 Error Rate":"18.2"},"uses_additional_data":false},{"leaderboard":"/sota/neural-architecture-search-on-imagenet","task":"Neural Architecture Search","dataset":"ImageNet","model":"NASViT-A4","rank_in_archive_order":10,"of":135,"metrics":{"Accuracy":"81.4","FLOPs":"591M","Top-1 Error Rate":"18.6"},"uses_additional_data":false},{"leaderboard":"/sota/neural-architecture-search-on-imagenet","task":"Neural Architecture Search","dataset":"ImageNet","model":"NASViT-A3","rank_in_archive_order":11,"of":135,"metrics":{"Accuracy":"81.0","FLOPs":"528M","Top-1 Error Rate":"19.0"},"uses_additional_data":false},{"leaderboard":"/sota/neural-architecture-search-on-imagenet","task":"Neural Architecture Search","dataset":"ImageNet","model":"NASViT-A2","rank_in_archive_order":16,"of":135,"metrics":{"Accuracy":"80.5","FLOPs":"421M","Top-1 Error Rate":"19.5"},"uses_additional_data":false},{"leaderboard":"/sota/neural-architecture-search-on-imagenet","task":"Neural Architecture Search","dataset":"ImageNet","model":"NASViT-A1","rank_in_archive_order":27,"of":135,"metrics":{"Accuracy":"79.7","FLOPs":"309M","Top-1 Error Rate":"20.3"},"uses_additional_data":false},{"leaderboard":"/sota/neural-architecture-search-on-imagenet","task":"Neural Architecture Search","dataset":"ImageNet","model":"NASViT-A0","rank_in_archive_order":51,"of":135,"metrics":{"Accuracy":"78.2","FLOPs":"208M","Top-1 Error Rate":"21.8"},"uses_additional_data":false}],"syntology":{"syntology_url":null,"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers","held":null,"reason":"no_arxiv_id","checked_against":null,"by_repo":[{"repo":"facebookresearch/NASViT","kind":"official","samples":0,"ran":0,"instrument":0,"unverified":0}],"by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":null,"arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"0ffd1911fa2eb9d06444cac235d8b000190d5a5fd6ab406951206a646604e419","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}