{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/deit-lt-distillation-strikes-back-for-vision","title":"DeiT-LT Distillation Strikes Back for Vision Transformer Training on Long-Tailed Datasets","arxiv_id":"2404.02900","date":"2024-04-03","proceeding":null,"authors":["Harsh Rangwani","Pradipto Mondal","Mayank Mishra","Ashish Ramayee Asokan","R. Venkatesh Babu"],"abstract":"Vision Transformer (ViT) has emerged as a prominent architecture for various computer vision tasks. In ViT, we divide the input image into patch tokens and process them through a stack of self attention blocks. However, unlike Convolutional Neural Networks (CNN), ViTs simple architecture has no informative inductive bias (e.g., locality,etc. ). Due to this, ViT requires a large amount of data for pre-training. Various data efficient approaches (DeiT) have been proposed to train ViT on balanced datasets effectively. However, limited literature discusses the use of ViT for datasets with long-tailed imbalances. In this work, we introduce DeiT-LT to tackle the problem of training ViTs from scratch on long-tailed datasets. In DeiT-LT, we introduce an efficient and effective way of distillation from CNN via distillation DIST token by using out-of-distribution images and re-weighting the distillation loss to enhance focus on tail classes. This leads to the learning of local CNN-like features in early ViT blocks, improving generalization for tail classes. Further, to mitigate overfitting, we propose distilling from a flat CNN teacher, which leads to learning low-rank generalizable features for DIST tokens across all ViT blocks. With the proposed DeiT-LT scheme, the distillation DIST token becomes an expert on the tail classes, and the classifier CLS token becomes an expert on the head classes. The experts help to effectively learn features corresponding to both the majority and minority classes using a distinct set of tokens within the same ViT architecture. We show the effectiveness of DeiT-LT for training ViT from scratch on datasets ranging from small-scale CIFAR-10 LT to large-scale iNaturalist-2018.","url_abs":"https://arxiv.org/abs/2404.02900v1","url_pdf":"https://arxiv.org/pdf/2404.02900v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"deit-lt-distillation-strikes-back-for-vision","repo_url":"https://github.com/val-iisc/DeiT-LT","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"deit-lt-distillation-strikes-back-for-vision","repo_url":"https://github.com/pwc-1/Paper-8/tree/main/deit","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":null}],"tasks":[{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"inductive-bias","task_name":"Inductive Bias"},{"task_slug":"long-tail-learning","task_name":"Long-tail Learning"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"focus","method_name":"Focus"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"set","method_name":"SET"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/image-classification-on-inaturalist","task":"Image Classification","dataset":"iNaturalist","model":"b_22DeiT-LT(ours)","rank_in_archive_order":19,"of":19,"metrics":{"Overall":"75.1"},"uses_additional_data":false},{"leaderboard":"/sota/long-tail-learning-on-cifar-10-lt-r-100","task":"Long-tail Learning","dataset":"CIFAR-10-LT (ρ=100)","model":"DeiT-LT","rank_in_archive_order":5,"of":28,"metrics":{"Error Rate":"12.5"},"uses_additional_data":false},{"leaderboard":"/sota/long-tail-learning-on-cifar-10-lt-r-50","task":"Long-tail Learning","dataset":"CIFAR-10-LT (ρ=50)","model":"DeiT-LT","rank_in_archive_order":3,"of":8,"metrics":{"Error Rate":"10.2"},"uses_additional_data":false},{"leaderboard":"/sota/long-tail-learning-on-cifar-100-lt-r-100","task":"Long-tail Learning","dataset":"CIFAR-100-LT (ρ=100)","model":"DeiT-LT","rank_in_archive_order":11,"of":66,"metrics":{"Error Rate":"44.4"},"uses_additional_data":false},{"leaderboard":"/sota/long-tail-learning-on-cifar-100-lt-r-50","task":"Long-tail Learning","dataset":"CIFAR-100-LT (ρ=50)","model":"DeiT-LT","rank_in_archive_order":8,"of":25,"metrics":{"Error Rate":"39.5"},"uses_additional_data":false},{"leaderboard":"/sota/long-tail-learning-on-imagenet-lt","task":"Long-tail Learning","dataset":"ImageNet-LT","model":"DeiT-LT","rank_in_archive_order":16,"of":69,"metrics":{"Top-1 Accuracy":"59.1"},"uses_additional_data":false},{"leaderboard":"/sota/long-tail-learning-on-inaturalist-2018","task":"Long-tail Learning","dataset":"iNaturalist 2018","model":"DeiT-LT","rank_in_archive_order":15,"of":43,"metrics":{"Top-1 Accuracy":"75.1%"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2404.02900","atlas_url":"https://app.syntology.ai/?focus=2404.02900","mcp":null,"developers":"https://syntology.ai/developers","held":true,"reason":"arxiv_id_in_atlas","checked_against":{"atlas_ids":549449,"atlas_ids_sha256":"afd72399dafd9906e9062927f70655952385a3d75488558e1f0484e950ab09b7","read_at":"2026-10-05T01:46:21Z"},"agent_calls":[{"tool":"get_citation_path","arguments":{"paper_1":"2404.02900"},"arguments_in":{"paper_2":"another paper's arXiv id or title"},"call":"get_citation_path(paper_1=\"2404.02900\", paper_2=\"…\")"},{"tool":"get_concepts_for_paper","arguments":{"arxiv_id":"2404.02900"},"call":"get_concepts_for_paper(arxiv_id=\"2404.02900\")"}],"by_repo":[{"repo":"val-iisc/DeiT-LT","kind":"official","samples":0,"ran":0,"instrument":0,"unverified":0},{"repo":"pwc-1/Paper-8","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0,"harvested_for_other_papers":1,"harvested_for":["2406.18321"],"harvested_for_calls":[{"arxiv_id":"2406.18321","tool":"get_harvested_code_for_paper","arguments":{"arxiv_id":"2406.18321"},"call":"get_harvested_code_for_paper(arxiv_id=\"2406.18321\")"}]}],"by_repo_harvested_for_is":"harvested_for: the arXiv ids (as Syntology's graph holds them) of every paper whose samples Syntology's graph links to this repository, in arXiv-id order, complete; the page names the first 3 and says how many more. A link only: it does not say the papers are the same work, that one copied the other, or that the code implements this paper's method; harvested_for_calls gives the get_harvested_code_for_paper call for each arXiv-shaped id among them","by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":null,"arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"b6b1a8bf3df16e99ea14e0f7e60172754e920296d474a9622c5a2fed1596f816","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}