{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/tune-it-or-don-t-use-it-benchmarking-data","title":"Tune It or Don't Use It: Benchmarking Data-Efficient Image Classification","arxiv_id":"2108.13122","date":"2021-08-30","proceeding":null,"authors":["Lorenzo Brigato","Björn Barz","Luca Iocchi","Joachim Denzler"],"abstract":"Data-efficient image classification using deep neural networks in settings, where only small amounts of labeled data are available, has been an active research area in the recent past. However, an objective comparison between published methods is difficult, since existing works use different datasets for evaluation and often compare against untuned baselines with default hyper-parameters. We design a benchmark for data-efficient image classification consisting of six diverse datasets spanning various domains (e.g., natural images, medical imagery, satellite data) and data types (RGB, grayscale, multispectral). Using this benchmark, we re-evaluate the standard cross-entropy baseline and eight methods for data-efficient deep learning published between 2017 and 2021 at renowned venues. For a fair and realistic comparison, we carefully tune the hyper-parameters of all methods on each dataset. Surprisingly, we find that tuning learning rate, weight decay, and batch size on a separate validation split results in a highly competitive baseline, which outperforms all but one specialized method and performs competitively to the remaining one.","url_abs":"https://arxiv.org/abs/2108.13122v1","url_pdf":"https://arxiv.org/pdf/2108.13122v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"tune-it-or-don-t-use-it-benchmarking-data","repo_url":"https://github.com/cvjena/deic","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"benchmarking","task_name":"Benchmarking"},{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"small-data","task_name":"Small Data Image Classification"},{"task_slug":"image-classification","task_name":"image-classification"}],"methods":[],"datasets_introduced":[{"slug":"deic-benchmark","name":"DEIC Benchmark","full_name":"Data-Efficient Image Classification Benchmark"},{"slug":"imagenet-50-samples-per-class","name":"ImageNet 50 samples per class","full_name":""}],"methods_introduced":[],"results":[{"leaderboard":"/sota/small-data-on-cub-200-2011-30-samples-per-1","task":"Small Data Image Classification","dataset":"CUB-200-2011, 30 samples per class","model":"Harmonic Networks (no pre-training)","rank_in_archive_order":2,"of":4,"metrics":{"Accuracy":"72.26"},"uses_additional_data":false},{"leaderboard":"/sota/small-data-on-cub-200-2011-30-samples-per-1","task":"Small Data Image Classification","dataset":"CUB-200-2011, 30 samples per class","model":"Cross-entropy baseline (no pre-training)","rank_in_archive_order":3,"of":4,"metrics":{"Accuracy":"71.44"},"uses_additional_data":false},{"leaderboard":"/sota/small-data-on-cub-200-2011-30-samples-per-1","task":"Small Data Image Classification","dataset":"CUB-200-2011, 30 samples per class","model":"DSK Networks (no pre-training)","rank_in_archive_order":4,"of":4,"metrics":{"Accuracy":"71.02"},"uses_additional_data":false},{"leaderboard":"/sota/small-data-image-classification-on-deic","task":"Small Data Image Classification","dataset":"DEIC Benchmark","model":"Harmonic Networks","rank_in_archive_order":1,"of":10,"metrics":{"Average Balanced Accuracy (across datasets)":"68.70"},"uses_additional_data":false},{"leaderboard":"/sota/small-data-image-classification-on-deic","task":"Small Data Image Classification","dataset":"DEIC Benchmark","model":"Cross-Entropy baseline","rank_in_archive_order":2,"of":10,"metrics":{"Average Balanced Accuracy (across datasets)":"67.90"},"uses_additional_data":false},{"leaderboard":"/sota/small-data-image-classification-on-deic","task":"Small Data Image Classification","dataset":"DEIC Benchmark","model":"Cosine + Cross-Entropy Loss","rank_in_archive_order":3,"of":10,"metrics":{"Average Balanced Accuracy (across datasets)":"64.92"},"uses_additional_data":false},{"leaderboard":"/sota/small-data-image-classification-on-deic","task":"Small Data Image Classification","dataset":"DEIC Benchmark","model":"T-vMF Similarity","rank_in_archive_order":4,"of":10,"metrics":{"Average Balanced Accuracy (across datasets)":"64.67"},"uses_additional_data":false},{"leaderboard":"/sota/small-data-image-classification-on-deic","task":"Small Data Image Classification","dataset":"DEIC Benchmark","model":"DSK Networks","rank_in_archive_order":5,"of":10,"metrics":{"Average Balanced Accuracy (across datasets)":"64.64"},"uses_additional_data":false},{"leaderboard":"/sota/small-data-image-classification-on-deic","task":"Small Data Image Classification","dataset":"DEIC Benchmark","model":"OLÉ","rank_in_archive_order":6,"of":10,"metrics":{"Average Balanced Accuracy (across datasets)":"64.15"},"uses_additional_data":false},{"leaderboard":"/sota/small-data-image-classification-on-deic","task":"Small Data Image Classification","dataset":"DEIC Benchmark","model":"Cosine Loss","rank_in_archive_order":7,"of":10,"metrics":{"Average Balanced Accuracy (across datasets)":"62.73"},"uses_additional_data":false},{"leaderboard":"/sota/small-data-image-classification-on-deic","task":"Small Data Image Classification","dataset":"DEIC Benchmark","model":"Full Convolution","rank_in_archive_order":8,"of":10,"metrics":{"Average Balanced Accuracy (across datasets)":"62.06"},"uses_additional_data":false},{"leaderboard":"/sota/small-data-image-classification-on-deic","task":"Small Data Image Classification","dataset":"DEIC Benchmark","model":"Deep Hybrid Networks","rank_in_archive_order":9,"of":10,"metrics":{"Average Balanced Accuracy (across datasets)":"60.33"},"uses_additional_data":false},{"leaderboard":"/sota/small-data-image-classification-on-deic","task":"Small Data Image Classification","dataset":"DEIC Benchmark","model":"Grad-l2 Penalty","rank_in_archive_order":10,"of":10,"metrics":{"Average Balanced Accuracy (across datasets)":"55.47"},"uses_additional_data":false},{"leaderboard":"/sota/small-data-image-classification-on-eurosat-50","task":"Small Data Image Classification","dataset":"EuroSAT 50 samples per class","model":"Harmonic Networks","rank_in_archive_order":1,"of":3,"metrics":{"Accuracy":"92.09"},"uses_additional_data":false},{"leaderboard":"/sota/small-data-image-classification-on-eurosat-50","task":"Small Data Image Classification","dataset":"EuroSAT 50 samples per class","model":"DSK Networks","rank_in_archive_order":2,"of":3,"metrics":{"Accuracy":"91.25"},"uses_additional_data":false},{"leaderboard":"/sota/small-data-image-classification-on-eurosat-50","task":"Small Data Image Classification","dataset":"EuroSAT 50 samples per class","model":"Deep Hybrid Networks","rank_in_archive_order":3,"of":3,"metrics":{"Accuracy":"91.15"},"uses_additional_data":false},{"leaderboard":"/sota/small-data-image-classification-on-imagenet","task":"Small Data Image Classification","dataset":"ImageNet 50 samples per class","model":"Harmonic Networks","rank_in_archive_order":1,"of":3,"metrics":{"1:1 Accuracy":"46.36"},"uses_additional_data":false},{"leaderboard":"/sota/small-data-image-classification-on-imagenet","task":"Small Data Image Classification","dataset":"ImageNet 50 samples per class","model":"DSK Networks","rank_in_archive_order":2,"of":3,"metrics":{"1:1 Accuracy":"45.21"},"uses_additional_data":false},{"leaderboard":"/sota/small-data-image-classification-on-imagenet","task":"Small Data Image Classification","dataset":"ImageNet 50 samples per class","model":"Cross-entropy baseline","rank_in_archive_order":3,"of":3,"metrics":{"1:1 Accuracy":"44.97"},"uses_additional_data":false},{"leaderboard":"/sota/small-data-image-classification-on-cifair-10-1","task":"Small Data Image Classification","dataset":"ciFAIR-10 50 samples per class","model":"Cross-entropy baseline","rank_in_archive_order":3,"of":5,"metrics":{"Accuracy":"58.22"},"uses_additional_data":false},{"leaderboard":"/sota/small-data-image-classification-on-cifair-10-1","task":"Small Data Image Classification","dataset":"ciFAIR-10 50 samples per class","model":"T-vMF Similarity","rank_in_archive_order":4,"of":5,"metrics":{"Accuracy":"57.50"},"uses_additional_data":false},{"leaderboard":"/sota/small-data-image-classification-on-cifair-10-1","task":"Small Data Image Classification","dataset":"ciFAIR-10 50 samples per class","model":"Harmonic Networks","rank_in_archive_order":5,"of":5,"metrics":{"Accuracy":"56.50"},"uses_additional_data":false}],"syntology":{"syntology_url":null,"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers","held":null,"reason":"arxiv_id_not_in_atlas","checked_against":{"atlas_ids":549449,"atlas_ids_sha256":"afd72399dafd9906e9062927f70655952385a3d75488558e1f0484e950ab09b7","read_at":"2026-10-05T01:46:21Z"},"by_repo":[{"repo":"cvjena/deic","kind":"official","samples":0,"ran":0,"instrument":0,"unverified":0}],"by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":null,"arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"4f288965e305157f9f779507cd0ef9239705a2c4ad4818254043ae611319b1a3","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}