{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/crafting-distribution-shifts-for-validation","title":"Crafting Distribution Shifts for Validation and Training in Single Source Domain Generalization","arxiv_id":"2409.19774","date":"2024-09-29","proceeding":null,"authors":["Nikos Efthymiadis","Giorgos Tolias","Ondřej Chum"],"abstract":"Single-source domain generalization attempts to learn a model on a source domain and deploy it to unseen target domains. Limiting access only to source domain data imposes two key challenges - how to train a model that can generalize and how to verify that it does. The standard practice of validation on the training distribution does not accurately reflect the model's generalization ability, while validation on the test distribution is a malpractice to avoid. In this work, we construct an independent validation set by transforming source domain images with a comprehensive list of augmentations, covering a broad spectrum of potential distribution shifts in target domains. We demonstrate a high correlation between validation and test performance for multiple methods and across various datasets. The proposed validation achieves a relative accuracy improvement over the standard validation equal to 15.4% or 1.6% when used for method selection or learning rate tuning, respectively. Furthermore, we introduce a novel family of methods that increase the shape bias through enhanced edge maps. To benefit from the augmentations during training and preserve the independence of the validation set, a k-fold validation process is designed to separate the augmentation types used in training and validation. The method that achieves the best performance on the augmented validation is selected from the proposed family. It achieves state-of-the-art performance on various standard benchmarks. Code at: https://github.com/NikosEfth/crafting-shifts","url_abs":"https://arxiv.org/abs/2409.19774v1","url_pdf":"https://arxiv.org/pdf/2409.19774v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"crafting-distribution-shifts-for-validation","repo_url":"https://github.com/nikosefth/crafting-shifts","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"domain-generalization","task_name":"Domain Generalization"},{"task_slug":"image-to-sketch-recognition","task_name":"Image to sketch recognition"},{"task_slug":"photo-to-rest-generalization","task_name":"Photo to Rest Generalization"},{"task_slug":"single-source-domain-generalization","task_name":"Single-Source Domain Generalization"}],"methods":[{"method_slug":"set","method_name":"SET"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/image-to-sketch-recognition-on-minidomainnet","task":"Image to sketch recognition","dataset":"MiniDomainNet","model":"Crafting-Shifts(ResNet18)","rank_in_archive_order":1,"of":1,"metrics":{"Accuracy":"57.51"},"uses_additional_data":false},{"leaderboard":"/sota/image-to-sketch-recognition-on-pacs","task":"Image to sketch recognition","dataset":"PACS","model":"Crafting-Shifts(ResNet18)","rank_in_archive_order":1,"of":7,"metrics":{"Accuracy":"74.13"},"uses_additional_data":false},{"leaderboard":"/sota/image-to-sketch-recognition-on-pacs","task":"Image to sketch recognition","dataset":"PACS","model":"Crafting-Shifts(AlexNet)","rank_in_archive_order":3,"of":7,"metrics":{"Accuracy":"68.5"},"uses_additional_data":false},{"leaderboard":"/sota/photo-to-rest-generalization-on-minidomainnet","task":"Photo to Rest Generalization","dataset":"MiniDomainNet","model":"Crafting-Shifts(ResNet18)","rank_in_archive_order":1,"of":1,"metrics":{"Accuracy":"57.35"},"uses_additional_data":false},{"leaderboard":"/sota/photo-to-rest-generalization-on-pacs","task":"Photo to Rest Generalization","dataset":"PACS","model":"Crafting-Shifts(ResNet18)","rank_in_archive_order":1,"of":8,"metrics":{"Accuracy":"65.85"},"uses_additional_data":false},{"leaderboard":"/sota/photo-to-rest-generalization-on-pacs","task":"Photo to Rest Generalization","dataset":"PACS","model":"Crafting-Shifts(AlexNet)","rank_in_archive_order":3,"of":8,"metrics":{"Accuracy":"60.97"},"uses_additional_data":false},{"leaderboard":"/sota/single-source-domain-generalization-on-digits","task":"Single-Source Domain Generalization","dataset":"Digits-five","model":"Crafting-Shifts(LeNet)","rank_in_archive_order":1,"of":7,"metrics":{"Accuracy":"82.61"},"uses_additional_data":false},{"leaderboard":"/sota/single-source-domain-generalization-on-pacs","task":"Single-Source Domain Generalization","dataset":"PACS","model":"Crafting-Shifts(ResNet18)","rank_in_archive_order":1,"of":10,"metrics":{"Accuracy":"70.37"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2409.19774","atlas_url":"https://app.syntology.ai/?focus=2409.19774","mcp":null,"developers":"https://syntology.ai/developers","held":true,"reason":"arxiv_id_in_atlas","checked_against":{"atlas_ids":549449,"atlas_ids_sha256":"afd72399dafd9906e9062927f70655952385a3d75488558e1f0484e950ab09b7","read_at":"2026-10-05T01:46:21Z"},"agent_calls":[{"tool":"get_citation_path","arguments":{"paper_1":"2409.19774"},"arguments_in":{"paper_2":"another paper's arXiv id or title"},"call":"get_citation_path(paper_1=\"2409.19774\", paper_2=\"…\")"},{"tool":"get_concepts_for_paper","arguments":{"arxiv_id":"2409.19774"},"call":"get_concepts_for_paper(arxiv_id=\"2409.19774\")"}],"by_repo":[{"repo":"nikosefth/crafting-shifts","kind":"official","samples":0,"ran":0,"instrument":0,"unverified":0}],"by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":null,"arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"8d4e0baad8de773dc263127dda00206cb72211a01e14f8e46d21d485e6499f8a","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}