{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/in-search-of-the-successful-interpolation-on","title":"In Search of the Successful Interpolation: On the Role of Sharpness in CLIP Generalization","arxiv_id":"2410.16476","date":"2024-10-21","proceeding":null,"authors":["Alireza Abdollahpoorrostam"],"abstract":"\\textit{Zero-shot} models like CLIP are often fine-tuned on a target dataset to improve its accuracy further, but this can compromise out-of-distribution (OOD) robustness. Robust Fine-Tuning (\\texttt{RFT} )~\\citep{wortsman2021robust}, which interpolates between the \\textit{zero-shot} and \\textit{fine-tuned} models, has been proposed to address this issue. However, understanding when \\texttt{RFT} actually improves OOD error remains limited. In this work, we empirically investigate the robustness of \\texttt{RFT} in CLIP models, with a focus on the \\textit{sharpness} of the CLIP model during interpolation. First, we demonstrate that while sharpness may not serve as a reliable indicator for predicting the generalization of modern architectures like CLIP on OOD data, this challenges the conventional belief in the generalization benefits of flat minima in foundation models. However, by examining the role of the \\textit{straggler layer} phenomenon, we show that, unlike overall sharpness, the \\textit{layer-wise} sharpness of \\textit{straggler} layers can reliably capture the generalization performance of interpolated CLIP models on OOD data. Our extensive experiments reveal that \\textit{layer-wise} sharpness correlates with generalization in OOD accuracy for \\texttt{RFT}. Furthermore, we demonstrate that by inducing sparsity in the \\textit{straggler} layers, we can mitigate the \\textit{failure mode} phenomenon in \\texttt{RFT}. To the best of our knowledge, this is the first work to study the role of sharpness in the \\textit{success} of interpolation in the weight space of CLIP foundation models. Our code is available at \\url{https://github.com/alirezaabdollahpour/CLIP_Mode_Connectivity}.","url_abs":"https://arxiv.org/abs/2410.16476v1","url_pdf":"https://arxiv.org/pdf/2410.16476v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"in-search-of-the-successful-interpolation-on","repo_url":"https://github.com/alirezaabdollahpour/clip_mode_connectivity","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":null}],"tasks":[],"methods":[{"method_slug":"clip","method_name":"CLIP"},{"method_slug":"focus","method_name":"Focus"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers","held":null,"reason":"arxiv_id_not_in_atlas","checked_against":{"atlas_ids":549449,"atlas_ids_sha256":"afd72399dafd9906e9062927f70655952385a3d75488558e1f0484e950ab09b7","read_at":"2026-10-05T01:46:21Z"},"by_repo":[{"repo":"alirezaabdollahpour/clip_mode_connectivity","kind":"official","samples":0,"ran":0,"instrument":0,"unverified":0}],"by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":null,"arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"fc0c22d8a75fbdcfe491f14b8024942a9b1f8378fdbc9423d100b0c40c8a0dcb","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}