{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/cross-modal-retrieval-with-partially","title":"Cross-Modal Retrieval with Partially Mismatched Pairs","arxiv_id":null,"date":"2023-02-22","proceeding":"IEEE Transactions on Pattern Analysis and Machine Intelligence 2023 2","authors":["Peng Hu","Zhenyu Huang","Dezhong Peng","Xu Wang","Xi Peng"],"abstract":"In this paper, we study a challenging but less-touched problem in cross-modal retrieval, i.e., partially mismatched pairs (PMPs). Specifically, in real-world scenarios, a huge number of multimedia data (e.g., the Conceptual Captions dataset) are collected from the Internet, and thus it is inevitable to wrongly treat some irrelevant cross-modal pairs as matched. Undoubtedly, such a PMP problem will remarkably degrade the cross-modal retrieval performance. To tackle this problem, we derive a unified theoretical Robust Cross-modal Learning framework (RCL) with an unbiased estimator of the cross-modal retrieval risk, which aims to endow the cross-modal retrieval methods with robustness against PMPs. In detail, our RCL adopts a novel complementary contrastive learning paradigm to address the following two challenges, i.e., the overfitting and underfitting issues. On the one hand, our method only utilizes the negative information which is much less likely false compared with the positive information, thus avoiding the overfitting issue to PMPs. However, these robust strategies could induce underfitting issues, thus making training models more difficult. On the other hand, to address the underfitting issue brought by weak supervision, we present to leverage of all available negative pairs to enhance the supervision contained in the negative information. Moreover, to further improve the performance, we propose to minimize the upper bounds of the risk to pay more attention to hard samples. To verify the effectiveness and robustness of the proposed method, we carry out comprehensive experiments on five widely-used benchmark datasets compared with nine state-of-the-art approaches w.r.t. the image-text and video-text retrieval tasks. The code is available at https://github.com/penghu-cs/RCL .","url_abs":"https://ieeexplore.ieee.org/document/10050111","url_pdf":"https://ieeexplore.ieee.org/document/10050111","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"cross-modal-retrieval-with-partially","repo_url":"https://github.com/penghu-cs/RCL","is_official":0,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"contrastive-learning","task_name":"Contrastive Learning"},{"task_slug":"cross-modal-retrieval","task_name":"Cross-Modal Retrieval"},{"task_slug":"cross-modal-retrieval-with-noisy","task_name":"Cross-modal retrieval with noisy correspondence"},{"task_slug":"retrieval","task_name":"Retrieval"},{"task_slug":"text-retrieval","task_name":"Text Retrieval"},{"task_slug":"video-text-retrieval","task_name":"Video-Text Retrieval"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/cross-modal-retrieval-with-noisy-1","task":"Cross-modal retrieval with noisy correspondence","dataset":"CC152K","model":"RCL-SGRAF","rank_in_archive_order":12,"of":15,"metrics":{"Image-to-text R@1":"41.7","Image-to-text R@10":"73.6","Image-to-text R@5":"66.0","R-Sum":"364.4","Text-to-image R@1":"41.6","Text-to-image R@10":"75.1","Text-to-image R@5":"66.4"},"uses_additional_data":false},{"leaderboard":"/sota/cross-modal-retrieval-with-noisy-3","task":"Cross-modal retrieval with noisy correspondence","dataset":"COCO-Noisy","model":"RCL-SGR","rank_in_archive_order":17,"of":17,"metrics":{"Image-to-text R@1":"77.0","Image-to-text R@10":"98.1","Image-to-text R@5":"95.5","R-Sum":"515.5","Text-to-image R@1":"61.3","Text-to-image R@10":"94.8","Text-to-image R@5":"88.8"},"uses_additional_data":false},{"leaderboard":"/sota/cross-modal-retrieval-with-noisy-2","task":"Cross-modal retrieval with noisy correspondence","dataset":"Flickr30K-Noisy","model":"RCL-SGR","rank_in_archive_order":16,"of":16,"metrics":{"Image-to-text R@1":"74.2","Image-to-text R@10":"96.9","Image-to-text R@5":"91.8","R-Sum":"487.2","Text-to-image R@1":"55.6","Text-to-image R@10":"87.5","Text-to-image R@5":"81.2"},"uses_additional_data":false}],"syntology":{"syntology_url":null,"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers","held":null,"reason":"no_arxiv_id","checked_against":null,"by_repo":[{"repo":"penghu-cs/RCL","kind":"named_in_paper","samples":0,"ran":0,"instrument":0,"unverified":0}],"by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":null,"arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"747e16c58ac692f918b7f51eb61e7d1a7db235983fa5999186c7fb3d5f8172ef","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}