{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/self-supervised-learning-from-web-data-for","title":"Self-Supervised Learning from Web Data for Multimodal Retrieval","arxiv_id":"1901.02004","date":"2019-01-07","proceeding":null,"authors":["Raul Gomez","Lluis Gomez","Jaume Gibert","Dimosthenis Karatzas"],"abstract":"Self-Supervised learning from multimodal image and text data allows deep\nneural networks to learn powerful features with no need of human annotated\ndata. Web and Social Media platforms provide a virtually unlimited amount of\nthis multimodal data. In this work we propose to exploit this free available\ndata to learn a multimodal image and text embedding, aiming to leverage the\nsemantic knowledge learnt in the text domain and transfer it to a visual model\nfor semantic image retrieval. We demonstrate that the proposed pipeline can\nlearn from images with associated textwithout supervision and analyze the\nsemantic structure of the learnt joint image and text embedding space. We\nperform a thorough analysis and performance comparison of five different state\nof the art text embeddings in three different benchmarks. We show that the\nembeddings learnt with Web and Social Media data have competitive performances\nover supervised methods in the text based image retrieval task, and we clearly\noutperform state of the art in the MIRFlickr dataset when training in the\ntarget data. Further, we demonstrate how semantic multimodal image retrieval\ncan be performed using the learnt embeddings, going beyond classical\ninstance-level retrieval problems. Finally, we present a new dataset,\nInstaCities1M, composed by Instagram images and their associated texts that can\nbe used for fair comparison of image-text embeddings.","url_abs":"http://arxiv.org/abs/1901.02004v1","url_pdf":"http://arxiv.org/pdf/1901.02004v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"self-supervised-learning-from-web-data-for","repo_url":"https://github.com/gombru/LearnFromWebData","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":null}],"tasks":[{"task_slug":"image-retrieval","task_name":"Image Retrieval"},{"task_slug":"retrieval","task_name":"Retrieval"},{"task_slug":"self-supervised-learning","task_name":"Self-Supervised Learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers","held":null,"reason":"arxiv_id_not_in_atlas","checked_against":{"atlas_ids":549449,"atlas_ids_sha256":"afd72399dafd9906e9062927f70655952385a3d75488558e1f0484e950ab09b7","read_at":"2026-10-05T01:46:21Z"},"by_repo":[{"repo":"gombru/LearnFromWebData","kind":"official","samples":0,"ran":0,"instrument":0,"unverified":0}],"by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":null,"arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"fdc3a9782927d0ca47d02bef7dc070f400a5294a3a31dce3175417c4a9a0d138","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}