{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/sycoca-symmetrizing-contrastive-captioners","title":"SyCoCa: Symmetrizing Contrastive Captioners with Attentive Masking for Multimodal Alignment","arxiv_id":"2401.02137","date":"2024-01-04","proceeding":null,"authors":["Ziping Ma","Furong Xu","Jian Liu","Ming Yang","Qingpei Guo"],"abstract":"Multimodal alignment between language and vision is the fundamental topic in current vision-language model research. Contrastive Captioners (CoCa), as a representative method, integrates Contrastive Language-Image Pretraining (CLIP) and Image Caption (IC) into a unified framework, resulting in impressive results. CLIP imposes a bidirectional constraints on global representation of entire images and sentences. Although IC conducts an unidirectional image-to-text generation on local representation, it lacks any constraint on local text-to-image reconstruction, which limits the ability to understand images at a fine-grained level when aligned with texts. To achieve multimodal alignment from both global and local perspectives, this paper proposes Symmetrizing Contrastive Captioners (SyCoCa), which introduces bidirectional interactions on images and texts across the global and local representation levels. Specifically, we expand a Text-Guided Masked Image Modeling (TG-MIM) head based on ITC and IC heads. The improved SyCoCa can further leverage textual cues to reconstruct contextual images and visual cues to predict textual contents. When implementing bidirectional local interactions, the local contents of images tend to be cluttered or unrelated to their textual descriptions. Thus, we employ an attentive masking strategy to select effective image patches for interaction. Extensive experiments on five vision-language tasks, including image-text retrieval, image-captioning, visual question answering, and zero-shot/finetuned image classification, validate the effectiveness of our proposed method.","url_abs":"https://arxiv.org/abs/2401.02137v1","url_pdf":"https://arxiv.org/pdf/2401.02137v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[{"task_slug":"image-captioning","task_name":"Image Captioning"},{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"image-reconstruction","task_name":"Image Reconstruction"},{"task_slug":"image-to-text","task_name":"Image to text"},{"task_slug":"image-text-retrieval","task_name":"Image-text Retrieval"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"text-generation","task_name":"Text Generation"},{"task_slug":"text-retrieval","task_name":"Text Retrieval"},{"task_slug":"visual-question-answering-1","task_name":"Visual Question Answering"},{"task_slug":"image-classification","task_name":"image-classification"}],"methods":[{"method_slug":"clip","method_name":"CLIP"},{"method_slug":"sycoca","method_name":"SyCoCa"}],"datasets_introduced":[],"methods_introduced":[{"slug":"sycoca","name":"SyCoCa","full_name":"Symmetrizing Contrastive Captioners with Attentive Masking for Multimodal Alignment"}],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2401.02137","atlas_url":"https://app.syntology.ai/?focus=2401.02137","mcp":null,"developers":"https://syntology.ai/developers","held":true,"reason":"arxiv_id_in_atlas","checked_against":{"atlas_ids":549449,"atlas_ids_sha256":"afd72399dafd9906e9062927f70655952385a3d75488558e1f0484e950ab09b7","read_at":"2026-10-05T01:46:21Z"},"agent_calls":[{"tool":"get_citation_path","arguments":{"paper_1":"2401.02137"},"arguments_in":{"paper_2":"another paper's arXiv id or title"},"call":"get_citation_path(paper_1=\"2401.02137\", paper_2=\"…\")"},{"tool":"get_concepts_for_paper","arguments":{"arxiv_id":"2401.02137"},"call":"get_concepts_for_paper(arxiv_id=\"2401.02137\")"}],"by_repo":[],"by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":"the archive holds no code link for this paper","arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"b5c3522685f56d9e28beb4b5c4d7531ee7f206a56018f48fb9de043313f97501","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}