{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/umg-clip-a-unified-multi-granularity-vision","title":"UMG-CLIP: A Unified Multi-Granularity Vision Generalist for Open-World Understanding","arxiv_id":"2401.06397","date":"2024-01-12","proceeding":null,"authors":["Bowen Shi","Peisen Zhao","Zichen Wang","Yuhang Zhang","Yaoming Wang","Jin Li","Wenrui Dai","Junni Zou","Hongkai Xiong","Qi Tian","Xiaopeng Zhang"],"abstract":"Vision-language foundation models, represented by Contrastive Language-Image Pre-training (CLIP), have gained increasing attention for jointly understanding both vision and textual tasks. However, existing approaches primarily focus on training models to match global image representations with textual descriptions, thereby overlooking the critical alignment between local regions and corresponding text tokens. This paper extends CLIP with multi-granularity alignment. Notably, we deliberately construct a new dataset comprising pseudo annotations at various levels of granularities, encompassing image-level, region-level as well as pixel-level captions and tags. Accordingly, we develop a Unified Multi-Granularity learning framework, termed UMG-CLIP, which simultaneously empowers the model with versatile perception abilities across different levels of detail. With parameter efficient tuning, UMG-CLIP surpasses current widely used CLIP variants and achieves state-of-the-art performance on diverse image understanding benchmarks, including open-world recognition, retrieval, semantic segmentation, and panoptic segmentation tasks. We believe that UMG-CLIP represents a valuable advancement in vision-language foundation models. The code is available at https://github.com/lygsbw/UMG-CLIP.","url_abs":"https://arxiv.org/abs/2401.06397v3","url_pdf":"https://arxiv.org/pdf/2401.06397v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"umg-clip-a-unified-multi-granularity-vision","repo_url":"https://github.com/lygsbw/umg-clip","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok"}}],"tasks":[{"task_slug":"open-vocabulary-panoptic-segmentation","task_name":"Open Vocabulary Panoptic Segmentation"},{"task_slug":"open-vocabulary-semantic-segmentation","task_name":"Open Vocabulary Semantic Segmentation"},{"task_slug":"panoptic-segmentation","task_name":"Panoptic Segmentation"},{"task_slug":"retrieval","task_name":"Retrieval"},{"task_slug":"segmentation","task_name":"Segmentation"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"}],"methods":[{"method_slug":"clip","method_name":"CLIP"},{"method_slug":"focus","method_name":"Focus"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/open-vocabulary-panoptic-segmentation-on","task":"Open Vocabulary Panoptic Segmentation","dataset":"ADE20K","model":"UMG-CLIP-E/14","rank_in_archive_order":1,"of":10,"metrics":{"PQ":"31.6"},"uses_additional_data":false},{"leaderboard":"/sota/open-vocabulary-panoptic-segmentation-on","task":"Open Vocabulary Panoptic Segmentation","dataset":"ADE20K","model":"UMG-CLIP-L/14","rank_in_archive_order":3,"of":10,"metrics":{"PQ":"29.1"},"uses_additional_data":false},{"leaderboard":"/sota/open-vocabulary-semantic-segmentation-on-2","task":"Open Vocabulary Semantic Segmentation","dataset":"ADE20K-150","model":"UMG-CLIP-E/14","rank_in_archive_order":3,"of":23,"metrics":{"mIoU":"38.2"},"uses_additional_data":false},{"leaderboard":"/sota/open-vocabulary-semantic-segmentation-on-2","task":"Open Vocabulary Semantic Segmentation","dataset":"ADE20K-150","model":"UMG-CLIP-L/14","rank_in_archive_order":7,"of":23,"metrics":{"mIoU":"36.1"},"uses_additional_data":false},{"leaderboard":"/sota/open-vocabulary-semantic-segmentation-on-3","task":"Open Vocabulary Semantic Segmentation","dataset":"ADE20K-847","model":"UMG-CLIP-E/14","rank_in_archive_order":1,"of":19,"metrics":{"mIoU":"17.3"},"uses_additional_data":false},{"leaderboard":"/sota/open-vocabulary-semantic-segmentation-on-3","task":"Open Vocabulary Semantic Segmentation","dataset":"ADE20K-847","model":"UMG-CLIP-L/14","rank_in_archive_order":5,"of":19,"metrics":{"mIoU":"15.4"},"uses_additional_data":false},{"leaderboard":"/sota/open-vocabulary-semantic-segmentation-on-7","task":"Open Vocabulary Semantic Segmentation","dataset":"PASCAL Context-459","model":"UMG-CLIP-E/14","rank_in_archive_order":2,"of":15,"metrics":{"mIoU":"25.2"},"uses_additional_data":false},{"leaderboard":"/sota/open-vocabulary-semantic-segmentation-on-7","task":"Open Vocabulary Semantic Segmentation","dataset":"PASCAL Context-459","model":"UMG-CLIP-L/14","rank_in_archive_order":5,"of":15,"metrics":{"mIoU":"23.2"},"uses_additional_data":false},{"leaderboard":"/sota/open-vocabulary-semantic-segmentation-on-1","task":"Open Vocabulary Semantic Segmentation","dataset":"PASCAL Context-59","model":"UMG-CLIP-L/14","rank_in_archive_order":6,"of":24,"metrics":{"mIoU":"61.0"},"uses_additional_data":false},{"leaderboard":"/sota/open-vocabulary-semantic-segmentation-on-5","task":"Open Vocabulary Semantic Segmentation","dataset":"PascalVOC-20","model":"UMG-CLIP-L/14","rank_in_archive_order":1,"of":20,"metrics":{"mIoU":"97.9"},"uses_additional_data":false},{"leaderboard":"/sota/open-vocabulary-semantic-segmentation-on-9","task":"Open Vocabulary Semantic Segmentation","dataset":"PascalVOC-20b","model":"UMG-CLIP-E/14","rank_in_archive_order":1,"of":4,"metrics":{"mIoU":"85.4"},"uses_additional_data":false},{"leaderboard":"/sota/panoptic-segmentation-on-coco-minival","task":"Panoptic Segmentation","dataset":"COCO minival","model":"UMG-CLIP-E/14","rank_in_archive_order":4,"of":31,"metrics":{"AP":"50.7","PQ":"59.5","mIoU":"69.7"},"uses_additional_data":true},{"leaderboard":"/sota/panoptic-segmentation-on-coco-minival","task":"Panoptic Segmentation","dataset":"COCO minival","model":"UMG-CLIP-L/14","rank_in_archive_order":7,"of":31,"metrics":{"AP":"49.7","PQ":"58.9","mIoU":"68.9"},"uses_additional_data":true}],"syntology":{"syntology_url":"https://syntology.ai/paper/2401.06397","atlas_url":"https://app.syntology.ai/?focus=2401.06397","mcp":null,"developers":"https://syntology.ai/developers","held":true,"reason":"arxiv_id_in_atlas","checked_against":{"atlas_ids":549449,"atlas_ids_sha256":"afd72399dafd9906e9062927f70655952385a3d75488558e1f0484e950ab09b7","read_at":"2026-10-05T01:46:21Z"},"agent_calls":[{"tool":"get_citation_path","arguments":{"paper_1":"2401.06397"},"arguments_in":{"paper_2":"another paper's arXiv id or title"},"call":"get_citation_path(paper_1=\"2401.06397\", paper_2=\"…\")"},{"tool":"get_concepts_for_paper","arguments":{"arxiv_id":"2401.06397"},"call":"get_concepts_for_paper(arxiv_id=\"2401.06397\")"}],"by_repo":[{"repo":"lygsbw/umg-clip","kind":"official","samples":0,"ran":0,"instrument":0,"unverified":0}],"by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":null,"arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"94ee2c14b8dba975e0525acfe54b2dcacec729027f9f28101c0e3788f2bc1767","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}