{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/a-simple-framework-for-open-vocabulary","title":"A Simple Framework for Open-Vocabulary Segmentation and Detection","arxiv_id":"2303.08131","date":"2023-03-14","proceeding":"ICCV 2023 1","authors":["Hao Zhang","Feng Li","Xueyan Zou","Shilong Liu","Chunyuan Li","Jianfeng Gao","Jianwei Yang","Lei Zhang"],"abstract":"We present OpenSeeD, a simple Open-vocabulary Segmentation and Detection framework that jointly learns from different segmentation and detection datasets. To bridge the gap of vocabulary and annotation granularity, we first introduce a pre-trained text encoder to encode all the visual concepts in two tasks and learn a common semantic space for them. This gives us reasonably good results compared with the counterparts trained on segmentation task only. To further reconcile them, we locate two discrepancies: $i$) task discrepancy -- segmentation requires extracting masks for both foreground objects and background stuff, while detection merely cares about the former; $ii$) data discrepancy -- box and mask annotations are with different spatial granularity, and thus not directly interchangeable. To address these issues, we propose a decoupled decoding to reduce the interference between foreground/background and a conditioned mask decoding to assist in generating masks for given boxes. To this end, we develop a simple encoder-decoder model encompassing all three techniques and train it jointly on COCO and Objects365. After pre-training, our model exhibits competitive or stronger zero-shot transferability for both segmentation and detection. Specifically, OpenSeeD beats the state-of-the-art method for open-vocabulary instance and panoptic segmentation across 5 datasets, and outperforms previous work for open-vocabulary detection on LVIS and ODinW under similar settings. When transferred to specific tasks, our model achieves new SoTA for panoptic segmentation on COCO and ADE20K, and instance segmentation on ADE20K and Cityscapes. Finally, we note that OpenSeeD is the first to explore the potential of joint training on segmentation and detection, and hope it can be received as a strong baseline for developing a single model for both tasks in open world.","url_abs":"https://arxiv.org/abs/2303.08131v3","url_pdf":"https://arxiv.org/pdf/2303.08131v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"a-simple-framework-for-open-vocabulary","repo_url":"https://github.com/idea-research/openseed","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"a-simple-framework-for-open-vocabulary","repo_url":"https://github.com/microsoft/X-Decoder","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"instance-segmentation","task_name":"Instance Segmentation"},{"task_slug":"panoptic-segmentation","task_name":"Panoptic Segmentation"},{"task_slug":"segmentation","task_name":"Segmentation"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"},{"task_slug":"zero-shot-segmentation","task_name":"Zero Shot Segmentation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/instance-segmentation-on-ade20k-val","task":"Instance Segmentation","dataset":"ADE20K val","model":"OpenSeeD","rank_in_archive_order":2,"of":14,"metrics":{"AP":"42.6"},"uses_additional_data":true},{"leaderboard":"/sota/instance-segmentation-on-cityscapes-val","task":"Instance Segmentation","dataset":"Cityscapes val","model":"OpenSeeD( SwinL, single-scale)","rank_in_archive_order":3,"of":17,"metrics":{"mask AP":"48.5"},"uses_additional_data":true},{"leaderboard":"/sota/panoptic-segmentation-on-ade20k-val","task":"Panoptic Segmentation","dataset":"ADE20K val","model":"OpenSeed(SwinL, single scale, 1280x1280)","rank_in_archive_order":3,"of":25,"metrics":{"PQ":"53.7"},"uses_additional_data":true},{"leaderboard":"/sota/panoptic-segmentation-on-coco-minival","task":"Panoptic Segmentation","dataset":"COCO minival","model":"OpenSeeD (SwinL, single-scale)","rank_in_archive_order":3,"of":31,"metrics":{"AP":"53.2","PQ":"59.5"},"uses_additional_data":true},{"leaderboard":"/sota/zero-shot-segmentation-on-segmentation-in-the","task":"Zero Shot Segmentation","dataset":"Segmentation in the Wild","model":"OpenSEED","rank_in_archive_order":7,"of":12,"metrics":{"Mean AP":"36.1"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2303.08131","atlas_url":"https://app.syntology.ai/?focus=2303.08131","mcp":null,"developers":"https://syntology.ai/developers","held":true,"reason":"arxiv_id_in_atlas","checked_against":{"atlas_ids":549449,"atlas_ids_sha256":"afd72399dafd9906e9062927f70655952385a3d75488558e1f0484e950ab09b7","read_at":"2026-10-05T01:46:21Z"},"agent_calls":[{"tool":"get_citation_path","arguments":{"paper_1":"2303.08131"},"arguments_in":{"paper_2":"another paper's arXiv id or title"},"call":"get_citation_path(paper_1=\"2303.08131\", paper_2=\"…\")"},{"tool":"get_concepts_for_paper","arguments":{"arxiv_id":"2303.08131"},"call":"get_concepts_for_paper(arxiv_id=\"2303.08131\")"}],"by_repo":[{"repo":"idea-research/openseed","kind":"official","samples":0,"ran":0,"instrument":0,"unverified":0,"harvested_for_other_papers":1,"harvested_for":["2306.09347"],"harvested_for_calls":[{"arxiv_id":"2306.09347","tool":"get_harvested_code_for_paper","arguments":{"arxiv_id":"2306.09347"},"call":"get_harvested_code_for_paper(arxiv_id=\"2306.09347\")"}]},{"repo":"microsoft/X-Decoder","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0,"harvested_for_other_papers":1,"harvested_for":["2306.09347"],"harvested_for_calls":[{"arxiv_id":"2306.09347","tool":"get_harvested_code_for_paper","arguments":{"arxiv_id":"2306.09347"},"call":"get_harvested_code_for_paper(arxiv_id=\"2306.09347\")"}]}],"by_repo_harvested_for_is":"harvested_for: the arXiv ids (as Syntology's graph holds them) of every paper whose samples Syntology's graph links to this repository, in arXiv-id order, complete; the page names the first 3 and says how many more. A link only: it does not say the papers are the same work, that one copied the other, or that the code implements this paper's method; harvested_for_calls gives the get_harvested_code_for_paper call for each arXiv-shaped id among them","by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":null,"arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"0b592c8801e6c679eca8e9fa9f810d2160df3f56347896a6c6ec304dd0b86d30","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}