{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/groundhog-grounding-large-language-models-to","title":"GROUNDHOG: Grounding Large Language Models to Holistic Segmentation","arxiv_id":"2402.16846","date":"2024-02-26","proceeding":"CVPR 2024 1","authors":["Yichi Zhang","Ziqiao Ma","Xiaofeng Gao","Suhaila Shakiah","Qiaozi Gao","Joyce Chai"],"abstract":"Most multimodal large language models (MLLMs) learn language-to-object grounding through causal language modeling where grounded objects are captured by bounding boxes as sequences of location tokens. This paradigm lacks pixel-level representations that are important for fine-grained visual understanding and diagnosis. In this work, we introduce GROUNDHOG, an MLLM developed by grounding Large Language Models to holistic segmentation. GROUNDHOG incorporates a masked feature extractor and converts extracted features into visual entity tokens for the MLLM backbone, which then connects groundable phrases to unified grounding masks by retrieving and merging the entity masks. To train GROUNDHOG, we carefully curated M3G2, a grounded visual instruction tuning dataset with Multi-Modal Multi-Grained Grounding, by harvesting a collection of segmentation-grounded datasets with rich annotations. Our experimental results show that GROUNDHOG achieves superior performance on various language grounding tasks without task-specific fine-tuning, and significantly reduces object hallucination. GROUNDHOG also demonstrates better grounding towards complex forms of visual input and provides easy-to-understand diagnosis in failure cases.","url_abs":"https://arxiv.org/abs/2402.16846v2","url_pdf":"https://arxiv.org/pdf/2402.16846v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[{"task_slug":null,"task_name":"Causal Language Modeling"},{"task_slug":"generalized-referring-expression-segmentation","task_name":"Generalized Referring Expression Segmentation"},{"task_slug":"hallucination","task_name":"Hallucination"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"object-hallucination","task_name":"Object Hallucination"},{"task_slug":"referring-expression-segmentation","task_name":"Referring Expression Segmentation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/generalized-referring-expression-segmentation","task":"Generalized Referring Expression Segmentation","dataset":"gRefCOCO","model":"GROUNDHOG","rank_in_archive_order":6,"of":13,"metrics":{"gIoU":"66.70"},"uses_additional_data":true},{"leaderboard":"/sota/referring-expression-segmentation-on","task":"Referring Expression Segmentation","dataset":"PhraseCut","model":"GROUNDHOG","rank_in_archive_order":2,"of":6,"metrics":{"Mean IoU":"54.5"},"uses_additional_data":false},{"leaderboard":"/sota/referring-expression-segmentation-on-refcoco-5","task":"Referring Expression Segmentation","dataset":"RefCOCO+ test B","model":"GROUNDHOG","rank_in_archive_order":9,"of":30,"metrics":{"Overall IoU":"64.9"},"uses_additional_data":true},{"leaderboard":"/sota/referring-expression-segmentation-on-refcoco-4","task":"Referring Expression Segmentation","dataset":"RefCOCO+ testA","model":"GROUNDHOG","rank_in_archive_order":11,"of":30,"metrics":{"Overall IoU":"75.0"},"uses_additional_data":true},{"leaderboard":"/sota/referring-expression-segmentation-on-refcoco-3","task":"Referring Expression Segmentation","dataset":"RefCOCO+ val","model":"GROUNDHOG","rank_in_archive_order":13,"of":33,"metrics":{"Overall IoU":"70.5"},"uses_additional_data":true},{"leaderboard":"/sota/referring-expression-segmentation-on-refcocog-1","task":"Referring Expression Segmentation","dataset":"RefCOCOg-test","model":"GROUNDHOG","rank_in_archive_order":8,"of":18,"metrics":{"Overall IoU":"74.6"},"uses_additional_data":true},{"leaderboard":"/sota/referring-expression-segmentation-on-refcocog","task":"Referring Expression Segmentation","dataset":"RefCOCOg-val","model":"GROUNDHOG","rank_in_archive_order":9,"of":23,"metrics":{"Overall IoU":"74.1"},"uses_additional_data":true},{"leaderboard":"/sota/referring-expression-segmentation-on-refcoco","task":"Referring Expression Segmentation","dataset":"RefCoCo val","model":"GROUNDHOG","rank_in_archive_order":14,"of":37,"metrics":{"Overall IoU":"78.5"},"uses_additional_data":true}],"syntology":{"syntology_url":"https://syntology.ai/paper/2402.16846","atlas_url":"https://app.syntology.ai/?focus=2402.16846","mcp":null,"developers":"https://syntology.ai/developers","held":true,"reason":"arxiv_id_in_atlas","checked_against":{"atlas_ids":549449,"atlas_ids_sha256":"afd72399dafd9906e9062927f70655952385a3d75488558e1f0484e950ab09b7","read_at":"2026-10-05T01:46:21Z"},"agent_calls":[{"tool":"get_citation_path","arguments":{"paper_1":"2402.16846"},"arguments_in":{"paper_2":"another paper's arXiv id or title"},"call":"get_citation_path(paper_1=\"2402.16846\", paper_2=\"…\")"},{"tool":"get_concepts_for_paper","arguments":{"arxiv_id":"2402.16846"},"call":"get_concepts_for_paper(arxiv_id=\"2402.16846\")"}],"by_repo":[],"by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":"the archive holds no code link for this paper","arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"7b4572f6d0a4824356be50694c5e7ad0bf946153cf0e0c5175acbccce2d5da64","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}