Browse State-of-the-Art › Phrase Grounding
Phrase Grounding
51 papers with code · 5 benchmarks · 6 datasets archive 2025-07-28
Given an image and a corresponding caption, the Phrase Grounding task aims to ground each entity mentioned by a noun phrase in the caption to a region in the image.
Source: Phrase Grounding by Soft-Label Chain Conditional Random Field
Description from the archive archive 2025-07-28.
Benchmarks archive 2025-07-28
5 leaderboard tables shown for this task, 5 with rows (a “benchmark” on this site is a table with at least one row, as on /sota), ordered by row count. “Best model” is the first row in the archive's own order at snapshot; nothing is re-ranked here and metric direction is not recorded in the archive. PwC's Trend sparklines are not in the archive, so that column is omitted.
Syntology column: samples harvested from the paper's repositories and executed on synthesized fixtures; “ran” is not a correctness claim and does not order the table. Each line reads “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the instrument figure counts failures of Syntology's instrument, not of the code. When the archive marks a repository official for the row's paper, the line starts with that repository's state (“official” is the archive's flag, not a verdict on who wrote the code; “community repositories only” when every sample that ran came from a community repository, “official: no sample here; runs from other or unrecorded repositories” when some came from a repository the paper names or has in its text, or from none recorded); hover it for the repositories the samples that ran came from. A dash means no Syntology record for that paper, not a recorded non-run. Each “ran” line links to that paper's sample list. Syntology's record for this page last changed ; when this build read Syntology's graph is in the build record. For agents: get_harvested_code_for_paper(arxiv_id) lists each paper's samples; how to connect.
Libraries
Not in the archive: the export carries no per-task library table, so there is nothing to show at snapshot 2025-07-28.
Datasets archive 2025-07-28
6 datasets whose archive record lists this task, ordered by the archive's paper count.
Subtasks archive 2025-07-28
1 subtask in the archive's task tree.
Papers archive 2025-07-28
archive papers tagged: 88 · with a code link: 51 · where Syntology ran a sample: 20 (17 with a run with no instrument failure, 3 where every run was a failure of Syntology's instrument) Syntology
Show: all tagged papersonly where code ran (20 of 88 tagged: 17 with a run with no instrument failure, 3 where every run was a failure of Syntology's instrument)
Page 1 of 1: papers 1 to 88 of 88, in archive order: by repositories listed in the archive (most first), then newest first, not by stars (the archive holds no stars, so PwC's “Social” and “Latest” sorts cannot be reproduced). Papers that list no repository come after every paper that lists one.
Papers without a page here are shown as plain text. A Syntology line reads “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the figure “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. When the archive marks a repository official for the paper, the line starts with that repository's state (the archive's flag, not a verdict on who wrote the code); hover it for the repositories the samples that ran came from. The first sentence of the abstract is shown for the first 30 papers; every abstract is on its paper's page.
-
6 Jun 2016 10 repositories listedApproaches to multimodal pooling include element-wise product or sum, as well as concatenation of the visual and textual representations.
-
26 Apr 2021 5 repositories listed Syntology community repositories only · 7 ran (of which 4 constructed an object rather than computing a result; 6 with no instrument failure: 0 honoured, 1 violated, 5 with no contract checked; 1 where Syntology's instrument failed) · 4 unverified (of 11 harvested samples)We also investigate the utility of our model as an object detector on a given label set when fine-tuned in a few-shot setting.
-
28 Dec 2024 4 repositories listed Syntology 13 ran (of which 0 constructed an object rather than computing a result; 7 with no instrument failure: 0 honoured, 0 violated, 7 with no contract checked; 6 where Syntology's instrument failed) · 2 unverified (of 15 harvested samples) · 9 pointer-only (licence)Finally, we outline the challenges confronting visual grounding and propose valuable directions for future research, which may serve as inspiration for subsequent researchers.
-
17 Nov 2018 3 repositories listedMost existing work that grounds natural language phrases in images starts with the assumption that the phrase in question is relevant to the image.
-
12 Nov 2015 3 repositories listedWe propose a novel approach which learns grounding by reconstructing a given phrase using an attention mechanism, which can be either latent or optimized directly.
-
4 Jan 2024 2 repositories listed Syntology official (archive's flag): 4 ran · 4 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 2 where Syntology's instrument failed) · 2 unverified (of 6 harvested samples) · 3 pointer-only (licence)Grounding-DINO is a state-of-the-art open-set detection model that tackles multiple vision tasks including Open-Vocabulary Detection (OVD), Phrase Grounding (PG), and Referring Expression Comprehension (REC).
-
26 Jun 2023 2 repositories listedWe introduce Kosmos-2, a Multimodal Large Language Model (MLLM), enabling new capabilities of perceiving object descriptions (e. g., bounding boxes) and grounding text to the visual world.
-
21 Apr 2022 2 repositories listed Syntology official (archive's flag): 3 ran · 3 ran (of which 3 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 0 violated, 3 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified; every one of the 3 samples that ran constructed an object rather than computing a result (of 4 harvested samples)We release a new dataset with locally-aligned phrase grounding annotations by radiologists to facilitate the study of complex semantic modelling in biomedical vision--language processing.
-
19 May 2015 2 repositories listed Syntology 2 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified (of 2 harvested samples) · 2 pointer-only (licence)The Flickr30k dataset has become a standard benchmark for sentence-based image description.
-
16 May 2025 1 repository listed Syntology official: no sample here; runs from other or unrecorded repositories · 3 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 1 violated, 1 with no contract checked; 1 where Syntology's instrument failed) · 3 unverified (of 6 harvested samples)Phrase grounding between images and their captions is a well-established task.
-
23 Feb 2025 1 repository listedMedical Phrase Grounding (MPG) maps radiological findings described in medical reports to specific regions in medical images.
-
29 Jan 2025 1 repository listedAs artificial intelligence (AI) becomes increasingly central to healthcare, the demand for explainable and trustworthy models is paramount.
-
16 Oct 2024 1 repository listed Syntology official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 1 harvested sample) · 1 pointer-only (licence)Many artwork collections contain textual attributes that provide rich and contextualised descriptions of artworks.
-
13 Sep 2024 1 repository listedIn this paper, we address a challenging task, synchronous motion captioning, that aim to generate a language description synchronized with human motion sequences.
-
20 Aug 2024 1 repository listedThe core of our approach involves freezing the MDETR backbone and training only the Universal Projection module (UP), which bridges vision and language representations.
-
1 Jul 2024 1 repository listedWe introduce the concept of "empathic grounding" in conversational agents as an extension of Clark's conceptualization of grounding in conversation in which the grounding criterion includes listener empathy for the…
-
19 Apr 2024 1 repository listed Syntology official (archive's flag): 7 ran · 7 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 2 honoured, 1 violated, 2 with no contract checked; 2 where Syntology's instrument failed) · 2 unverified (of 9 harvested samples)In addition, we aim to perform this task in a zero-shot manner, i. e., without any training on the target task, meaning that the model's weights remain frozen.
-
14 Mar 2024 1 repository listedLarge Vision Language Models have achieved fine-grained object perception, but the limitation of image resolution remains a significant obstacle to surpass the performance of task-specific experts in complex and dense…
-
22 Nov 2023 1 repository listed Syntology official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 1 honoured, 1 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 2 unverified (of 5 harvested samples) · 5 pointer-only (licence)Extending image-based Large Multimodal Models (LMMs) to videos is challenging due to the inherent complexity of video data.
-
5 Nov 2023 1 repository listedWhile we demonstrate our data augmentation method with MDETR framework, the proposed approach is applicable to common grounding-based vision and language tasks with other frameworks.
-
23 Oct 2023 1 repository listedThe ability to actively ground task instructions from an egocentric view is crucial for AI agents to accomplish tasks or assist humans virtually.
-
12 Sep 2023 1 repository listedIt designs a correlation weighting mechanism to adjust the correlation between masked chest X-ray image patches and their corresponding reports, thereby enhancing the model's representation learning capabilities.
-
7 Sep 2023 1 repository listed Syntology official (archive's flag): 6 ran · 6 ran (of which 0 constructed an object rather than computing a result; 6 with no instrument failure: 3 honoured, 0 violated, 3 with no contract checked; 0 where Syntology's instrument failed) · 5 unverified (of 11 harvested samples) · 11 pointer-only (licence)It has been established that training a box-based detector network can enhance the localization performance of weakly supervised and unsupervised methods.
-
6 Sep 2023 1 repository listed Syntology official (archive's flag): 10 ran · 10 ran (of which 0 constructed an object rather than computing a result; 10 with no instrument failure: 1 honoured, 0 violated, 9 with no contract checked; 0 where Syntology's instrument failed) · 2 unverified (of 12 harvested samples) · 12 pointer-only (licence)Key to tasks that require reasoning about natural language in visual contexts is grounding words and phrases to image regions.
-
5 Sep 2023 1 repository listedIn recent years, cross-modal reasoning (CMR), the process of understanding and reasoning across different modalities, has emerged as a pivotal area with applications spanning from multimedia analysis to healthcare…
-
31 Mar 2023 1 repository listedRecent advancements in diffusion models have significantly impacted the trajectory of generative machine learning research, with many adopting the strategy of fine-tuning pre-trained models using domain-specific…
-
11 Jan 2023 1 repository listedPrior work in biomedical VLP has mostly relied on the alignment of single image and report pairs even though clinical notes commonly refer to prior images.
-
1 Jan 2023 1 repository listedA phrase grounding model receives an input image and a text phrase and outputs a suitable localization map.
-
28 Nov 2022 1 repository listedAs phrase extraction can be regarded as a 1D text segmentation problem, we formulate PEG as a dual detection problem and propose a novel DQ-DETR model, which introduces dual queries to probe different features from…
-
23 Oct 2022 1 repository listed Syntology official (archive's flag): 6 ran · 6 ran (of which 0 constructed an object rather than computing a result; 6 with no instrument failure: 0 honoured, 0 violated, 6 with no contract checked; 0 where Syntology's instrument failed) · 3 unverified (of 9 harvested samples)First, we construct a dataset of phrase grounding with both noun phrases and pronouns to image regions.
-
10 Sep 2022 1 repository listed
-
19 Jun 2022 1 repository listed
-
15 Jun 2022 1 repository listed Syntology official: harvested, nothing ran · 2 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 1 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 1 unverified (of 3 harvested samples) · 2 pointer-only (licence)
-
12 Jun 2022 1 repository listed Syntology official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 1 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 2 harvested samples)
-
23 May 2022 1 repository listed Syntology official (archive's flag): 5 ran · 5 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 3 where Syntology's instrument failed) · 2 unverified (of 7 harvested samples)
-
27 Mar 2022 1 repository listed
-
20 Apr 2021 1 repository listed
-
1 Jan 2021 1 repository listed
-
1 Nov 2020 1 repository listed
-
12 Oct 2020 1 repository listed
-
3 Jul 2020 1 repository listed Syntology official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified (of 3 harvested samples)
-
18 Jun 2020 1 repository listed Syntology official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 0 where Syntology's instrument failed) · 2 unverified (of 4 harvested samples)
-
17 Jun 2020 1 repository listed
-
13 Feb 2020 1 repository listed
-
1 Sep 2019 1 repository listed
-
20 Aug 2019 1 repository listed Syntology official (archive's flag): 16 ran · 16 ran (of which 0 constructed an object rather than computing a result; 12 with no instrument failure: 0 honoured, 0 violated, 12 with no contract checked; 4 where Syntology's instrument failed) · 2 unverified (of 18 harvested samples) · 1 pointer-only (licence)
-
7 Apr 2019 1 repository listed
-
28 Nov 2018 1 repository listed
-
9 May 2018 1 repository listed
-
22 Nov 2017 1 repository listed
-
13 Nov 2015 1 repository listed Syntology 2 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 1 unverified (of 3 harvested samples) · 3 pointer-only (licence)
-
Anatomy-Grounded Weakly Supervised Prompt Tuning for Chest X-ray Latent Diffusion Models12 Jun 2025 0 repositories listed
-
A Comparison of Object Detection and Phrase Grounding Models in Chest X-ray Abnormality Localization using Eye-tracking Data2 Mar 2025 0 repositories listed
-
Progressive Local Alignment for Medical Multimodal Pre-training25 Feb 2025 0 repositories listed
-
Contextual Self-paced Learning for Weakly Supervised Spatio-Temporal Video Grounding28 Jan 2025 0 repositories listed
-
Hierarchical Alignment-enhanced Adaptive Grounding Network for Generalized Referring Expression Comprehension2 Jan 2025 0 repositories listed
-
ViCaS: A Dataset for Combining Holistic and Pixel-level Video Understanding using Captions with Grounded Segmentation12 Dec 2024 0 repositories listed
-
Pre-Training Multimodal Hallucination Detectors with Corrupted Grounding Data30 Aug 2024 0 repositories listed
-
CXR-Agent: Vision-language models for chest X-ray interpretation with uncertainty aware radiology reporting11 Jul 2024 0 repositories listed
-
Q-GroundCAM: Quantifying Grounding in Vision Language Models via GradCAM29 Apr 2024 0 repositories listed
-
MedRG: Medical Report Grounding with Multi-modal Large Language Model10 Apr 2024 0 repositories listed
-
Contrastive Region Guidance: Improving Grounding in Vision-Language Models without Training4 Mar 2024 0 repositories listed
-
How to Understand "Support"? An Implicit-enhanced Causal Inference Approach for Weakly-supervised Phrase Grounding29 Feb 2024 0 repositories listed
-
Phrase Grounding-based Style Transfer for Single-Domain Generalized Object Detection2 Feb 2024 0 repositories listed
-
Enhancing the vision-language foundation model with key semantic knowledge-emphasized report refinement21 Jan 2024 0 repositories listed
-
Catalog Phrase Grounding (CPG): Grounding of Product Textual Attributes in Product Images for e-commerce Vision-Language Applications30 Aug 2023 0 repositories listed
-
Read, look and detect: Bounding box annotation from image-caption pairs9 Jun 2023 0 repositories listed
-
ELVIS: Empowering Locality of Vision Language Pre-training with Intra-modal Similarity11 Apr 2023 0 repositories listed
-
CAVL: Learning Contrastive and Adaptive Representations of Vision and Language10 Apr 2023 0 repositories listed
-
21 Mar 2023 0 repositories listed Syntology 1 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified (of 2 harvested samples) · 2 pointer-only (licence)
-
Investigating the Role of Attribute Context in Vision-Language Models for Object Recognition and Detection17 Mar 2023 0 repositories listed
-
Medical Phrase Grounding with Region-Phrase Context Contrastive Alignment14 Mar 2023 0 repositories listed
-
Detailed Annotations of Chest X-Rays via CT Projection for Report Understanding7 Oct 2022 0 repositories listed
-
Lite-MDETR: A Lightweight Multi-Modal Detector1 Jan 2022 0 repositories listed
-
Improving Pre-trained Vision-and-Language Embeddings for Phrase Grounding1 Nov 2021 0 repositories listed
-
Unsupervised Vision-Language Grammar Induction with Shared Structure Modeling29 Sep 2021 0 repositories listed
-
Grounding Plural Phrases: Countering Evaluation Biases by Individuation1 Jun 2021 0 repositories listed
-
13 Apr 2021 0 repositories listed
-
Utilizing Every Image Object for Semi-supervised Phrase Grounding5 Nov 2020 0 repositories listed
-
Propagating Over Phrase Relations for One-Stage Visual Grounding1 Aug 2020 0 repositories listed
-
Language Features Matter: Effective Language Representations for Vision-Language Tasks17 Aug 2019 0 repositories listed
-
Align2Ground: Weakly Supervised Phrase Grounding Guided by Image-Caption Alignment27 Mar 2019 0 repositories listed
-
Neural Sequential Phrase Grounding (SeqGROUND)18 Mar 2019 0 repositories listed
-
PIRC Net : Using Proposal Indexing, Relationships and Context for Phrase Grounding7 Dec 2018 0 repositories listed
-
Dynamic Conditional Networks for Few-Shot Learning1 Sep 2018 0 repositories listed
-
Knowledge Aided Consistency for Weakly Supervised Phrase Grounding11 Mar 2018 0 repositories listed
-
Query-guided Regression Network with Context Policy for Phrase Grounding4 Aug 2017 0 repositories listed
-
19 Nov 2015 0 repositories listed
Syntology lines on 20 of the papers shown; no Syntology record for the others (a paper without an arXiv id cannot be joined to the graph, and absence from the graph layer is not a recorded non-run). “Ran” means the sample executed on a synthesized fixture, not that the paper's result was reproduced; each line links to that paper's sample list. Syntology's record for this page last changed ; when this build read Syntology's graph is in the build record. For agents: get_harvested_code_for_paper(arxiv_id) lists each paper's samples; how to connect.
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections