Browse State-of-the-Art › 3D dense captioning
3D dense captioning
13 papers with code · 2 benchmarks · 2 datasets archive 2025-07-28
Dense captioning in 3D point clouds is an emerging vision-and-language task involving object-level 3D scene understanding. Apart from coarse semantic class prediction and bounding box regression as in traditional 3D object detection, 3D dense captioning aims at producing a further and finer instance-level label of natural language description on visual appearance and spatial relations for each scene object of interest.
Description from the archive archive 2025-07-28.
Benchmarks archive 2025-07-28
2 leaderboard tables shown for this task, 2 with rows (a “benchmark” on this site is a table with at least one row, as on /sota), ordered by row count. “Best model” is the first row in the archive's own order at snapshot; nothing is re-ranked here and metric direction is not recorded in the archive. PwC's Trend sparklines are not in the archive, so that column is omitted.
| Dataset | Best model (first row in archive order) | Paper | Code | Syntology | Compare |
|---|---|---|---|---|---|
| ScanRefer Dataset (12 rows) | 3D CoCa | 3D CoCa: Contrastive Learners are 3D Captioners | code | — | Compare |
| Nr3D (10 rows) | 3D CoCa | 3D CoCa: Contrastive Learners are 3D Captioners | code | — | Compare |
Syntology column: samples harvested from the paper's repositories and executed on synthesized fixtures; “ran” is not a correctness claim and does not order the table. Each line reads “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the instrument figure counts failures of Syntology's instrument, not of the code. When the archive marks a repository official for the row's paper, the line starts with that repository's state (“official” is the archive's flag, not a verdict on who wrote the code; “community repositories only” when every sample that ran came from a community repository, “official: no sample here; runs from other or unrecorded repositories” when some came from a repository the paper names or has in its text, or from none recorded); hover it for the repositories the samples that ran came from. A dash means no Syntology record for that paper, not a recorded non-run. Each “ran” line links to that paper's sample list. Syntology's record for this page last changed ; when this build read Syntology's graph is in the build record. For agents: get_harvested_code_for_paper(arxiv_id) lists each paper's samples; how to connect.
Libraries
Not in the archive: the export carries no per-task library table, so there is nothing to show at snapshot 2025-07-28.
Datasets archive 2025-07-28
2 datasets whose archive record lists this task, ordered by the archive's paper count.
Subtasks archive 2025-07-28
No subtask under this task in the archive's task tree.
Parent tasks archive 2025-07-28
Papers archive 2025-07-28
archive papers tagged: 26 · with a code link: 13 · where Syntology ran a sample: 6 (6 with a run with no instrument failure, 0 where every run was a failure of Syntology's instrument) Syntology
Show: all tagged papersonly where code ran (6 of 26 tagged: 6 with a run with no instrument failure, 0 where every run was a failure of Syntology's instrument)
Page 1 of 1: papers 1 to 26 of 26, in archive order: by repositories listed in the archive (most first), then newest first, not by stars (the archive holds no stars, so PwC's “Social” and “Latest” sorts cannot be reproduced). Papers that list no repository come after every paper that lists one.
Papers without a page here are shown as plain text. A Syntology line reads “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the figure “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. When the archive marks a repository official for the paper, the line starts with that repository's state (the archive's flag, not a verdict on who wrote the code); hover it for the repositories the samples that ran came from.
-
26 Nov 2024 2 repositories listed Syntology official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 0 where Syntology's instrument failed) · 4 unverified (of 6 harvested samples)Generating high-fidelity 3D content from text prompts remains a significant challenge in computer vision due to the limited size, diversity, and annotation depth of the existing datasets.
-
13 Apr 2025 1 repository listed3D captioning, which aims to describe the content of 3D scenes in natural language, remains highly challenging due to the inherent sparsity of point clouds and weak cross-modal alignment in existing methods.
-
17 Apr 2024 1 repository listed3D Visual Grounding (3DVG) and 3D Dense Captioning (3DDC) are two crucial tasks in various 3D applications, which require both shared and complementary information in localization and visual-language relationships.
-
28 Mar 2024 1 repository listed Syntology official (archive's flag): 12 ran · 12 ran (of which 0 constructed an object rather than computing a result; 6 with no instrument failure: 4 honoured, 1 violated, 1 with no contract checked; 6 where Syntology's instrument failed) · 4 unverified (of 16 harvested samples) · 16 pointer-only (licence)However, the exploration of 3D dense captioning in outdoor scenes is hindered by two major challenges: 1) the domain gap between indoor and outdoor scenes, such as dynamics and sparse visual inputs, makes it difficult…
-
1 Jan 2024 1 repository listedRecent progress in Large Multimodal Models (LMM) has opened up great possibilities for various applications in the field of human-machine interactions.
-
30 Nov 2023 1 repository listed Syntology official (archive's flag): 6 ran · 6 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 2 honoured, 0 violated, 1 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified (of 6 harvested samples) · 5 pointer-only (licence)However, developing LMMs that can comprehend, reason, and plan in complex and diverse 3D environments remains a challenging topic, especially considering the demand for understanding permutation-invariant point cloud 3D…
-
18 Nov 2023 1 repository listed Syntology official (archive's flag): 11 ran · 11 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 0 honoured, 1 violated, 3 with no contract checked; 7 where Syntology's instrument failed) · 2 unverified (of 13 harvested samples) · 9 pointer-only (licence)However, several significant challenges remain: (i) most of these models rely on 2D images yet exhibit a limited capacity for 3D input; (ii) these models rarely explore the tasks inherently defined in 3D world, e. g.…
-
6 Sep 2023 1 repository listedMoreover, we argue that object localization and description generation require different levels of scene understanding, which could be challenging for a shared set of queries to capture.
-
6 Jan 2023 1 repository listed Syntology official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 0 where Syntology's instrument failed) · 2 unverified (of 3 harvested samples)Compared with prior arts, our framework has several appealing advantages: 1) Without resorting to numerous hand-crafted components, our method is based on a full transformer encoder-decoder architecture with a learnable…
-
1 Jan 2023 1 repository listedThe current approaches for 3D visual reasoning are task-specific, and lack pre-training methods to learn generic representations that can transfer across various tasks.
-
22 Apr 2022 1 repository listedDense captioning in 3D point clouds is an emerging vision-and-language task involving object-level 3D scene understanding.
-
10 Mar 2022 1 repository listed Syntology official (archive's flag): 8 ran · 8 ran (of which 0 constructed an object rather than computing a result; 7 with no instrument failure: 3 honoured, 0 violated, 4 with no contract checked; 1 where Syntology's instrument failed) · 1 unverified (of 9 harvested samples) · 3 pointer-only (licence)3D dense captioning is a recently-proposed novel task, where point clouds contain more geometric information than the 2D counterpart.
-
2 Mar 2022 1 repository listedThus, a more faithful caption can be generated only using point clouds during the inference.
-
3D Spatial Understanding in MLLMs: Disambiguation and Evaluation9 Dec 2024 0 repositories listedMultimodal Large Language Models (MLLMs) have made significant progress in tasks such as image captioning and question answering.
-
3D Scene Graph Guided Vision-Language Pre-training27 Nov 2024 0 repositories listed3D vision-language (VL) reasoning has gained significant attention due to its potential to bridge the 3D physical world with natural language descriptions.
-
Hint-AD: Holistically Aligned Interpretability in End-to-End Autonomous Driving10 Sep 2024 0 repositories listedEnd-to-end architectures in autonomous driving (AD) face a significant challenge in interpretability, impeding human-AI trust.
-
14 Aug 2024 0 repositories listed3D dense captioning is a task to localize objects in a 3D scene and generate descriptive sentences for each object.
-
13 Aug 2024 0 repositories listed3D dense captioning is a task involving the localization of objects and the generation of descriptions for each object in a 3D scene.
-
1 Aug 2024 0 repositories listed3D dense captioning aims to semantically describe each object detected in a 3D scene, which plays a significant role in 3D scene understanding.
-
A Comprehensive Survey of 3D Dense Captioning: Localizing and Describing Objects in 3D Scenes12 Mar 2024 0 repositories listedThree-Dimensional (3D) dense captioning is an emerging vision-language bridging task that aims to generate multiple detailed and accurate descriptions for 3D scenes.
-
UniT3D: A Unified Transformer for 3D Dense Captioning and Visual Grounding1 Dec 2022 0 repositories listedPerforming 3D dense captioning and visual grounding requires a common and shared understanding of the underlying multimodal relationships.
-
8 Oct 2022 0 repositories listed3D dense captioning, as an emerging vision-language task, aims to identify and locate each object from a set of point clouds and generate a distinctive natural language sentence for describing each located object.
-
19 Jun 2022 0 repositories listedThe step function is one of the simplest and most natural activation functions for deep neural networks (DNNs).
-
1 Jan 2022 0 repositories listedObserving that the 3D captioning task and the 3D grounding task contain both shared and complementary information in nature, in this work, we propose a unified framework to jointly solve these two distinct but closely…
-
2 Dec 2021 0 repositories listedRecent studies on dense captioning and visual grounding in 3D have achieved impressive results.
-
3 Dec 2020 0 repositories listedWe introduce the task of dense captioning in 3D scans from commodity RGB-D sensors.
Syntology lines on 6 of the papers shown; no Syntology record for the others (a paper without an arXiv id cannot be joined to the graph, and absence from the graph layer is not a recorded non-run). “Ran” means the sample executed on a synthesized fixture, not that the paper's result was reproduced; each line links to that paper's sample list. Syntology's record for this page last changed ; when this build read Syntology's graph is in the build record. For agents: get_harvested_code_for_paper(arxiv_id) lists each paper's samples; how to connect.
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections