Papers › Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning

Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning

27 Feb 2023CVPR 2023 1arXiv:2302.14115archive 2025-07-28

Antoine Yang, Arsha Nagrani, Paul Hongsuck Seo, Antoine Miech, Jordi Pont-Tuset, Ivan Laptev, Josef Sivic, Cordelia Schmid

In this work, we introduce Vid2Seq, a multi-modal single-stage dense event captioning model pretrained on narrated videos which are readily-available at scale. The Vid2Seq architecture augments a language model with special time tokens, allowing it to seamlessly predict event boundaries and textual descriptions in the same output sequence. Such a unified model requires large-scale training data, which is not available in current annotated datasets. We show that it is possible to leverage unlabeled narrated videos for dense video captioning, by reformulating sentence boundaries of transcribed speech as pseudo event boundaries, and using the transcribed speech sentences as pseudo event captions. The resulting Vid2Seq model pretrained on the YT-Temporal-1B dataset improves the state of the art on a variety of dense video captioning benchmarks including YouCook2, ViTT and ActivityNet Captions. Vid2Seq also generalizes well to the tasks of video paragraph captioning and video clip captioning, and to few-shot settings. Our code is publicly available at https://antoyang.github.io/vid2seq.html.

PaperPDFConference PDFCode

In Syntology View this paper on Syntology, its page in Syntology's graph. That page lists the repositories linked to the paper, the abstract and the calls for agents.

Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.

For agents, on Syntology's MCP service (how to connect):

Code

google-research/scenic officialjaxSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to TokenLearner: What Can 8 Learned Tokens Do for Images and Videos? (arXiv:2106.11297; that paper's own run record: 3 ran (of which 1 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 0 violated, 3 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified), Learning with Neighbor Consistency for Noisy Labels (arXiv:2202.02200; that paper's own run record: 4 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 4 where Syntology's instrument failed) · 0 unverified), Location-Aware Self-Supervised Transformers for Semantic Segmentation (arXiv:2212.02400; that paper's own run record: 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified) and 3 more (every one in this page's JSON twin); nothing checks that they implement this paper's method. report
KastanDay/video-pretrained-transformer mentioned on GitHubpytorchMITSyntology: not harvested report
antoyang/VidChapters pytorchMITSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to VidChapters-7M: Video Chapters at Scale (arXiv:2309.13952; that paper's own run record: 6 ran (of which 0 constructed an object rather than computing a result; 6 with no instrument failure: 0 honoured, 0 violated, 6 with no contract checked; 0 where Syntology's instrument failed) · 3 unverified); nothing checks that they implement this paper's method. report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

A paper named beside a repository with no sample linked to this paper is shown with that paper's own run record, not this paper's: “ran” means executed on a synthesized input, not that the code is correct, and “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. Papers are listed in arXiv-id order, at most three per repository.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Dense Video CaptioningLanguage ModelingLanguage ModellingSentenceVideo Captioning

Results from the paper archive 2025-07-28

TaskDatasetModelMetricValueRank at snapshotLeaderboardReport
Dense Video Captioning ActivityNet Captions Vid2Seq CIDEr 28 #1 of 12 Archive leaderboard report
Dense Video Captioning ActivityNet Captions Vid2Seq METEOR 17 #1 of 12 Archive leaderboard report
Dense Video Captioning ViTT Vid2Seq CIDEr 43.5 #3 of 4 Archive leaderboard report
Dense Video Captioning ViTT Vid2Seq METEOR 8.5 #3 of 4 Archive leaderboard report
Dense Video Captioning ViTT Vid2Seq SODA 0.135 #3 of 4 Archive leaderboard report
Dense Video Captioning YouCook2 Vid2Seq CIDEr 47.1 #3 of 7 Archive leaderboard report
Dense Video Captioning YouCook2 Vid2Seq METEOR 9.3 #3 of 7 Archive leaderboard report
Dense Video Captioning YouCook2 Vid2Seq SODA 7.9 #3 of 7 Archive leaderboard report
Video Captioning MSR-VTT Vid2Seq CIDEr 64.6 #12 of 24 Archive leaderboard report
Video Captioning MSR-VTT Vid2Seq METEOR 30.8 #12 of 24 Archive leaderboard report
Video Captioning MSVD Vid2Seq CIDEr 146.2 #8 of 16 Archive leaderboard report
Video Captioning MSVD Vid2Seq METEOR 45.3 #8 of 16 Archive leaderboard report

Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.

Methods

Absolute Position EncodingsAdamAttentionBPECLIPDense ConnectionsDropoutLabel SmoothingLayer NormalizationLinear LayerMulti-Head AttentionPosition-Wise Feed-Forward LayerResidual ConnectionSoftmaxTransformer

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections