Papers › Moments in Time Dataset: one million videos for event understanding

Moments in Time Dataset: one million videos for event understanding

9 Jan 2018arXiv:1801.03150archive 2025-07-28

Mathew Monfort, Alex Andonian, Bolei Zhou, Kandan Ramakrishnan, Sarah Adel Bargal, Tom Yan, Lisa Brown, Quanfu Fan, Dan Gutfruend, Carl Vondrick, Aude Oliva

We present the Moments in Time Dataset, a large-scale human-annotated collection of one million short videos corresponding to dynamic events unfolding within three seconds. Modeling the spatial-audio-temporal dynamics even for actions occurring in 3 second videos poses many challenges: meaningful events do not include only people, but also objects, animals, and natural phenomena; visual and auditory events can be symmetrical in time ("opening" is "closing" in reverse), and either transient or sustained. We describe the annotation process of our dataset (each video is tagged with one action or activity label among 339 different classes), analyze its scale and diversity in comparison to other large-scale video datasets for action recognition, and report results of several baseline models addressing separately, and jointly, three modalities: spatial, temporal and auditory. The Moments in Time dataset, designed to have a large coverage and diversity of events in both visual and auditory modalities, can serve as a new challenge to develop models that scale to the level of complexity and abstract reasoning that a human processes on a daily basis.

PaperPDFCode

In Syntology View this paper on Syntology, its page in Syntology's graph. That page lists the repositories linked to the paper, the abstract and the calls for agents.

Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.

For agents, on Syntology's MCP service (how to connect):

Code

metalbubble/moments_models mentioned on GitHubpytorchBSD-2-ClauseSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to Multi-Moments in Time: Learning and Interpreting Models for Multi-Action Video Understanding (arXiv:1911.00232; that paper's own run record: 8 ran (of which 0 constructed an object rather than computing a result; 8 with no instrument failure: 0 honoured, 0 violated, 8 with no contract checked; 0 where Syntology's instrument failed) · 2 unverified); nothing checks that they implement this paper's method. report
shubhambitsg/activity-recognition mentioned on GitHubpytorchSyntology: not harvested report
thefonseca/predictive-coding mentioned on GitHubSyntology: not harvested report
zhoubolei/moments_models mentioned on GitHubpytorchBSD-2-ClauseSyntology: not harvested report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

A paper named beside a repository with no sample linked to this paper is shown with that paper's own run record, not this paper's: “ran” means executed on a synthesized input, not that the code is correct, and “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. Papers are listed in arXiv-id order, at most three per repository.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Action RecognitionDiversityMultimodal Activity RecognitionTemporal Action Localization

Results from the paper archive 2025-07-28

TaskDatasetModelMetricValueRank at snapshotLeaderboardReport
Action Recognition Something-Something V1 ResNet50 I3D (Moments pretrained) Top 1 Accuracy 50 #52 of 74 Archive leaderboard report
Action Recognition Something-Something V1 ResNet50 I3D (Kinetics pretrained) Top 1 Accuracy 48.6 #60 of 74 Archive leaderboard report
Multimodal Activity Recognition Moments in Time Dataset Ensemble (SVM) Top-1 (%) 31.16 #2 of 6 Archive leaderboard report
Multimodal Activity Recognition Moments in Time Dataset Ensemble (SVM) Top-5 (%) 57.67 #2 of 6 Archive leaderboard report
Multimodal Activity Recognition Moments in Time Dataset I3D Top-1 (%) 29.51 #3 of 6 Archive leaderboard report
Multimodal Activity Recognition Moments in Time Dataset I3D Top-5 (%) 56.06 #3 of 6 Archive leaderboard report
Multimodal Activity Recognition Moments in Time Dataset TRN-Multiscale Top-1 (%) 28.27 #4 of 6 Archive leaderboard report
Multimodal Activity Recognition Moments in Time Dataset TRN-Multiscale Top-5 (%) 53.87 #4 of 6 Archive leaderboard report
Multimodal Activity Recognition Moments in Time Dataset TSN-Flow Top-1 (%) 15.71 #5 of 6 Archive leaderboard report
Multimodal Activity Recognition Moments in Time Dataset TSN-Flow Top-5 (%) 34.65 #5 of 6 Archive leaderboard report
Multimodal Activity Recognition Moments in Time Dataset SoundNet Top-1 (%) 7.60 #6 of 6 Archive leaderboard report
Multimodal Activity Recognition Moments in Time Dataset SoundNet Top-5 (%) 18.00 #6 of 6 Archive leaderboard report

Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.

Methods

1x1 ConvolutionAverage PoolingBatch NormalizationBottleneck Residual BlockConvolutionGlobal Average PoolingKaiming InitializationMax PoolingReLUResidual BlockResidual Connection

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections