Source-linked AI summary
Moments in Time Dataset: one million videos for event understanding
Mathew Monfort, Alex Andonian, Bolei Zhou, Kandan Ramakrishnan, Sarah Adel Bargal, Tom Yan, Lisa Brown, Quanfu Fan, Dan Gutfruend, Carl Vondrick, Aude Oliva
TL;DR
Video understanding needs to recognize short events that vary across agents, settings, time, and sensory modalities. The paper introduces and annotates a one-million-video, 339-class dataset, then evaluates spatial, temporal, and auditory baselines. The dataset exposes challenges from fine-grained actions, background clutter, cross-agent variation, and the complementary but underused information in sound and temporal dynamics.
Problem
Recognizing three-second events is difficult because actions span diverse agents and settings, may reverse temporally, and can depend on sound.
Method
The paper builds a human-annotated dataset of one million three-second videos labeled across 339 classes and evaluates models using spatial, temporal, and auditory modalities.
Results
Baseline performance is limited: I3D reaches 29.51% Top-1 accuracy and 56.06% Top-5 accuracy, while auditory models provide category patterns complementary to visual models.
Takeaways & Limitations
Moments in Time provides a challenging benchmark for models that must handle abstract event labels, varied agents, and visual-auditory-temporal dynamics.
Abstract
from arXiv · showhide
We present the Moments in Time Dataset, a large-scale human-annotated collection of one million short videos corresponding to dynamic events unfolding within three seconds. Modeling the spatial-audio-temporal dynamics even for actions occurring in 3 second videos poses many challenges: meaningful events do not include only people, but also objects, animals, and natural phenomena; visual and auditory events can be symmetrical in time ("opening" is "closing" in reverse), and either transient or sustained. We describe the annotation process of our dataset (each video is tagged with one action or activity label among 339 different classes), analyze its scale and diversity in comparison to other large-scale video datasets for action recognition, and report results of several baseline models addressing separately, and jointly, three modalities: spatial, temporal and auditory. The Moments in Time dataset, designed to have a large coverage and diversity of events in both visual and auditory modalities, can serve as a new challenge to develop models that scale to the level of complexity and abstract reasoning that a human processes on a daily basis.
1 INTRODUCTION
Moments in Time targets short dynamic events across diverse agents, settings, and modalities, creating a difficult recognition problem. The dataset contains one million three-second videos labeled across 339 action classes and supports spatial, temporal, and auditory baselines.
- The dataset contains one million three-second videos, each labeled with one event from 339 action classes.
- Short actions such as picking and running provide the temporal units used to describe longer compound activities.
- Actions such as opening span people, animals, objects, and natural phenomena, while reversing the frames can depict closing instead.
- The dataset is designed to capture visual and audible events with substantial variation within action classes.
- Baseline models address spatial, temporal, and auditory modalities separately and jointly.
2 RELATED WORK
Prior video-understanding research progressed from handcrafted motion representations to large-scale datasets and deep spatiotemporal models. Related work also developed dedicated environmental-sound datasets and multimodal sound-vision representations.
- Large video datasets such as UCF101, ActivityNet, Kinetics, YouTube-8M, Something-Something, AVA, and Charades support action and activity recognition.
- Video-classification methods evolved from space-time descriptors and dense trajectories to CNN-LSTM, two-stream I3D, and non-local 3D networks.
- Environmental-sound research introduced datasets and deep convolutional models for recognizing urban, ambient, and other sound categories.
- Prior work also learned joint sound and vision representations, providing a foundation for multimodal video recognition.
3 THE MOMENTS IN TIME DATASET
Moments in Time is built as a large, balanced, and diverse collection of three-second videos spanning verbs, agents, sound-dependent events, and visual contexts. Its vocabulary and annotation pipeline combine semantic verb selection, web-scale collection, binary human verification, and dataset comparisons.
- Dataset scope: The dataset contains over one million three-second videos covering 339 verbs, with over 1,000 videos associated with each verb.
- Vocabulary construction: Verbs are selected from common-language vocabulary and organized using semantic features from VerbNet, FrameNet, PropBank, and OntoNotes.
- Collection and annotation: Candidate videos are collected from Internet sources, randomly clipped to three seconds, and grouped with their corresponding verbs for annotation.
- Collection and annotation: Annotators verify video-verb pairs with binary Yes/No judgments, training questions, control videos, and a 90% HIT accuracy threshold.
- Dataset statistics: The collection generalizes beyond human agents to animals, objects, and natural phenomena, with class distributions ranging from human-centered actions to storming, roaring, and erupting.
- Dataset statistics: Sound-dependent classes are included alongside visually recognizable events, so purely visual models cannot fully solve the dataset.
- Dataset comparisons: The authors compare action-label scale, videos per class, and recognized object-scene coverage against several large-scale action-recognition datasets.
4 EXPERIMENTS
Experiments evaluate spatial, temporal, auditory, and combined baselines on Moments in Time, revealing strong but limited recognition and varied transfer behavior.
- Baselines: The experiments evaluate separate spatial, temporal, auditory, and spatiotemporal modalities, plus an ensemble that fuses their representations.The ensemble concatenates final hidden-layer features from ResNet50-ImageNet, I3D, and SoundNet before linear SVM classification.
- Baseline Results: 29.51% Top-1 and 56.06% Top-5 accuracy make I3D the best single model on the validation set.The ensemble exceeds I3D on Top-5 accuracy, reaching 57.67%.
- Baseline Results: Models perform better when actions are well-framed and close up, but often fail on fine-grained categories, cluttered backgrounds, or different agents.Missed top-5 detections commonly involve background clutter or poor generalization across humans, animals, and objects.
- Baseline Results: Categories with clear appearances and low intra-class variation perform best, whereas actions spanning many scenes and objects remain difficult.Bowling and surfing are easier examples; covering, slipping, and plugging are more challenging.
- Baseline Results: Auditory models provide category-wise performance that differs from visual models, but the ensemble remains fairly correlated with a single-image spatial model.The authors identify remaining room to exploit temporal and auditory dynamics.
- Cross Dataset Transfer: Pretraining on Moments in Time transfers better to HMDB51, while Kinetics transfers better to UCF101; Moments in Time also improves Something-Something performance.The reported differences align with dataset source and class similarities described by the authors.
5 CONCLUSION
The paper introduces a large three-second video dataset spanning diverse agents and dynamic events. Its abstraction-rich labels make it a difficult benchmark for models requiring visual-temporal understanding.
- 5 CONCLUSION: The dataset contains three-second videos covering dynamic events involving people, animals, objects, and natural phenomena.The dataset is presented as a large-scale collection designed around diverse short events.
- 5 CONCLUSION: Its labels operate at different abstraction levels, since verbs such as “falling” apply across agents and scenarios.The authors identify this diversity as a challenge for computer vision systems.
- 5 CONCLUSION: The dataset is intended as a challenge for developing models that scale to the complexity and abstract reasoning needed to process each video.