Source-linked AI summary
The "something something" video database for learning and evaluating visual common sense
Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzyńska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, Florian Hoppe, Christian Thurau, Ingo Bax, Roland Memisevic
TL;DR
Still-image datasets and many existing video datasets provide limited evidence for learning fine-grained physical common sense and natural-language-linked visual understanding. The paper introduces an ongoing supervised collection of caption-template videos, currently spanning 108,499 videos and 174 labels, and reports that subtle distinctions make the task extraordinarily difficult for standard architectures. The collection is intended to support increasingly complex fine-grained learning as networks improve, while natural-language labels provide a weaker learning signal than one-hot labels.
Problem
Still-image and high-level video datasets provide limited support for learning fine-grained physical-world understanding and linking visual knowledge with natural language.
Method
The paper constructs an ongoing supervised dataset of crowdworker-acted short videos labeled with natural-language caption templates and adjustable description complexity.
Results
Subtle distinctions in the labels make the resulting classification problem extraordinarily difficult for standard architectures.
Takeaways & Limitations
The collection is designed to support fine-grained visual learning whose solutions require deeper understanding of physical-world situations.
Takeaways & Limitations
The database is an ongoing collection that will continue to grow in response to networks’ ability to learn from it.
Abstract
from arXiv · showhide
Neural networks trained on datasets such as ImageNet have led to major advances in visual object classification. One obstacle that prevents networks from reasoning more deeply about complex scenes and situations, and from integrating visual knowledge with natural language, like humans do, is their lack of common sense knowledge about the physical world. Videos, unlike still images, contain a wealth of detailed information about the physical world. However, most labelled video datasets represent high-level concepts rather than detailed physical aspects about actions and scenes. In this work, we describe our ongoing collection of the "something-something" database of video prediction tasks whose solutions require a common sense understanding of the depicted situation. The database currently contains more than 100,000 videos across 174 classes, which are defined as caption-templates. We also describe the challenges in crowd-sourcing this data at scale.
1. Introduction
Still-image networks miss changes and physical cues that support understanding objects, actions, affordances, and common-sense situations. The “something-something” database uses short, fine-grained videos with textual labels to expose these aspects of the physical world.
- Motivation: Still-image object-recognition datasets do not observe changes in object pose, position, or distance that provide cues about object properties.Multiple views can reveal 3-D geometry and other physical characteristics.
- Motivation: Video motion and deformation cues can reveal rigidity, articulation, softness, stiffness, elasticity, and other material properties.These properties express themselves through changes across multiple views.
- Motivation: Affordances connect object properties to possible actions, while gravity and object permanence are additional physical concepts that still-image tasks may poorly capture.For example, softness can make an object suitable for covering another object.
- Motivation: Fine-grained visual concepts intertwine objects and actions, so predicting one cannot be treated independently of predicting the other.The visual meaning of an action such as “opening” changes substantially with the object involved.
- Motivation: Visual grounding may help address limitations of pure-text understanding because common-sense language puzzles can require spatial, material, and activity knowledge.The cited Winograd example depends on understanding size and the activity of putting one object into another.
2. Related work
Prior work learns physical information from robotics, video prediction, still-image features, or specialized datasets, but each approach leaves gaps in physical or visual grounding. The related literature motivates learning richer world representations from video.
- Physical-world learning: Robotics-based intuitive-physics studies use physical interactions, but their supervision is based on still images showing objects before and after actions.The passage identifies still-image input as a shortcoming of this line of work.
- Video prediction: Video-prediction methods use future frames as labels, but raw-pixel prediction is computationally difficult and high-level feature prediction can inherit still-image limitations.Predicting ImageNet features has not yet shown substantial additional physical-world information.
- Synthetic physical interactions: A game-engine study predicts block-tower collapse and part trajectories from tower images, so its predictions are based on still images rather than videos.The approach is described as a hybrid between learning from video and learning from interactions.
- Fine-grained visual labels: Fine-grained image labels can associate roles with actions and objects, whereas this work targets basic physical concepts expressed in simple phrases.The cited contrasting dataset describes sophisticated cultural concepts such as clipping a sheep’s wool.
3. Learning world models from video
Existing labeled video datasets commonly emphasize high-level global actions or elaborate cultural descriptions rather than low-level physical concepts. The paper motivates crowd-recorded videos as a scalable route toward detailed visual world models.
- Existing video datasets: Action-recognition datasets usually map long videos to a small set of global labels, often allowing prediction from a single cropped image.Such tasks need not require detailed understanding of what happens throughout the sequence.
- Existing video datasets: High-level action labels do not encourage learning motion primitives that encode object properties and intuitive physics.Examples include recognizing sports or activities such as getting out of a car and fighting.
- Existing video datasets: Video-captioning datasets provide elaborate descriptions but typically emphasize high-level cultural aspects and rare or unusual facts and language.Their descriptions summarize relatively long video sequences.
- Existing video datasets: A lower-level physical dataset contains 17,408 videos of a small object set in experiments such as sliding or dropping, with supervision from known physical properties.The paper contrasts that scale and setup with its own larger collection.
- Data collection: Many datasets inherit shortcomings from annotating existing footage, including Hollywood excerpts, motivating crowdworkers who record videos themselves.The paper uses a scalable crowd-sourced video-recording framework and focuses on basic physical concepts.
4. The “something-something” dataset
The dataset uses crowd-recorded short videos paired with compositional natural-language templates to capture fine-grained physical concepts. Its design combines scalable collection, structured labels, contrastive action groups, and a dedicated platform, while acknowledging learning and dataset-bias challenges.
- Dataset construction: 108,499 videos across 174 textual labels form the current dataset, with 2–6-second clips recorded by crowdworkers acting out templates.Workers choose objects and enter noun phrases describing them when uploading videos.
- Dataset construction: 23,137 distinct object names were submitted, with at least a few thousand estimated to represent genuinely distinct objects.The count includes differences in case, stemming, and determiners.
- Natural language labels and curriculum learning: Natural-language templates provide a compositional label space that can range from simple objects and actions toward more complex captions.The current version emphasizes short descriptions containing important parts of speech such as verbs, nouns, and prepositions, with complexity intended to increase over time.
- Grouping and contrastive examples: The collection targets fine-grained physical understanding, but networks may exploit dataset bias through object appearance, hand position, velocity, or camera shake.Action groups place similar actions with minor visual differences together, encouraging discrimination based on the activity and underlying physics.
- Data collection platform: Crowdsourcing at scale requires handling batch submissions, worker-conditional label choices, feedback, and convenience constraints.The authors created a data collection platform for workers and operators to manage these challenges during collection.
5. Baseline experiments
The baseline experiments compare temporal and frame-based video encoders on selected class subsets and find that 3D convolutions generally outperform 2D convolutions, with their combination performing best. Performance becomes substantially harder as classes and subtle distinctions increase.
- Encoding methods: 3D-convolutional networks generally outperform 2D-convolutional networks, while combining both types works best on the classification tasks.Dropout on the first fully connected layer and batch normalization on the last layer also significantly improve training.
- Encoding methods: The baselines include frame-averaged 2D-CNNs, recurrent encodings with an LSTM, stacked or averaged 3D-CNN clip features, and concatenated 2D+3D representations.The 3D-CNN approaches model clips directly, whereas the 2D methods represent individual frames before aggregation.
- Evaluation subsets: 28,198 videos form the 10-class subset, while 53,267 videos form the 40-class subset used to compare the networks.The subsets were constructed from hand-selected classes, beginning with 41 relatively easy classes and adding 30 common classes for the larger benchmark.
- Results: Increasing the number of classes significantly increases task difficulty despite the corresponding growth of the training set.Label ambiguities make naive classification appear weaker, but even top-2 performance indicates a substantial challenge for the tested architectures.
- Results: 88.5% top-1 error and 70.3% top-5 error were obtained on all 174 classes using a 3D-CNN pretrained on the 40 selected classes.This experiment extends evaluation beyond the selected subsets to the full label set.
- Results: Subtle distinctions introduced through grouping, contrastive examples, and related design choices make the problem extraordinarily difficult for standard architectures.The reported results characterize the dataset as challenging even for the baseline encoding methods.
6. Discussion
The paper frames common-sense learning as learning from interactions or from visual observation, and places its dataset in the latter category. It proposes supervised learning on fine-grained labels and identifies the database as an ongoing collection effort.
- Discussion: Common-sense reasoning can be learned through interactions with the world or through observing the world.The interaction-based route relies crucially on advances in robotics, while the paper focuses on visual observation.
- Discussion: The dataset and learning methods belong to the visual-observation approach to learning about the world.The paper distinguishes this approach from learning through physical interaction.
- Discussion: The paper proposes supervised learning on fine-grained labeling tasks rather than unsupervised video prediction.This contrast describes the paper’s learning formulation.
- Discussion: The database is an ongoing collection effort that will continue to grow in response to networks’ ability to learn from it.The authors plan to extend the dataset over time.
A.1. 10-selected classes
The 10-selected-class subset is constructed by manually choosing 41 classes and remapping them into 10 groups.
- A.1. 10-selected classes: The 10 classes are formed by selecting 41 classes by hand and remapping them into 10 groups.Selection uses class definitions and visual inspection of the videos.
A.2. 40-selected classes
The 40-selected-class subset extends the 10-class subset with 30 additional common classes, whose list is provided in Table 6.
- A.2. 40-selected classes: The 40-class subset retains the 10 selected classes and adds 30 additional common classes.The complete class list is shown in Table 6.
A.3. All data - 175 classes
The full dataset organizes 175 caption-template classes into action groups, while experiments use selected 40- and 10-class subsets. The classes cover physical interactions including object movement, placement, pouring, pushing, pulling, and related outcomes.
- 175 caption-template classes are organized into corresponding action groups in the complete dataset.
- A 40-class subset is used for one experiment and its class names are listed in Table 6.
- A 10-class subset maps groups of semantically similar classes from the selected data.
- The action vocabulary includes moving objects relative to one another, touching or opening objects, and picking, piling, plugging, or poking them.
- Other classes describe pouring, pulling, pushing, placing, rolling, sliding, spinning, spilling, spreading, sprinkling, and squeezing.
A.4. Data Collection Platform
The data collection platform supports crowd-workers in selecting classes and uploading corresponding videos with input text. Figures 7 and 8 show the class-selection and upload interfaces.
- Crowd-workers use the platform to select classes and upload corresponding videos.
- The interface includes controls for choosing available classes and entering input text when uploading videos.