Source-linked AI summary
Oops! Predicting Unintentional Action in Video
Dave Epstein, Boyuan Chen, Carl Vondrick
TL;DR
Recognizing whether an action is intentional remains difficult for machine vision, and existing action-recognition data offers limited natural variation in intention. This paper introduces an in-the-wild dataset and recognition, localization, and forecasting tasks, while testing self-supervised cues such as video speed. Speed prediction helps most among the self-supervised tasks, but a substantial gap from human consistency remains.
Problem
Action recognition captures physical motions more readily than intent, while success-biased datasets provide limited realistic variation for learning and evaluating intentionality.
Source: §1 · p. 1Method
The paper introduces an annotated in-the-wild dataset and tasks for classifying intentionality, localizing its transition, and forecasting onset, while exploring self-supervised visual cues including video speed.
Source: §1 · pp. 1–2, §4.1 · p. 4Results
Self-supervised tasks are useful, video-speed prediction helps most, and a large gap remains between model performance and human consistency.
Source: §1 · p. 2Takeaways & Limitations
The dataset and video-speed signal offer a practical benchmark and a way to study intentionality with minimal annotation, though learning human intentions remains a fundamental challenge.
Source: §6 · p. 8A significant performance gap remains between models and human agreement on this dataset.
Source: §6 · p. 8
Abstract
from arXiv · showhide
From just a short glance at a video, we can often tell whether a person's action is intentional or not. Can we train a model to recognize this? We introduce a dataset of in-the-wild videos of unintentional action, as well as a suite of tasks for recognizing, localizing, and anticipating its onset. We train a supervised neural network as a baseline and analyze its performance compared to human consistency on the tasks. We also investigate self-supervised representations that leverage natural signals in our dataset, and show the effectiveness of an approach that uses the intrinsic speed of video to perform competitively with highly-supervised pretraining. However, a significant gap between machine and human performance remains. The project website is available at https://oops.cs.columbia.edu
1. Introduction
The paper targets intentionality, which action recognition captures less directly than physical motions, amid a lack of realistic data with natural variation in intention. It introduces a web-crawled dataset and tasks for recognizing, localizing, and forecasting unintentional action, then studies visual and self-supervised cues.
1. Introduction
- Action-recognition datasets emphasize physical motions and competent outcomes, leaving realistic variation in intention limited for learning and evaluation.The authors argue that success bias makes intentionality difficult to discriminate and localize.Source: §1 · p. 1
- The dataset contains over 50 hours of diverse in-the-wild video collected from web “fail” clips, with temporal labels for the transition into unintentional action.It spans hundreds of scenes and activities.Source: §1 · p. 1
- The dataset supports three tasks: classifying intentionality, localizing the transition to unintentional action, and forecasting its onset.The paper presents these as contributions alongside models that use incidental clues in unlabeled video.Source: §1 · p. 2
- The paper studies visual cues including video-speed prediction, temporal-context predictability, and event order, where intentional action usually precedes unintentional action.Video speed is presented as incidental supervision available in unlabeled video.Source: §1 · p. 2
- Self-supervised tasks are useful, with video-speed prediction helping most, yet models remain far below human consistency.The authors also report that their models do not rely solely on low-level motion clues.Source: §1 · p. 2
2. Related Work
The related work spans action recognition and prediction, while this paper shifts attention from physical activity labels toward goal-directed action and failed goals. It also builds on self-supervised video representations, introducing video-speed estimation as a pretext task.
2. Related Work
- The paper focuses on goal-directed action and failed goals, including errors in planning and execution, rather than only classifying or localizing activities.The cited dataset includes errors associated with limited skill, knowledge, perceptual ability, and environmental factors.Source: §2 · p. 2
- Prior action-recognition work includes visual fusion, two-stream CNNs, 3D CNNs, and temporal reasoning; this paper bases its methods on 3D CNNs.The passage cites prior techniques as,,, and, respectively.Source: §2 · pp. 2–3
- Related action-prediction methods use action-label supervision or unlabeled video representations to anticipate future actions.The passage cites prediction methods as,, and [57] [82].Source: §2 · p. 3
- Self-supervised learning predicts signals naturally present in data, and this paper introduces video-speed estimation as a pretext task for learning video representations.Prior signals include spatial arrangement, context, color, temporal order, future prediction, and motion consistency.Source: §2 · p. 3
3. The _ops_ Dataset
The _ops_ dataset uses real-world web videos to study unintentional human action, with annotations marking when failure begins. It combines quality checks and human-agreement analysis with a benchmark that limits labeled training data.
3. The _ops_ Dataset
- The _ops_ dataset contains 20,338 YouTube fail-compilation videos totaling over 50 hours, spanning diverse actions, environments, intentions, and causes of failure.The videos include physical and social errors, planning and execution mistakes, and environmental factors.Source: §3 · p. 3
3.1. Data Collection and Processing
- The authors collect unconstrained videos from online fail-video channels and separate compilation scenes, discarding clips under 3 seconds or over 30 seconds.They also crop letterbox borders from portrait videos collated in landscape.Source: §3.1 · pp. 3–4
3.2. Annotation
- Workers mark when failure starts, and quality checks repeat annotation, remove clips with unsuitable failure labels or timing, and screen against manually labeled examples.The test set receives a fourth annotation to measure human agreement; median inter-worker standard deviation is about half a second, or 6.6% of clip duration.Source: §3.2 · p. 4
3.3. Dataset Statistics
- The dataset analysis plots distributions of video clip lengths, failure-time labels, standard deviations across three labels from different workers (around half a second on average), and model-predicted action and scene classes, covering varied scenes and activities.Source: §3.3 · p. 4
3.4. Benchmark
- The benchmark separates unlabeled pretraining videos from labeled training and test sets, with 7,368 labeled training videos and 6,739 test videos.Models may use any number of unlabeled videos but only a small number of labeled videos; the tasks include classification, localization, and forecasting.Source: §3.4 · p. 4, Figure 4 · p. 4
4. Intentionality from Perceptual Clues
This section frames intentionality recognition as self-supervised representation learning from incidental clues in unlabeled video, aiming for features that transfer to recognition tasks.
4.1. Predicting Video Speed
- Predicting synthetically altered video frame rates uses speed intrinsic to unlabeled video as supervision for learning action representations.The model uses 16-frame clips sampled at 4, 8, 16, or 30 frames per second; the authors hypothesize that speed prediction encourages motion and expected-event-duration features.Source: §4.1 · p. 4
4.2. Predicting Video Context
- A predictive model estimates the middle clip’s representation from surrounding contextual clips, contrasting it with the target and negative examples.The objective pulls the target representation toward the contextual embedding and pushes it away from other negatives in the mini-batch.Source: §4.2 · p. 5
4.3. Predicting Event Order
- The event-order approach predicts the permutation of three clips sampled 0.5 seconds apart, yielding six possible orders.A network encodes clips, forms pairwise clip relations, and predicts their sort order.Source: §4.3 · p. 5
4.4. Fitting the Classifier
- Self-supervised representations are evaluated with a linear classifier trained on minimal labels to distinguish intentional, unintentional, and transitional action.A clip is transitional when it overlaps the worker-labeled transition point; the approaches use a shared ResNet3D-18 video backbone and 16-frame inputs.Source: §4.4 · p. 5
5. Experiments
Experiments test perceptual clues for recognizing intentionality across classification, transition localization, and anticipation. Video-speed prediction is the strongest self-supervised clue, but supervised models and human agreement remain ahead in key comparisons.
5. Experiments
- Experiments evaluate self-supervised methods on classification, localization, and anticipation, using quantitative ablations and qualitative visualizations to analyze limitations.Source: §5 · p. 6
5.1. Baselines
- Baselines include optical-flow motion magnitude, Kinetics supervision, fixed priors and chance, and held-out human labels; classifiers are generally linear, with some fine-tuning.The motion baseline uses a 100-bin optical-flow histogram and an MLP.Source: §5.1 · p. 6
5.2. Classification
- All self-supervised methods outperform baselines on classification, but full Kinetics supervision performs best overall; video speed ties Kinetics when many labels train a linear layer.Reducing labeled examples by an order of magnitude causes only a minor accuracy drop.Source: §5.2 · p. 6
5.3. Localization
- Video speed is the best self-supervised method for localization, outperforming other self-supervised methods by over 10%; human consistency is 88%, with a substantial model–human gap.Localization accepts predictions within either one second or one quarter second of a ground-truth transition.Source: Table 2 · p. 7, §5.3 · p. 7
5.4. Anticipation
- Video speed gives the best self-supervised anticipation performance, while full Kinetics supervision is about 3% higher.Models predict the three-way label 1.5 seconds into the future.Source: Table 3 · p. 7, §5.4 · p. 7
5.5. Analysis
- Nearest neighbors from the video-speed representation often share activities and objects despite being learned without ground-truth labels.The authors interpret this as evidence that video-speed prediction is promising incidental supervision for activity features.Source: §5.5 · pp. 7–8
- Both video-speed and Kinetics models struggle most at the intentional-to-unintentional boundary; the self-supervised model more often mistakes intentional action for failure onset.The comparison suggests room to improve fine-grained localization in self-supervised models.Source: §5.5 · p. 7, Figure 8 · p. 8
- Models perform worst for environmental causes, unexpected interventions, limited visibility, and limited knowledge, but better in multi-agent scenes.The breakdown uses a manually labeled diagnostic set of 270 videos across nine types.Source: §5.5 · p. 8
6. Discussion
The paper investigates mid-level perceptual clues for recognizing unintentional action and uses video speed for representation learning with minimal annotation. A significant gap from human agreement remains, so learning intentions in video remains a fundamental challenge.
6. Discussion
- A significant gap remains between model performance and human agreement, leaving learning human intentions in video a fundamental challenge.The paper frames this as a remaining challenge for recognizing unintentional action.Source: §6 · p. 8
- Nearest neighbors learned by predicting video frame rate tend to share activities despite substantial appearance variation.Figure 9 visualizes neighbors in the self-supervised feature space.Source: Figure 9 · p. 8
- The diagnostic set groups unintentional-action videos into categories to break down model performance and limitations.Figure 10 presents the performance breakdown; the passage does not report category-level outcomes.Source: Figure 10 · p. 8