Source-linked AI summary
SpeedNet: Learning the Speediness in Videos
Sagie Benaim, Ariel Ephrat, Oran Lang, Inbar Mosseri, William T. Freeman, Michael Rubinstein, Michal Irani, Tali Dekel
TL;DR
The paper asks whether machines can infer whether object motion is natural, sped up, or slowed down. It introduces SpeedNet, a self-supervised binary classifier trained on normal and 2× playback, then applies its learned representation to speediness prediction, adaptive speedup, action recognition, and retrieval. The model works beyond motion magnitude, supports more natural variable speedup, and yields competitive action-recognition and promising retrieval results.
Problem
The paper asks whether a machine can reliably learn object-motion priors to distinguish natural-speed motion from sped-up or slowed-down motion.
Method
SpeedNet is trained self-supervised to classify normal versus 2× playback, then used to predict arbitrary speediness rates and learn space-time features.
Results
SpeedNet predicts speediness on complex natural motions, enables adaptive speedup, and provides competitive self-supervised action recognition plus promising cross-video retrieval.
Takeaways & Limitations
SpeedNet’s speed-classification features support adaptive video playback, self-supervised action recognition, and video clip retrieval.
Takeaways & Limitations
Extreme camera or object motion can challenge SpeedNet because speediness correlates with motion magnitude, and the training proxy does not always reflect naturalness.
Abstract
from arXiv · showhide
We wish to automatically predict the "speediness" of moving objects in videos---whether they move faster, at, or slower than their "natural" speed. The core component in our approach is SpeedNet---a novel deep network trained to detect if a video is playing at normal rate, or if it is sped up. SpeedNet is trained on a large corpus of natural videos in a self-supervised manner, without requiring any manual annotations. We show how this single, binary classification network can be used to detect arbitrary rates of speediness of objects. We demonstrate prediction results by SpeedNet on a wide range of videos containing complex natural motions, and examine the visual cues it utilizes for making those predictions. Importantly, we show that through predicting the speed of videos, the model learns a powerful and meaningful space-time representation that goes beyond simple motion cues. We demonstrate how those learned features can boost the performance of self-supervised action recognition, and can be used for video retrieval. Furthermore, we also apply SpeedNet for generating time-varying, adaptive video speedups, which can allow viewers to watch videos faster, but with less of the jittery, unnatural motions typical to videos that are sped up uniformly.
1. Introduction
The paper studies whether machines can infer whether object motion is natural, sped up, or slowed down, using SpeedNet trained without manual labels. It applies the learned model to speediness prediction, adaptive video speedup, action recognition, and video retrieval.
- Humans use prior knowledge of typical object dynamics to notice unnatural fast or jittery motion in sped-up videos.
- SpeedNet classifies whether an object moves at normal speed or faster than normal, treating speediness as a binary learning problem.The model receives L frames from an L-fps video and predicts whether they depict 1 second or more than 1 second of movement.
- The self-supervised model is trained on natural videos and can predict arbitrary speediness rates, including slowed-down and sped-up motion.Training uses normal and temporally modified videos without manual labels.
- Speediness differs from motion magnitude: SpeedNet remains stable across changing object-camera distances while recognizing 2× walking as faster-than-normal.Static segments are classified as normal speed, even in 2× playback.
- The method generates adaptive, time-varying speedups by speeding objects more when their speediness score is low, reducing jitter relative to uniform speedup.It computes a smooth per-frame speedup curve for a desired global speedup factor.
- SpeedNet learns space-time representations that support self-supervised action recognition and video retrieval, with competitive benchmark performance and advantages over Kinetics-pretrained methods.
2. Related work
Related work covers playback-speed classification, non-uniform video time remapping, and self-supervised learning from video properties. SpeedNet differs by targeting positive speed changes across general videos and using intrinsic motion speed for multiple applications.
- Prior sports-replay methods detect playback speed using domain-specific analysis and supervised learning, whereas SpeedNet is designed for general videos.
- Earlier time-remapping methods optimize non-uniform sampling using visual objectives such as frame-pair errors or frame importance.
- SpeedNet focuses on detecting regions played slower than their natural speed to optimize varying playback rates and reduce noticeable speedup artifacts.
- Video self-supervision has used cycle consistency, frame-order puzzles, future prediction, and optical-flow prediction as learning signals.
- Methods predicting the arrow of time classify playback direction or speeds between −1 and +1, while SpeedNet discriminates between different positive video speeds.
3. SpeedNet
SpeedNet is a self-supervised binary classifier trained to distinguish normal from 2× playback while learning space-time features. Its training design addresses shortcut cues, and its architecture pools spatially dominant motion across the full temporal segment.
- SpeedNet classifies video segments as normal speed or 2× speed by predicting whether L frames contain 1 or 2 seconds of real-world movement.
- The 1×/2× proxy does not always match naturalness because sped-up slow walking may remain natural and static scenes show no motion at either rate.
- Binary classification is preferred to playback-rate regression because the goal is natural versus unnatural motion, not exact rate estimation.
- Training is self-supervised: each video segment supplies normal-speed and temporally subsampled sped-up versions without manual labels.
- Spatial, temporal, and same-batch augmentations reduce reliance on compression and other artificial shortcuts during training.
- The fully convolutional network preserves temporal resolution, reduces spatial resolution, and uses global max spatial pooling plus global average temporal pooling.This emphasizes the dominant moving object while integrating motion across the entire segment.
4. Adaptive video speedup
The method adaptively changes playback speed over time by using SpeedNet’s local speediness predictions while enforcing a target overall speedup and smooth playback.
- From predictions to speedup scores: SpeedNet evaluates sliding windows from exponentially sped-up versions to estimate the probability that each timestep is at normal speed.Each window’s prediction is assigned to its middle frame, producing a temporally varying prediction curve for every speedup factor.
- From predictions to speedup scores: The predictions are interpolated, thresholded, multiplied by their corresponding speedup factors, and combined by taking the maximum permissible local speedup.The resulting vector contains the greatest speedup at each timestep that remains classified as not sped-up.
- From predictions to speedup scores: A threshold is selected from nine values, ρ ∈ {0.1, . . . 0.9}, to make the resulting overall speedup closest to the desired target.
- Optimizing for adaptive speedup: The optimization balances speeding frames according to their estimated score, matching the user’s overall rate, and regularizing the speed curve’s smoothness.The optimal curve S* is then used to play the video adaptively.
5. Experiments
Experiments evaluate SpeedNet on Kinetics and NFS, examine augmentation and training choices, and compare its predictions with optical-flow features and qualitative examples.
- Datasets and evaluation: SpeedNet is evaluated on Kinetics and NFS using 16-frame clips that are either unchanged or uniformly sped up.Kinetics clips are recorded at 25 fps, while NFS videos are manipulated to create normal-speed and sped-up conditions.
- Prediction examples: Figure 4 plots sped-up probabilities over time for normal-speed and 2× gymnast videos, with blue and red curves respectively.Segments from the video are positioned on the plot; slight camera motion is classified as normal speed in both versions.
- Ablation study: Training without temporal, spatial, and same-batch augmentations encourages artificial shortcuts that produce misleadingly high Kinetics accuracy.
- Ablation study: 96.9% to 75.6%: using the three training measures reduces Kinetics accuracy while improving cross-dataset generalization, leaving only a 2% Kinetics–NFS gap.On NFS, shortcut-based training reaches 57.4%; chance level is 50%, and perfect accuracy is not expected for inherently ambiguous clips.
- Comparison to optical flow: 55% accuracy: a model using only per-frame mean optical-flow magnitude performs poorly on both Kinetics and NFS.The flow baseline is confounded by object distance from the camera, while SpeedNet can also fail under extreme camera or object motion.
5.2. Generalization to arbitrary speediness rates
SpeedNet generalizes beyond its normal-speed versus 2× training proxy to videos containing slow-motion effects and complex natural actions.
- Generalization to arbitrary speediness rates: SpeedNet accurately identifies normal-speed and slowed-down segments in real-world videos containing natural camera motion and complex human actions.Examples include ballet, Olympic gymnastics, skiing, and other downloaded web videos.
5.3. Adaptive speedup of real-world videos
The adaptive speedup method is evaluated against uniform speedup on videos whose objects vary substantially in speediness, using a user preference study.
- User study: Five YouTube videos with large within-video differences in object speediness were processed with adaptive and corresponding uniform speedups.Thirty users saw the alternatives in random order and selected the version that looked better.
- User study: 30 users: adaptive speedup was preferred by a clear margin over uniform speedup for all five presented videos.
5.4. SpeedNet for self-supervised tasks
SpeedNet’s learned representation supports self-supervised action recognition and video retrieval, capturing motion behavior beyond action-class similarity.
- Action recognition: Fine-tuning SpeedNet on UCF101 or HMDB51 significantly improves action recognition over random initialization.
- Action recognition: SpeedNet outperforms other self-supervised methods pre-trained on Kinetics for action recognition on UCF101 and HMDB51.
- Nearest neighbor retrieval: SpeedNet clip embeddings use max and average-pooled space-time activations as 1024D feature vectors for nearest-neighbor retrieval.
- Nearest neighbor retrieval: The retrieval experiments search for similar 16-frame clips within longer videos and across video collections using cosine similarity.
- Nearest neighbor retrieval: The embeddings emphasize object behavior’s type and speed, which can differ from the video’s action class.
- Nearest neighbor retrieval: Recall-at-topK results are competitive with other self-supervised methods and only slightly below Xu et al.’s method.
5.5. Visualizing salient space-time regions
CAM visualizations show that SpeedNet attends to salient movers and can distinguish spatially separate regions playing at different speeds.
- CAM heat maps identify space-time regions contributing to SpeedNet’s speediness predictions.Red and blue indicate high and low activation, respectively.
- The model attends to dominant movers during complex actions, including flips and articulated motions, even with significant camera motion.
- In a split-speed video, SpeedNet accurately distinguishes normal-speed and slow-motion regions within the same frame.The visualization maps normal speed to blue and slow motion to red.
6. Conclusion
The paper presents SpeedNet as a self-supervised model that learns sophisticated motion priors and supports adaptive speedup, action recognition, and video retrieval.
- SpeedNet classifies videos as normal-speed or twice-speed and learns high-level object-motion priors beyond motion magnitude.
- The learned model supports adaptive video speedup intended to look more natural than uniform speedup.
- SpeedNet features serve as self-supervised pre-training for action recognition and as a feature extractor for video clip retrieval.
A. Adaptive speedup optimization
Adaptive speedup is optimized by balancing speediness-dependent playback, the desired overall rate, and temporal smoothness.
- Espeed encourages frame-wise speedup to follow the estimated speediness score, favoring lower rates for speediness-high frames.
- The rate regularization term prevents high spikes in the estimated speedup curve, while Esmooth imposes smoothness.
- Erate constrains the video’s overall speedup to match the user’s desired rate Ro.
- The optimization has a closed-form linear-system solution, but the authors minimize it with a TensorFlow optimizer and play the video using S*.