Source-linked AI summary
Seeing Fast and Slow: Learning the Flow of Time in Videos
Yen-Siang Wu, Rundong Luo, Jingsen Zhu, Tao Tu, Ali Farhadi, Matthew Wallingford, Yu-Chiang Frank Wang, Steve Marschner, Wei-Chiu Ma
TL;DR
Video models often lack reliable temporal-speed reasoning because standard-rate training data provides limited variation in cadence. The paper learns speed understanding from audio-visual and temporal structure, curates a large slow-motion dataset, and uses it for controllable generation and temporal super-resolution, achieving strong results across understanding and generation tasks.
Problem
Current video models often mispredict playback speed and fail to generate motion at precisely specified timings, while standard-frame-rate data lacks broad temporal-speed variation.
Method
The paper uses audio pitch shifts and temporal-resampling equivariance for self-supervised speed understanding, then applies these models to curate slow-motion data for temporal control.
Results
The models achieve strong performance across understanding and generation tasks, including 92.4% speed-change detection accuracy and lifelike motion at controllable playback speeds.
Takeaways & Limitations
The findings establish time as a manipulable dimension in video learning and support temporal forensics, controllable generation, and higher-frame-rate reconstruction.
Takeaways & Limitations
Speed understanding can be misled by limited motion cues or deliberately slow human movement, while generation relies on a pretrained Wan backbone.
Abstract
from arXiv · showhide
How can we tell whether a video has been sped up or slowed down? How can we generate videos at different speeds? Although videos have been central to modern computer vision research, little attention has been paid to perceiving and controlling the passage of time. In this paper, we study time as a learnable visual concept and develop models for reasoning about and manipulating the flow of time in videos. We first exploit the multimodal cues and temporal structure naturally present in videos to learn, in a self-supervised manner, to detect speed changes and estimate playback speed. We then show that these learned temporal reasoning models enable us to curate the largest slow-motion video dataset to date from noisy in-the-wild sources. Such slow-motion footage, typically filmed by high-speed cameras, contains substantially richer temporal detail than standard videos. Using this data, we further develop models capable of temporal control, including speed-conditioned video generation, which produces motion at specified playback speed, and temporal super-resolution, which tranforms low-FPS, blurry videos into high-FPS sequences with fine-grained temporal details. Our findings highlight time as a manipulable, perceptual dimension in video learning, opening doors to temporally controllable video generation, temporal forensics detection, and potentially richer world-models that understand how events unfold over time.
1 Introduction
The paper treats temporal speed as a learnable visual concept, addressing models’ weak ability to perceive playback changes and control motion timing. It uses audio-visual and temporal cues for self-supervised speed understanding, curates slow-motion data, and applies it to temporal generation and super-resolution.
- Motivation: Current models often mispredict playback speed and ignore instructions to generate video at specified timings.The paper attributes this gap partly to training primarily on standard-frame-rate videos.
- Self-supervised temporal cues: Audio pitch shifts provide cross-modal supervision for visually detecting moments when playback speed changes.Speeding up raises pitch, while slowing down lowers it; audio is used during training.
- Self-supervised temporal cues: Temporal resampling supplies a self-supervised signal because predicted playback speed should change proportionally with the resampling factor.This enables speed estimation without clean temporal labels.
- Dataset construction: The learned models automatically curate diverse slow-motion videos that preserve fine temporal details and reduce motion blur.These videos are typically captured at 1000+ FPS and played back at 30 FPS.
- Temporal manipulation: Finetuning on the slow-motion dataset enables lifelike motion at controllable playback speeds and frame interpolation at higher effective frame rates.The approach targets speed-conditioned generation and temporal super-resolution despite motion blur and precise-timing challenges.
- Evaluation: 92% accuracy is achieved on speed-change detection, alongside near-human playback-speed estimation and an 80.3% human-preference win rate for temporal super-resolution.The experiments evaluate understanding and generation tasks with dedicated datasets and human perceptual studies.
2 Related Works
Prior work studies temporal understanding through direction, pace, and frame-rate prediction, while temporal manipulation includes interpolation, motion control, and video editing. This paper extends these lines with broad-range speed-conditioned generation and a large, general-purpose slow-motion dataset.
- Temporal understanding: Arrow-of-time methods determine whether video plays forward or backward, while pace classification uses altered sampling rates as a self-supervised pretext task.SpeedNet extends pace classification toward video speed prediction.
- Temporal understanding: Prior FPS prediction is limited by training data concentrated at standard frame rates and a narrow set of speed values.Pulse-of-Motion used few high-frame-rate videos and modeled speeds in the range 1/2, 1, 2, 4×.
- Temporal manipulation: Frame interpolation increases temporal resolution by synthesizing intermediate frames, using kernels, optical flow, or generative priors to model motion.Recent generative approaches aim to handle complex motion and improve realism.
- Temporal manipulation: Video editing methods provide spatio-temporal control by jointly editing camera motion and temporal progression.These approaches differ from generation-centric synthesis because they manipulate an existing video.
- This paper: Speed-conditioned generation spans normal speed 1.0× to extreme slow motion 0.01× and requires fabricating physical motion at the requested rate.The paper uses SloMo-44K to support high-fidelity generation across this broad range.
- This paper: SloMo-44K is presented as the first large-scale general-purpose slow-motion dataset for learning fine-grained dynamics.It addresses prior datasets’ limitations in scale, frame rate, or scene diversity.
3 Perceiving the Flow of Time
The paper develops models that detect speed changes and estimate playback speed from temporal and multimodal cues, then uses them to annotate in-the-wild slow-motion videos. The resulting dataset contains 44,632 clips and supports temporal learning and manipulation.
- Goals: The section frames speed-change detection and playback-speed estimation as foundations for deeper temporal reasoning and temporal control.These abilities locate whether, when, and by how much playback speed changes.
- Speed-change detection: Audio time–frequency scaling provides supervision because speeding up shifts frequencies upward, while slowing down shifts them downward.The detector is trained with audio-derived labels but operates only on visual input at inference.
- Speed estimation: Temporal resampling trains the speed estimator by enforcing proportional predictions for clips accelerated by factor k.The estimator combines self-supervision with direct regression on videos with known frame rates.
- Speed estimation: Calibration anchors self-supervised speed predictions to absolute playback rates using videos with known ground-truth speeds.Adobe240FPS provides an example source for this calibration.
- Speed estimation: Iterative prediction refines extremely slow-motion estimates by accelerating a clip toward normal speed and re-estimating repeatedly.The method is empirically unrolled three times because subtle motion differences make ultra-slow videos difficult to estimate.
- Dataset construction: The dataset pipeline segments clips by homogeneous playback speed, estimates speed, and adds dense semantic and aesthetic captions.Source videos are collected from online platforms and filtered for quality and unwanted overlays.
- Dataset construction: 44,632 clips and 18 million frames make SloMo-44K the largest curated slow-motion dataset of its kind.The dataset spans diverse real-world scenes, motion patterns, and video durations.
4 Manipulating the Flow of Time
The paper develops explicit temporal control for video generation and temporal super-resolution. It uses speed-annotated data and motion-aware conditioning to generate dynamics at requested speeds and reconstruct sharp, high-FPS video from blurry low-FPS inputs.
- Speed-conditioned video generation: Given an image, text prompt, and target speed, the model generates dynamic content whose physical motion unfolds at the requested temporal rate.The target includes dynamics such as fluid motion, shattering, and vibration that may only be perceptible at particular temporal scales.
- Speed-conditioned video generation: Textual speed modifiers alone often fail to produce noticeably different playback speeds because physical motion cues are entangled with semantic prompt factors.
- Speed-conditioned video generation: The model discretizes target playback speed into logarithmically spaced buckets spanning 0.01× to 1.0×, then injects the bucket representation into the video model.The conditioning is added to timestep embeddings and latent features for stronger temporal control.
- Speed-conditioned video generation: Speed-conditioned generation is visualized through average-frame sharpness and optical-flow magnitude, with faster motion producing blurrier averages and more saturated flow.
- Temporal super-resolution: Temporal super-resolution transforms low-FPS, motion-blurred videos into high-FPS, clear sequences, addressing the harder reconstruction problem caused by longer exposure times.
- Temporal super-resolution: Training uses temporally averaged and subsampled slow-motion footage to synthesize blurry inputs, while paired conditioning enables joint motion deblurring and frame interpolation.The model is based on Wan2.1-VACE with LoRA adapters and performs 8-times upsampling.
5 Experiments
Experiments evaluate speed-change detection, playback-speed estimation, speed-conditioned generation, and temporal super-resolution. Across these tasks, the proposed models outperform baselines, with strong detection accuracy, improved controllability, and high user preference for reconstructed video.
- Speed Change Detection: 92.4% test accuracy, versus 59.5% for Gemini 2.5 and 80.4% for the flow-based baseline, is achieved by the speed-change detector.The model also identifies a speed transition in an X-Men time-freeze scene, supporting video-forensics applications.
- Playback Speed Estimation: The playback-speed estimator significantly outperforms Gemini, SpeedNet, and optical flow while narrowing the gap between human and model performance.
- Speed-conditioned Video Generation: The speed-conditioned generator provides higher slow-motion quality than Wan2.1 and ATI while producing optical-flow magnitudes that follow the requested speed ordering.Smaller playback-speed values generate slower-looking motion than larger values.
- Extreme Temporal Super-Resolution: Over 90% user preference in the Real-input setting accompanies consistent superiority over baselines for temporal super-resolution.The evaluation covers clear, blurred, and real-input settings, including qualitative and user-study comparisons.
- Analysis: Iterative prediction improves accuracy for both the proposed model and VideoLLM, indicating robustness across model types.
- Analysis: Models trained on SloMo-44K outperform those trained only on standard-speed data, while artificial slowdowns produce stuttering artifacts instead of realistic slow motion.
6 Discussion
The discussion presents time as a manipulable dimension for video learning while acknowledging limitations in temporal understanding and generation. The dataset emphasizes diverse contexts, objects, motions, and video lengths.
- Discussion: Speed understanding can be misled by limited motion cues or by people deliberately moving slowly, and generation remains dependent on a pretrained Wan backbone.The authors identify architectural innovation and full fine-tuning as possible areas for improvement.
- Discussion: The work positions temporal modeling as a route toward temporal forensics and richer world models that capture how events unfold.
- Discussion: The dataset spans contextual diversity, varied objects and human or object motions, and a range of video lengths.These properties are illustrated through sample frames, noun and verb word clouds, and a video-length histogram.
B Dataset Details
This section provides additional details on the curation and statistical composition of the SloMo-44K dataset.
- The supplementary section covers the curation process and statistical composition of SloMo-44K.
B.1 Dataset Curation
The curation pipeline expands raw online videos into annotated clips by filtering content, segmenting playback rates, retaining slow motion, and adding speed and scene metadata.
- Video Acquisition and Generic Filtering: The pipeline collects online videos, segments shots, removes text-heavy, CGI, screen-captured, and low-quality content, and filters generic video quality.Sources include YouTube, Vimeo, and Flickr; TransNetv2, OCR, Qwen2.5-VL, and VQA metrics support filtering.
- Slow-Motion–Focused Processing: Speed-change detection segments videos into clips with homogeneous playback rates.This step addresses heterogeneous or unspecified playback speeds in online videos.
- Slow-Motion–Focused Processing: A dedicated slow-motion detector retains clips exhibiting slow motion rather than allowing the dataset to be dominated by standard 1.0× content.Speed segmentation alone does not guarantee that clips contain slow motion.
- Annotation: Each retained clip receives a predicted playback speed and dense captions covering semantic and aesthetic scene attributes.Captions include short and long descriptions, background, style, shot type, lighting, and atmosphere.
B.2 Slow-Motion Video Classification
The paper combines VideoLLM localization with a clip-level classifier to identify genuine slow-motion footage, achieving high precision when their predictions are jointly filtered.
- Dataset Statistics: SloMo-44K spans a wide spectrum of playback speeds.The pseudo-speed histogram indicates broad temporal coverage.
- Approach: The classification pipeline combines VideoLLM localization with a ViT-based classifier to exploit global context and fine-grained visual patterns.The VideoLLM localizes slow-motion segments, while the classifier predicts whether a two-second clip is slow motion.
- Implementation Details: Eight annotators label 2,400 clips as slow motion, not slow motion, or unknown for classifier training and evaluation.Unknown clips are discarded.
- Results: 98% precision and 44% recall result from combined filtering using VideoLLM and classifier predictions.Clips are discarded when VideoLLM slow-motion localization falls below 10% or classifier probability falls below 0.998.
B.3 Dataset Statistics
SloMo-44K covers diverse visual content and temporal scales, supporting learning across varied real-world motions.
- Dataset Statistics: SloMo-44K covers diverse scenes, objects, motions, video lengths, and temporal scales.The dataset overview combines sample frames, caption-derived noun and verb word clouds, video-length statistics, and pseudo-speed annotations.
C Experimental Details
The experimental details define self-supervised speed-estimation equivariance, iterative prediction, and evaluations for controllable generation and temporal super-resolution. Results describe controllable speed changes and clearer intermediate frames, while VBench is limited for slow-motion quality.
- C.1 Speed Estimation: Equivariance requires the estimated speed of a temporally accelerated video to scale by the same factor as the acceleration.For temporal acceleration k, the estimator is expected to satisfy fθ(k · V) = k · fθ(V).
- C.1 Speed Estimation: Three iterative prediction steps substantially improve speed estimates, while additional steps produce negligible changes.The first iteration is especially inaccurate for extremely slow videos, and predictions converge by the third iteration.
- C.2 Speed-conditioned Video Generation: VBench does not accurately capture slow-motion generation quality because it targets standard-speed videos and visual convincingness rather than real-world principles.The paper reports VBench results for completeness but identifies this evaluation boundary.
- C.2 Speed-conditioned Video Generation: The speed-conditioned model produces slower motion at smaller playback-speed values and faster dynamics at larger values.Baseline outputs show minimal variation across speed-control prompt modifiers.
- C.3 Temporal Super-Resolution: The temporal super-resolution model generates clear, temporally consistent intermediate frames from heavily blurred inputs.It outperforms traditional temporal super-resolution methods in the reported qualitative comparisons.
D Comparison to BulletTime and SpaceTimePilot
Unlike editing methods that remap timestamps in existing videos, this work generates motion from scratch at absolute real-world speeds. Its speed-perception component also supports video forensics and construction of the SloMo-44K dataset.
- Absolute-speed generation: BulletTime and SpaceTimePilot edit existing videos through relative time-remapping, whereas this framework generates image-to-video motion conditioned on absolute temporal speed.The generation task requires knowledge of real-world motion speeds rather than merely rescaling an existing temporal axis.
- Real-world temporal detail: Training on SloMo-44K enables modeling complex real-world dynamics across broad temporal granularities, including up to 100× slow motion.The comparison attributes competing methods' narrower capabilities to finetuning only on synthetic video data.