Source-linked AI summary
Learning Long-term Motion Embeddings for Efficient Kinematics Generation
Nick Stracke, Kolja Bauer, Stefan Andreas Baumann, Miguel Angel Bautista, Josh Susskind, Björn Ommer
TL;DR
The paper addresses the inefficiency of modeling multiple possible futures through full video synthesis and the limitations of motion representations tied to flow, sparse tracks, or appearance. It learns a highly compressed long-term motion embedding from tracker trajectories, then uses conditional flow matching to generate goal-directed motion latents. The resulting motion distributions outperform video models and task-specific approaches while supporting efficient long-horizon generation.
Problem
Full video synthesis is prohibitively inefficient for exploring possible futures, while existing representations either capture limited motion information or entangle motion with appearance.
Method
The framework learns a long-term motion embedding from tracker-derived trajectories with 64× temporal compression, then trains a conditional flow-matching model in that space.
Results
The generated motion distributions significantly outperform those of modern video models and task-specific approaches across multiple benchmarks.
Takeaways & Limitations
The motion embedding provides a compact, expressive, and flexible representation for efficient long-horizon kinematics generation conditioned on text prompts or spatial pokes.
Takeaways & Limitations
Comparisons with video models require tracking their generated videos, which can introduce correspondence loss and inconsistent motion estimates.
Abstract
from arXiv · showhide
Understanding and predicting motion is a fundamental component of visual intelligence. Although modern video models exhibit strong comprehension of scene dynamics, exploring multiple possible futures through full video synthesis remains prohibitively inefficient. We model scene dynamics orders of magnitude more efficiently by directly operating on a long-term motion embedding that is learned from large-scale trajectories obtained from tracker models. This enables efficient generation of long, realistic motions that fulfill goals specified via text prompts or spatial pokes. To achieve this, we first learn a highly compressed motion embedding with a temporal compression factor of 64x. In this space, we train a conditional flow-matching model to generate motion latents conditioned on task descriptions. The resulting motion distributions outperform those of both state-of-the-art video models and specialized task-specific approaches.
1. Introduction
The paper introduces a compact long-term motion embedding that represents scene-level kinematics beyond sparse tracks or frame-to-frame flow. A conditional generator then operates in this space to produce goal-directed motion efficiently across diverse tasks.
- Existing approaches emphasize low-level flow or sparse tracks, while video models entangle motion with appearance and require costly per-pixel temporal modeling.
- The proposed embedding aggregates global kinematic structure across trajectories into a continuous scene-level representation that supports reasoning and generation beyond observed samples.
- The framework first maps sparse tracker-derived motion samples into a spatially queryable latent embedding for dense, context-aware motion prediction.
- A conditional flow-matching model generates plausible, goal-directed motion in the learned space from text or spatial queries.
- The approach delivers efficiency and reasoning gains across diverse motion tasks, outperforming raw tracks and enabling controllable generation with orders-of-magnitude greater compression than video models.
- Evaluations on open-set internet videos and robotics benchmarks report better performance than specialized trajectory predictors and video-based baselines, with substantially greater efficiency.
2. Related Work
Related work spans compact motion representations, video models that implicitly learn motion with appearance, and explicit motion-conditioned generation. The paper positions its approach as a semantic, structured motion space for open-domain planning and controllable kinematics.
- Motion Representations: Prior motion-representation methods include stochastic VAEs, masked autoencoders, vector quantization, and trajectory autoencoders for reconstructing or predicting motion.
- Motion Representations: The paper instead targets a semantic and structured motion space that captures generic kinematics for open-domain planning.
- Motion in Video Models: Modern video models learn motion implicitly alongside appearance, entangling kinematic reasoning and control with texture and lighting.
- Motion in Video Models: The approach encodes sparse trajectories into a latent space that supports dense, temporally consistent motion reconstruction at arbitrary spatial queries.
- Motion-conditioned Video Generation: Motion-conditioned video methods generate optical flow, depth, or manipulated motion vectors as intermediate controls before synthesizing frames.
- Goal-Conditioned Motion Generation: Goal-conditioned trajectory models predict explicit trajectories, whereas this work learns a compact, semantically grounded generative motion prior in latent space.
3. Method
The method learns a compact latent motion space from sparse tracker trajectories, then generates motion latents with conditional flow matching. The learned space supports dense reconstruction, arbitrary spatial queries, and conditioning on text or pokes.
- Learning Motion Spaces: Tracks are represented as sequences of normalized x, y coordinates over time, providing the input motion samples.Each trajectory records the motion of one tracked point in normalized image coordinates.
- Learning Motion Spaces: The first stage trains a variational autoencoder to compress sparse trajectories and reconstruct motion from a latent grid.The encoder processes trajectories with first-frame features, while the decoder reconstructs trajectories from latent representations.
- Learning Motion Spaces: The latent grid summarizes trajectory kinematics and enables decoding motion at arbitrary spatial positions, including points not used during encoding.Query tokens combine spatial and temporal information with the latent grid and first-frame features to predict motion coordinates.
- Learning Motion Spaces: The VAE uses Fourier-embedded trajectory samples, 3D RoPE, global self-attention, and cross-attention to first-frame features.These components jointly encode time, trajectory identity, spatial position, and scene context.
- Reasoning in Motion Spaces: A conditional flow-matching model operates directly in latent motion space to generate diverse plausible motions conditioned on text prompts, pokes, and scene context.It learns a vector field that transports noise toward encoded motion latents, with conditioning integrated through cross-attention.
4. Experiments
The experiments evaluate compressed motion representations for semantic compression, goal-conditioned generation, motion reasoning, and robotics planning. Across open-domain video and closed-domain robotics settings, the approach shows efficient, diverse motion generation and strong task performance.
- Experimental Setup: The study evaluates compression, motion generation, reasoning, and planning in open-domain video and closed-domain robotics settings.The evaluations cover plausible motion generation, semantic structure, goal achievement, and action execution.
- Open-Domain Motion Generation: Table 1 compares explicit motion predictors trained on general video data under different conditioning densities, with Track2Act reported only for dense conditioning.The table caption states that the approach is significantly faster as well as more accurate than the compared methods.
- Semantic Motion Compression: 64× temporal compression improves motion-generation quality, inference efficiency, and semantic organization under a fixed compute budget.Increasing compression monotonically improves kNN accuracy, indicating that semantically similar motions become more closely grouped.
- Open-Domain Motion Generation: The model consistently outperforms flow-based and trajectory-based baselines across conditioning sparsities while producing diverse, coherent, physically plausible motions.The comparison uses varying conditioning densities, and qualitative examples include multiple plausible eagle-flight hypotheses and varied motion regimes.
- Closed-Domain Robotics: In LIBERO, generated motion embeddings drive a policy head that predicts robot actions from rolling long-horizon plans updated after each observation.The policy head receives only generated motion embeddings, separating task-conditional planning from action prediction.
- Comparison to Generative Video Models: Compared with video models, the method is evaluated using motion metrics in sample-matched and time-matched regimes, with the sample-matched results outperforming Wan and showing a larger gap for Veo.Video outputs are tracked with CoTracker3 to obtain comparable trajectories, while tracking errors can affect the comparison.
5. Conclusion
The paper introduces a highly compressed motion embedding and a conditional flow-matching generator for efficient, long-horizon scene-dynamics modeling. Across multiple benchmarks, the generated motion distributions outperform modern video models and task-specific approaches.
- Conclusion: The framework models long-term scene dynamics through a learned motion embedding with 64× temporal compression.The embedding is learned from large-scale tracker-derived trajectories and captures long-term kinematic structure.
- Conclusion: Across multiple benchmarks, the generated motion distributions significantly outperform modern video models and task-specific approaches.The conclusion characterizes the motion embeddings as compact, expressive, and flexible for kinematics modeling.
A. Additional Evaluation Details
The evaluation covers trajectory prediction, robot action prediction, video-model comparisons, and curated real-world motion data. Protocols align sequence lengths and sampling conditions to compare motion quality and diversity fairly.
- LIBERO Trajectory Prediction: Across all LIBERO tasks, the model outperforms discriminative and generative trajectory-prediction baselines.Generative baselines are evaluated with Min MSE for k = 8 samples and with Single for k = 1.
- LIBERO Action Prediction: The LIBERO evaluation trains a policy head to predict 7-dimensional robot actions from generated motion embeddings.The policy head is a shallow 6-layer transformer, and the motion planner uses 10 sampling steps during training and rollouts.
- Video Models: The video comparison uses Wan and Veo 3 as start-and-end-frame-conditioned generative baselines.The evaluation focuses on a curated Pexels subset because video-model sampling is computationally and monetarily expensive.
- Dataset: The curated evaluation set contains 68 videos spanning humans, animals, vehicles, landscapes, and urban scenes with varied object counts and motion magnitudes.Videos retain clearly visible primary moving objects throughout each sequence.
- Ground Truth: Each ground-truth sequence contains 64 frames and 40 dynamic tracks sampled from 1024 tracked query points.Original videos run at 24–30 fps, and the 64-frame windows correspond to roughly 4–5 seconds.
- Video Evaluation: Video baselines generate 8 samples per start–end pair, after which tracks are extracted and downsampled to 64 frames for metric computation.Wan produces 81 frames at 12 fps and Veo 3 produces 96 frames at 24 fps.
B. Implementation Details
The implementation uses a two-stage latent-motion pipeline trained on 10M open-set video clips: a VAE compresses tracking data, and a flow-matching planner generates task-conditioned latent motion.
- Architecture: The first stage is a VAE that compresses tracker-derived motion into a latent grid, while the second stage generates task-conditioned latents with flow matching.Both stages use TAPNext tracks from 10M diverse open-set video clips.
- VAE: The VAE uses dual encoders with 12 self-attention and 12 cross-attention layers, conditioning tracking tokens on frozen DINOv2 image features during initial training.Tracker points are Fourier-embedded before transformer processing.
- Motion Planner: The motion planner is a 24-layer transformer that performs flow matching in the learned latent space.It processes image and latent-grid tokens through shared self-attention, with task specifications supplied through cross-attention.
- Positional Encoding: The models use 2D RoPE for the planner’s latent grid, while the VAE uses 3D RoPE for its spatiotemporal inputs.Task specifications include Fourier-embedded pokes or text embeddings.
- Ablations: The planner performs decently with as few as 10 sampling steps, supporting time-sensitive applications.Ablations also find slightly better performance when latent tokens are arranged as a grid rather than an unstructured sequence.
- Track Densification: The framework densifies sparse tracks by generating a dense latent grid and querying the autoencoder at arbitrary spatial locations.This capability follows from the planner’s dense output and the autoencoder’s spatial queryability.
- Temporal Compression: With temporal compression factor t_c = 2, the model denoises 32× as many tokens as a more strongly compressed model.Compute-matched experiments examine how compression affects generation performance during training.
E. Tracker Model Ablation
Tracker ablations find limited sensitivity to tracker choice and graceful degradation under imperfect supervision, while downstream evaluations show advantages over flow- and RGB-supervised alternatives.
- Tracker Choice: TapNext and CoTracker3 produce near-identical reconstruction accuracy, and models generalize well when evaluated with the other tracker.This indicates limited sensitivity to the tracker used for supervision.
- Supervision Quality: Track dropout and filtering occluded tracks cause graceful degradation on an occlusion-containing evaluation split.The ablation directly stress-tests robustness to imperfect supervision.
- Supervision Quality: 96.8 is the reconstruction accuracy for the original model, compared with 94.0 with track dropout and 93.2 with filtered occlusions.These values are reported in Table I for systematic tracker degradation during training.
- Downstream Evaluation: The method outperforms baselines on downstream tasks and exceeds motion-prediction performance from flow- and RGB-supervised models.The comparisons span downstream results in Table 2 and motion-prediction results in Tables 1, 3, and 4.
F. Why CoTracker for Video Evaluation
CoTracker3 is used for video evaluation because TapNext can lose tracks and create missing values that complicate unbiased metric computation.
- Tracker Choice: TapNext-based evaluation worsens video-model metrics when missing tracks are inpainted with their last observed positions.Min MSE increases from 29 to 37 for Wan and from 36 to 48 for Veo 3.
G. Additional Track Prediction Results
The model is evaluated beyond static-scene settings on DAVIS 2017 and Physics IQ, where it outperforms Motion I2V in its strongest Dense setting.
- On DAVIS 2017 and the solid mechanics split of Physics IQ, the model outperforms Motion I2V in its strongest Dense setting.DAVIS 2017 includes 150 videos with often significant camera motion, while Physics IQ targets physical understanding.
H. Addional Qualitative Examples
Additional examples visualize goal-conditioned trajectories, sparse-poke conditioning, temporal speed, and motion-generation behavior across compression, positional encoding, and sampling settings.
- Qualitative Examples: Additional qualitative samples show four model-generated trajectories from a prompt, with color progressing from blue to red as motion speed increases.Arrows mark pokes specifying where selected start positions should end.
- Ablations: Adding 2D RoPE positional information slightly improves PCK reconstruction for encoded and unseen positions.Performance is measured with a 1 pixel threshold in AE and MAE settings.
- Ablations: Motion-generation performance is reported across numbers of function evaluations, using dense target-position conditioning and 128 samples per start frame.
- Qualitative Examples: Sparse tracker inputs can condition dense, globally coherent flow toward goal endpoints, including plausible interpolation and inpainting across the scene.Only poke endpoints are provided as conditioning information.
- Ablations: Higher temporal compression yields more efficient learning and faster emergence of realistic motion when evaluated against wall-clock time.The comparison considers both matching step time and matching batch size.