Source-linked AI summary
TrajLoom: Dense Future Trajectory Generation from Video
Zewei Zhang, Jia Jun Cheng Xian, Kaiwen Liu, Ming Liang, Hang Chu, Jun Chen, Renjie Liao
TL;DR
Future trajectory forecasting must model uncertain motion from observed video rather than relying only on appearance. TrajLoom combines anchor-offset encoding, latent trajectory modeling, and flow-based generation, achieving stable long-horizon predictions that support video generation and editing.
Problem
Dense future motion remains challenging to predict from observed video, although trajectories encode current dynamics and constrain plausible futures.
Method
TrajLoom encodes points relative to grid anchors, learns structured trajectory latents with TrajLoom-VAE, and generates future windows with TrajLoom-Flow conditioned on trajectories and video.
Results
TrajLoom achieves state-of-the-art performance on TrajLoomBench, improving motion realism, temporal consistency, and stability across quantitative and qualitative evaluations.
Takeaways & Limitations
The predicted trajectories provide a foundation for motion-controlled video generation and editing.
Abstract
from arXiv · showhide
Predicting future motion is crucial in video understanding and controllable video generation. Dense point trajectories are a compact, expressive motion representation, but modeling their future evolution from observed video remains challenging. We propose a framework that predicts future trajectories and visibility from past trajectories and video context. Our method has three components: (1) Grid-Anchor Offset Encoding, which reduces location-dependent bias by representing each point as an offset from its pixel-center anchor; (2) TrajLoom-VAE, which learns a compact spatiotemporal latent space for dense trajectories with masked reconstruction and a spatiotemporal consistency regularizer; and (3) TrajLoom-Flow, which generates future trajectories in latent space via flow matching, with boundary cues and on-policy K-step fine-tuning for stable sampling. We also introduce TrajLoomBench, a unified benchmark spanning real and synthetic videos with a standardized setup aligned with video-generation benchmarks. Compared with state-of-the-art methods, our approach extends the prediction horizon from 24 to 81 frames while improving motion realism and stability across datasets. The predicted trajectories directly support downstream video generation and editing. Code, model checkpoints, and datasets are available at https://trajloom.github.io/.
1 Introduction
TrajLoom addresses dense future-trajectory forecasting by conditioning generation on observed trajectories and video history. It combines offset-based encoding, structured latent modeling, and flow-based prediction, and evaluates them through a unified benchmark.
- Observed trajectories encode current dynamics and constrain plausible futures, motivating generation conditioned on trajectory and video history.
- Grid-Anchor Offset Encoding represents each point as a displacement from a fixed pixel-center anchor, reducing location-dependent bias.Absolute coordinates are recovered by adding the anchors back.
- TrajLoomBench unifies real and synthetic videos with standardized setups, while the method supports motion-controlled video generation and editing.
- TrajLoom-VAE maps trajectory segments to compact spatiotemporal tokens using masked reconstruction and spatiotemporal regularization.The regularizer aligns velocities with local neighbors to preserve motion structure.
- TrajLoom-Flow generates full future windows in latent space from observed trajectories and video, using boundary cues and on-policy K-step fine-tuning for stable sampling.
2 Related Works
The paper situates future trajectory forecasting within dense tracking and motion-guided video generation. Figure 1 illustrates the long-horizon trajectory prediction setup used by the framework.
- Trajectories for motion anticipation: Modern tracking-any-point methods provide dense, long-range trajectories with visibility and occlusion information in unconstrained videos.
- Figure 1 shows an 81-frame observed history on the left and predicted trajectories for the next 81 frames on the right.Predictions are displayed at early, middle, and final times, with colors indicating the spatial order of query points.
- Motion-guided generation and editing: Controllable video generation uses optical flow, camera trajectories, and point tracks to guide temporal dynamics.
3 TrajLoom: Dense Future Trajectory Generation
TrajLoom converts observed trajectories into grid-anchored dense offsets, compresses them into structured spatiotemporal latents, and generates future motion through a history-conditioned latent flow. Its representation and regularization target location bias, temporal jitter, and local spatial inconsistency.
- Pipeline: The pipeline rasterizes observed trajectories into dense offset fields, compresses them with TrajLoom-VAE, generates future latents with TrajLoom-Flow, and decodes future trajectories.TrajLoom-Flow conditions generation on history latents and video features, using rectified-flow integration with boundary hints.
- Grid-Anchor Offset Encoding: Grid-Anchor Offset Encoding represents each trajectory point as displacement from a fixed pixel-center anchor rather than absolute coordinates.The offset field is paired with a dense visibility mask, and absolute coordinates can be recovered.
- Grid-Anchor Offset Encoding: Absolute coordinates have high location-explained variance, whereas offsets strongly reduce this location-driven component and focus representation on local displacement.The comparison attributes the absolute-coordinate variance to grid position, even when local motion is similar.
- TrajLoom-VAE: TrajLoom-VAE maps temporal offset-field segments to compact latent tokens and reconstructs dense tracks using visible-location reconstruction.Its training objective combines reconstruction error, KL divergence, and spatiotemporal consistency regularization.
- Spatiotemporal consistency regularizer: The spatiotemporal regularizer matches temporal velocities and multiscale spatial neighbor relations between target and reconstructed trajectories.Consistency terms apply only to valid visible pairs and are normalized by the number of such pairs; spatial neighborhoods use multiple hop distances.
4 Experiments
The experiments evaluate trajectory reconstruction and future generation across unified real and synthetic benchmarks, using fidelity, realism, consistency, and stability metrics. TrajLoom improves long-horizon motion quality and supports downstream motion-controlled video generation.
- Evaluation setup: The evaluation compares TrajLoom-VAE reconstruction and TrajLoom-Flow future generation against WHN (L).The benchmark covers Kinetics, RoboTAP, Kubric, and MagicData, with dense trajectory fields and visibility masks.
- Evaluation metrics: FVMD measures motion realism and temporal consistency, while FlowTV and DivCurlE diagnose spatial tearing and unstable deformation.VEPE measures visibility-masked endpoint error for VAE reconstruction.
- Future-trajectory generation: 2.5–3.6× lower FVMD is achieved across datasets, including a reduction from 4872 to 1338 on Kubric.The method also lowers FlowTV and DivCurlE, indicating fewer spatial discontinuities and more stable motion.
- Track-VAE reconstruction: TrajLoom-VAE achieves low VEPE and outperforms WHN (L)-VAE across datasets.Performance remains stable when reconstructed segment length increases from 24 to 81 frames.
- Qualitative results: Conditioning on observed motion history produces futures that better continue current dynamics and remain spatially coherent than WHN (L).WHN (L) more often exhibits drift and spatial tearing in qualitative comparisons.
- Ablation study: Removing Grid-Anchor Offset Encoding worsens 81-frame generation, increasing FVMD-Long and degrading FlowTV and DivCurlE.The ablation indicates more drift and tearing without offsets.
- Downstream applications: Predicted future trajectories guide Wan-Move to synthesize motion-consistent videos from a single input image.Observed history trajectories are forecast first and then supplied as motion-control signals.
5 Conclusion
The framework combines offset-based trajectory representation, consistency-regularized latent compression, and rectified-flow generation to produce stable long-horizon motion. It outperforms state-of-the-art methods on TrajLoomBench and supports future controllability improvements.
- The framework combines Grid-Anchor Offset Encoding, TrajLoom-VAE, and TrajLoom-Flow with boundary cues and on-policy fine-tuning.
- The combined components produce stable long-horizon motion and outperform state-of-the-art methods on TrajLoomBench in quantitative and qualitative evaluations.
- Predicted trajectories support motion-guided video generation and editing, while future work targets user-driven trajectory editing and broader integration.
A Model and Training Details
TrajLoom compresses dense trajectory segments into spatiotemporal latents, then generates future latents with conditioned rectified flow. Training combines masked reconstruction, consistency regularization, flow matching, boundary anchoring, and on-policy rollout fine-tuning.
- TrajLoom-VAE: TrajLoom-VAE encodes 81-frame 2D-offset trajectory segments into compact spatiotemporal latents and decodes dense tracks.Temporal compression reduces 81 frames to 21 latent steps.
- TrajLoom-VAE: TrajLoom-VAE uses visibility-masked Huber reconstruction, KL regularization, and spatiotemporal consistency losses to preserve motion structure.The consistency weights are λtemporal = 0.1 and λspatial = 0.2.
- TrajLoom-Flow: TrajLoom-Flow predicts future latents with rectified flow conditioned on history trajectory latents, visibility, video features, and pooled text embeddings.
- TrajLoom-Flow: The flow source anchors its first future latent slice to the last history slice with additive noise to encourage history–future continuity.
- TrajLoom-Flow: On-policy K=8 Euler rollout fine-tuning reduces train–test mismatch from ODE sampling and adds endpoint consistency for stability.The rollout uses a logit-spaced time grid and endpoint-consistency weight γ = 0.1.
B.1 Quantifying Location Bias Removed by Grid-Anchored Offsets
The appendix measures location-driven coordinate variance and shows that grid-anchored offsets reduce this bias. It also demonstrates why pointwise reconstruction must be supplemented by spatiotemporal consistency to avoid jittery motion.
- Location-bias analysis: The analysis compares variance explained by static grid location with residual temporal motion variance across dense point tracks.Tracks are evaluated on 128 randomly sampled clips using a stride-32 grid.
- Location-bias analysis: The explained-variance ratio measures the fraction of total coordinate variation attributable to differences between grid locations.A high percentage indicates that grid position, rather than motion, explains most variation.
- Location-bias analysis: Absolute coordinates have high location-explained variance, whereas offsets strongly reduce the location-driven component.
- Consistency regularization: Two reconstructions of x(t) = t can have identical pointwise errors and masked Huber loss while implying different motion dynamics.A constant-bias reconstruction is smooth, whereas an alternating-bias reconstruction is jittery.
- Consistency regularization: Velocity matching penalizes jittery solutions that pointwise reconstruction cannot distinguish, motivating consistency regularization for dense trajectory fields.The same failure mode extends to temporal jitter and local inconsistency across neighboring 2D points.
C Additional Qualitative Results
Additional qualitative results compare TrajLoom with WHN and demonstrate downstream video synthesis. The examples emphasize long-horizon coherence and use generated trajectories to control an 81-frame video.
- Comparisons with WHN: Side-by-side comparisons show TrajLoom conditioned on history and WHN conditioned on an image, each producing a predicted future.
- Comparisons with WHN: TrajLoom more consistently preserves long-horizon coherence with fewer drift and tearing artifacts than WHN (L).
- Controllable video synthesis: For downstream synthesis, TrajLoom-Flow samples 81-frame future trajectories that Wan-Move uses with a conditioning image to generate motion-following video.
D.1 TrajLoom-VAE Ablations
The ablations show that both Grid-Anchor Offset Encoding and spatiotemporal consistency regularization improve TrajLoom-VAE reconstruction across datasets and forecast horizons.
- Removing Grid-Anchor Offset Encoding consistently increases VEPE across datasets and 24- and 81-frame horizons.The result supports offsets as a way to reduce location-dependent bias and simplify trajectory modeling.
- Removing the spatiotemporal consistency regularizer also degrades VEPE, especially for longer trajectory windows.The degradation is consistent with the regularizer suppressing temporal jitter and improving local coherence.
- Both components contribute to reconstructing dense trajectories reliably over extended forecast horizons.
D.2 VAE Training Dynamics
Training curves show that offset encoding and spatiotemporal regularization improve TrajLoom-VAE convergence and reconstruction loss through different mechanisms.
- Removing Grid-Anchor Offset Encoding causes worse convergence and a gradually increasing masked L1 reconstruction loss.This pattern is consistent with overfitting to location-dependent coordinate biases in absolute space.
- Removing the spatiotemporal consistency regularizer produces a higher-loss plateau during training.Pointwise reconstruction alone does not sufficiently constrain motion structure.
- The full design combines better conditioning from offset encoding with motion-level constraints for temporally smooth, locally coherent latent trajectories.
D.3 TrajLoom-Flow Ablation Study
TrajLoom-Flow uses boundary cues, history fusion, and on-policy fine-tuning to stabilize 81-frame latent trajectory generation, with solver choice offering an additional quality-efficiency trade-off.
- Ablations: 81-frame future-prediction ablations show that on-policy fine-tuning adds gains beyond longer flow-matching training across datasets.The comparison separates training duration from the proposed on-policy stage.
- Ablations: Disabling first-slice boundary anchoring causes the largest degradation among boundary-hint variants, while removing token-aligned history fusion causes a smaller consistent drop.The ablations evaluate generator components using motion and flow-field diagnostics with Euler integration for 10 steps.
- Sampling: DOPRI5-100 consistently reduces FlowTV and DivCurlE for TrajLoom, while TrajLoom substantially outperforms WHN under both solver settings.Euler-10 is used as the default efficiency-quality trade-off, with DOPRI5-100 available when additional smoothness is desired.
- Boundary cues: Boundary anchoring repeats the last history latent across future indices with Gaussian noise to initialize the rectified-flow source state.This design supplies a boundary cue at the start of the future latent sequence.
- Boundary cues: Token-aligned history fusion adds a gated, progressively extended boundary and velocity cue to model input tokens.The cue is formed from the last two history latent slices, and its fusion weight increases from the first to the last future step.
E.2 Camera Movement Caption Dataset Augmentation
The pipeline enriches MagicData captions with camera-motion descriptions and predicts future token visibility as a separate auxiliary output.
- Camera caption augmentation: MagicData captions are augmented with camera-motion phrases estimated from point-track displacements.The augmentation estimates global translation and decomposes residual motion into zoom, roll, and handheld or shake components.
- Visibility prediction: The visibility predictor outputs per-token visibility logits from generated future latent tokens.It uses a linear projection, temporal convolutions, and a final projection along the latent-time axis.
- Visibility prediction: At inference, thresholded logits produce future visibility masks independently of the trajectory generator.Training targets are obtained by max-pooling dense future visibility masks onto the VAE token grid.