PaperlayerView exact revision on arXiv

Source-linked AI summary

Trace Anything: Representing Any Video in 4D via Trajectory Fields

Xinhang Liu, Yuxi Xiao, Donny Y. Chen, Jiashi Feng, Yu-Wing Tai, Chi-Keung Tang, Bingyi Kang

arXiv:2510.13802v1cs.CV

TL;DR

Video dynamics require a representation that captures continuous 3D motion rather than isolated per-frame geometry. The paper introduces Trajectory Fields and Trace Anything, which predicts them in one pass; experiments report strong benchmark performance, efficiency gains, and emergent spatial capabilities, with limitations from synthetic-data domain gaps and restricted curve expressiveness.

  • Problem

    Existing approaches do not provide a scalable, geometrically grounded representation that directly captures how dynamic scenes evolve across space and time.

  • Method

    Trajectory Fields assign every pixel in every frame a continuous parametric 3D trajectory, while Trace Anything predicts these fields feed-forward from video frames.

  • Results

    Trace Anything achieves state-of-the-art trajectory-field benchmark performance, competitive point-tracking results, significant efficiency gains, and emergent motion-forecasting, fusion, and manipulation capabilities.

  • Takeaways & Limitations

    A single jointly predicted trajectory field can support efficient 4D scene modeling and several spatial reasoning capabilities without auxiliary estimators or per-scene optimization.

  • Takeaways & Limitations

    Synthetic training data introduces a real-world domain gap, while limited control points restrict highly complex motions and performance degrades as frame counts increase.

Abstract

from arXiv · show

Effective spatio-temporal representation is fundamental to modeling, understanding, and predicting dynamics in videos. The atomic unit of a video, the pixel, traces a continuous 3D trajectory over time, serving as the primitive element of dynamics. Based on this principle, we propose representing any video as a Trajectory Field: a dense mapping that assigns a continuous 3D trajectory function of time to each pixel in every frame. With this representation, we introduce Trace Anything, a neural network that predicts the entire trajectory field in a single feed-forward pass. Specifically, for each pixel in each frame, our model predicts a set of control points that parameterizes a trajectory (i.e., a B-spline), yielding its 3D position at arbitrary query time instants. We trained the Trace Anything model on large-scale 4D data, including data from our new platform, and our experiments demonstrate that: (i) Trace Anything achieves state-of-the-art performance on our new benchmark for trajectory field estimation and performs competitively on established point-tracking benchmarks; (ii) it offers significant efficiency gains thanks to its one-pass paradigm, without requiring iterative optimization or auxiliary estimators; and (iii) it exhibits emergent abilities, including goal-conditioned manipulation, motion forecasting, and spatio-temporal fusion. Project page: https://trace-anything.github.io/.

1 Introduction

The paper represents videos as Trajectory Fields that assign each pixel a parametric 3D trajectory, then predicts these fields with Trace Anything in one feed-forward pass. A synthetic data platform supports large-scale training and evaluation, where the model achieves strong accuracy, efficiency, and additional spatial reasoning capabilities.

  • 1 Introduction: Trajectory Fields represent each pixel in every frame as a parametric 3D trajectory, offering a direct 4D model of scene dynamics.The representation is intended to capture continuous spacetime structure rather than disjoint per-frame point clouds.
  • 1 Introduction: Trace Anything predicts all pixel trajectories jointly from video frames in a single feed-forward pass without auxiliary estimators or per-scene optimization.It outputs control point maps that define spline-based trajectories for every pixel in a shared world coordinate system.
  • 1 Introduction: The Blender-based platform provides large-scale training and benchmarking data with dense annotations for trajectory field estimation.The released resources include 10,000+ training videos and a 200-video benchmark.
  • 1 Introduction: Trace Anything achieves state-of-the-art results on the trajectory field benchmark, competitive point-tracking performance, and significant efficiency gains.The reported advantages arise from the one-pass paradigm and do not require iterative optimization or auxiliary estimators.
  • 1 Introduction: The trajectory-field paradigm exhibits capabilities including motion forecasting, spatio-temporal fusion, and goal-conditioned manipulation.

2 Related Work

Prior work reconstructs dynamic scenes through point maps, tracking systems, or time-dependent appearance representations. Trace Anything instead emphasizes geometry-centric trajectory fields inferred directly from raw videos, complementing these approaches with direct 3D motion modeling.

  • Dynamic 3D scene reconstruction: Dynamic 3D reconstruction methods commonly produce disjoint per-frame point clouds and face challenges from dynamic scenes or limited camera baselines.The related methods include optimization-based SLAM and extensions of pairwise 3D reconstruction networks.
  • Point tracking: Point-tracking research spans long-range particle trajectories, transformer-based occlusion handling, efficient 4D correlation, and test-time optimization for 3D tracking.
  • 4D representations for NVS: Dynamic novel-view-synthesis methods model time-varying radiance fields, canonical-space deformations, or factorized 4D volumes for rendering.
  • 4D representations for NVS: Trace Anything is orthogonal to appearance-focused NVS work because it directly infers geometry-centric trajectory fields from raw videos.The paper identifies initialization of dynamic 3D Gaussian Splatting models as a possible future integration.

3 Method

Trajectory Fields represent each pixel in each frame as a continuous 3D trajectory, and Trace Anything predicts these fields directly in one feed-forward pass. The model parameterizes trajectories with control-point maps and trains them using trajectory supervision plus structural regularization.

  • 3.1 Problem Formulation: Trajectory Fields map each pixel in each frame to a continuous 3D trajectory over time.The field maps frame and pixel indices to spline-based curves in R3.
  • 3.2 Network Architecture: Trace Anything predicts dense control-point maps for all input frames, defining parametric curves in a shared world coordinate system.A fusion transformer integrates spatio-temporal context before the control point head outputs per-pixel trajectories.
  • 3.2 Network Architecture: The predicted spline trajectories can be queried at arbitrary times, including another frame’s acquisition time to obtain cross-frame 3D coordinates.Evaluating a trajectory at its own frame time recovers that frame’s 3D point map.
  • 3.2 Network Architecture: A single network inference produces the trajectory field for all frames, avoiding pairwise inference, global alignment, and external depth, flow, or track estimators.This one-pass design also avoids per-scene iterative optimization.
  • 3.3 Training Scheme: Training directly supervises predicted 3D positions at target timestamps and adds correspondence, confidence, timestamp, static, and rigidity regularization.Confidence adjustment downweights uncertain predictions while discouraging overconfidence; the final objective combines trajectory supervision with regularization terms.

4 Trace Anything Data Platform

The Trace Anything data platform generates photo-realistic dynamic scenes with dense 4D annotations, addressing the shortage of scalable datasets for diverse scene dynamics. It supports a 10K-scene training dataset and a 200-video all-to-all benchmark that evaluates trajectories across complete sequences.

  • 4 Trace Anything Data Platform: The Blender-based platform synthesizes photo-realistic dynamic scenes with dense annotations for trajectories, depth, semantics, flow, and camera poses.It is designed to scale data-driven modeling beyond small, rigid-motion-biased datasets.
  • Trace Anything dataset: The Trace Anything dataset contains about 10K unique scenes with 120 annotated frames each, spanning diverse environments and articulated or movable dynamics.Its annotations include per-pixel 2D/3D trajectories, depth maps, camera poses, and semantic masks.
  • Trace Anything benchmark: The benchmark contains 200 videos with 120 frames and evaluates trajectories for pixels sampled from every frame rather than only the first frame.It also uses denser annotations and world coordinates to test joint reasoning about global geometry and motion.

5 Experiments

Experiments evaluate Trace Anything on video, image-pair, and unordered-image settings, showing accurate trajectory fields, efficient inference, and capabilities beyond basic estimation.

  • Trajectory Field Estimation: On DAVIS videos, the model reconstructs dense pixel-level 3D trajectories for rigid and non-rigid motion while handling severe occlusions.The predictions preserve both dynamic and static scene components and global structure.
  • Trajectory Field Estimation: Given image pairs, Trace Anything interpolates plausible 3D motion for robot arms and manipulated objects, with trajectories re-projected into 2D.The setup uses BridgeData V2 image pairs separated by 10–20 frames and supports goal-conditioned manipulation.
  • Trajectory Field Estimation: The model predicts plausible trajectory fields and camera poses from unordered image sets without receiving temporal ordering or continuous camera motion.The inputs are presented chronologically only for visualization; sequence information is withheld from the model.
  • Quantitative Evaluation: Trace Anything achieves the best performance across all trajectory-field metrics and runs over an order of magnitude faster than optimization-based approaches.The evaluation uses video-based and image-pair-based inference with all-to-all trajectory predictions.
  • Emergent Capabilities: Trajectory fields support velocity-based and instruction-based forecasting, as well as spatio-temporal fusion across views and occlusions.These capabilities use trajectory extrapolation, generated future states, or alignment into a common canonical frame.

6 Conclusion

The paper concludes that Trajectory Fields represent each video pixel as a continuous 3D trajectory, while Trace Anything estimates this representation feed-forward without auxiliary estimators or scene optimization.

  • Conclusion: Trajectory Fields encode every pixel in every frame as a continuous 3D trajectory, providing a 4D representation of scene dynamics.The field maps frame and pixel indices to continuous functions from [0, 1] to R3.
  • Conclusion: Trace Anything predicts these trajectory fields directly from input frames using control-point maps and spline-based parametric curves.The representation supports querying a pixel’s 3D coordinate at arbitrary time values.
  • Conclusion: The approach eliminates auxiliary estimators and per-scene optimization while delivering competitive accuracy, inference efficiency, and new capabilities.The paper also describes a synthetic data platform supporting large-scale learning and evaluation.
  • Parametric Curves: Cubic B-splines provide local control through knot-vector-defined basis-function intervals, with clamped non-uniform knots used in the implementation.The implementation uses degree-3 splines, and confidence values are interpolated with the same basis functions.

C Additional Experimental Results

Additional experiments provide qualitative results and supplementary video-based demonstrations of the model’s capabilities.

  • Additional Experimental Results: Additional experimental results and supplementary videos present the model’s features, interactive visualization demos, and qualitative comparisons.The section directs readers to supplementary materials for the video results.

C.1 2D Trajectories, Dynamic Masks, Scene Flow, and Camera Poses

The predicted trajectory fields can be converted into 2D trajectories, dynamic masks, scene flow, and camera poses, extending the representation to multiple downstream outputs.

  • 2D Trajectories: Projected 2D trajectories are obtained from predicted per-pixel 3D trajectories using known or estimated camera parameters.The projected trajectories can be overlaid on an input frame and are demonstrated for image-pair inputs.
  • Dynamic Masks: Thresholding per-pixel control-point variance produces dynamic masks that separate static and dynamic regions.The variance is computed over the control-point set associated with each pixel.
  • Scene Flow: Scene flow from an image pair is computed as the difference between the endpoints of predicted trajectories.The paper demonstrates this on non-consecutive frames to highlight robustness under long-range motion.
  • Camera Poses: World-coordinate point maps enable estimation of focal length, rotation, and translation for both continuous video motion and unordered image sets.The method is shown recovering camera motion in dynamic scenes, including free-fall and perpendicular object motion.
  • Qualitative Results: Qualitative comparisons report better preservation of fine object details, complex motion handling, and separation of static and dynamic objects.Examples include the elephant’s tail and flamingo’s neck in reconstructed point clouds.

C.3 Additional Quantitative Comparison

Trace Anything remains competitive for 3D tracking without task- or dataset-specific fine-tuning and preserves advantages on out-of-distribution data. Its per-pixel, single-pass design enables substantially faster inference than query-limited SpaTracker.

  • Out-of-distribution input: On out-of-distribution PointOdyssey videos, the model maintains advantages across all reported metrics and inference efficiency without PointOdyssey training or fine-tuning.The benchmark contains 50 videos of 30 frames each, and the model has never been trained or fine-tuned on PointOdyssey.
  • Additional quantitative comparison: The out-of-distribution comparison reports EPEmix, EPEsta, EPEdyn, CA, SDD, and runtime, while the 3D-tracking comparison reports APD3D, AJ, and runtime.Table A reports CA in 10^-2 and SDD in 10^-3; both evaluations mark best and second-best results.
  • 3D tracking: The 3D-tracking evaluation uses TAPVid-3D subsets ADT, DriveTrack, and PStudio, reporting APD3D for spatial accuracy and AJ for spatial and occlusion correctness.Each subset contributes 50 videos of 60 frames, with every other frame used as input.
  • 3D tracking: Trace Anything remains competitive with SpaTracker on 3D tracking, surpasses it on some metrics, and runs orders of magnitude faster.SpaTracker is designed and trained for 3D tracking, but supports only a fixed number of query points per run; Trace Anything performs per-pixel tracking in one forward pass.

C.4 Ablation Study

The ablation study favors B-spline trajectories with ten control points and pretrained geometric backbones, while polynomial curves and random initialization perform worse.

  • Parametric curves: Polynomial curves underperform because their parameters lack clear geometric and physical interpretability.The study compares polynomial, Bézier, and B-spline curve types.
  • Parametric curves: B-spline curves with ten control points achieve the best overall performance, leading the authors to adopt them as Trace Anything’s default curve configuration.Accuracy generally improves as the number of control points increases.
  • Geometric backbone: Without pretrained initialization, the backbone struggles to converge, while VGGT provides modest metric gains over Fast3R at substantially higher runtime.VGGT may still help when fine structural detail or large-baseline scenarios are important.

D Limitations

The paper identifies limitations from synthetic-data training, restricted trajectory expressiveness for complex or long motions, and lower precision than sparse 3D tracking methods.

  • Data limitations: Synthetic training data inevitably creates a domain gap with real-world scenarios, motivating future use of partial real-data annotations.The authors identify real-data annotation as a possible way to bridge this gap.
  • Representation limitations: Limited control points restrict expressive power for highly complex motions, and repeated back-and-forth motion may defeat clipping or frame-downsampling remedies.Performance also degrades as the number of frames increases.
  • Accuracy limitations: Dense per-pixel prediction offers efficiency advantages but may be less precise than sparse 3D tracking methods.The authors suggest incorporating fine-grained point-level estimation from sparse trackers.
Loading 2510.13802v1…