TrackingWorld: World-centric Monocular 3D Tracking of Almost All Pixels

1 Introduction

• Many 3D tracking methods model motion in camera coordinates and cannot distinguish camera motion from dynamic-object motion, although downstream tasks require that separation.Some recent world-centric methods attempt to estimate camera motion, which remains challenging when dynamic objects are present.Source: §1 · pp. 1–2
• Existing methods largely track sparse pixels from the first frame, leaving dense tracking of newly emerging objects in intermediate frames unresolved.Tracking all pixels across frames also brings substantial computational complexity.Source: §1 · p. 2
• TrackingWorld estimates camera poses and world-centric 3D tracks while targeting dense coverage of almost all pixels across all frames.“Almost all” excludes noisy and outlier tracks filtered for robustness and accuracy.Source: §1 · p. 2
• The method upsamples arbitrary sparse 2D tracks and repeats tracking across frames, removing tracks in overlapping regions to reduce redundancy.The authors use DELTA’s tracking upsampler for the sparse-to-dense step.Source: §1 · p. 2
• TrackingWorld estimates 3D tracks and camera poses from dense 2D tracks and depth maps, treating initially static points as potentially dynamic under an as-static-as-possible camera constraint.This strategy is intended to exclude dynamic background points from camera-pose estimation.Source: §1 · p. 2
• TrackingWorld reports superior performance across camera-pose, dense-track depth, sparse 3D tracking, and dense 2D tracking metrics.The evaluations assess the system from these four perspectives.Source: §1 · pp. 2–3

2.1 2D Point Tracking

• 2D point tracking progresses from optical-flow-based PIPs and standardized TAP-Vid benchmarks to methods using trajectory correlations, transformer queries, and 4D correlation volumes.CoTrackers exploit correlations across trajectories; TAPTR models points as decoder queries, while LocoTrack uses lightweight correlation encoding.Source: §2.1 · p. 3

2.2 3D Point Tracking

• 3D trackers include optimization-based and feedforward methods, with world-centric approaches addressing camera motion but differing in drift and dense-tracking coverage.St4RTrack uses pairwise correspondences and may accumulate drift; TAPIP3D focuses on sparse tracking and cannot recover camera motion, whereas TrackingWorld targets dense tracking of emerging objects.Source: §2.2 · p. 3

2.3 4D Reconstruction

• 4D reconstruction methods recover camera and object motion using geometric, optimization-based, or learned approaches, while TrackingWorld focuses on dense pixel-level correspondences over time.The paper relates dense tracking to motion analysis, scene understanding, and video editing.Source: §2.3 · pp. 3–4

3.1 Overview

• The overview defines the output as dense 3D trajectories for almost all pixels together with per-frame camera poses, built by generating dense 2D tracks and back-projecting them into world-centric 3D.The preprocessing stage uses 2D tracks, dynamic masks, and monocular depth maps from auxiliary models.Source: §3.1 · p. 4

3.2 Dense 2D Tracking for Every Pixel

• The DELTA upsampler generalizes to arbitrary sparse 2D tracks, lifting them to dense tracks for a frame.The upsampler correlates each dense track with neighboring sparse tracks.Source: §3.2 · p. 5
• Tracking and upsampling are repeated on every frame, then pixels near visible previous trajectories are discarded to reduce redundant tracks.Most regions have already appeared in earlier frames, while only a few are new.Source: §3.2 · p. 5

3.3 Lifting 2D Tracks to World-centric 3D Tracks

• Initial camera poses are optimized from static-region tracks by unprojecting them with depth maps and minimizing reprojection error; the poses can be estimated per clip in parallel and merged.The inlier count includes static tracks whose projection errors fall within threshold τ.Source: §3.3 · p. 5
• Because coarse foreground masks can leave dynamic background objects in static regions, the method introduces residual offsets and an as-static-as-possible constraint while jointly optimizing camera poses and static 3D coordinates.The constraint minimizes offset magnitudes to keep genuinely static regions minimally moving while modeling background dynamics.Source: §3.3 · pp. 5–6
• Dynamic-region tracks, including background points with sufficiently large offsets, are initialized from UniDepth and refined in 3D using projection, depth-consistency, as-rigid-as-possible, and temporal-smoothness losses.The outputs include dynamic and static 3D tracks and camera poses; the pipeline explicitly models background motion rather than simply discarding those points.Source: §3.3 · p. 6

4.1 Implementation details

• A 30-frame video takes approximately 20 minutes to process on an RTX 4090 GPU.Experiments use CoTrackerV3 and DELTA for dense tracking and UniDepth as the depth prior.Source: §4.1 · p. 6

4.2.1 Camera pose estimation results

• TrackingWorld estimates camera poses more accurately than previous approaches on Sintel, Bonn, and TUM-D, with either DELTA or CoTrackerV3 tracks.The pose is refined jointly with world-centric dense tracking.Source: §4.2.1 · p. 7, Table 1 · p. 7

4.2.2 Depth accuracy of the dense 3D tracks

• Bundle adjustment significantly improves dense tracking depth across all evaluated datasets.Depth is compared with ground truth after single-scale-and-shift alignment for tracked points within image bounds.Source: §4.2.2 · p. 7, Table 2 · p. 7

4.2.3 Sparse 3D tracking results

• Sparse 3D tracking gains are strongest on moving-camera ADT, smaller on static-camera PStudio, and absent for visibility accuracy.The method samples evaluation points from optimized dense tracks and adopts the trackers’ predicted visibility maps.Source: §4.2.3 · p. 8, Table 3 · p. 8

4.2.4 Accuracy of dense 2D tracks

• The upsampler generalizes to CoTrackerV3 for long-range optical flow with performance comparable to DELTA.Evaluation uses the CVO test set’s CVO-Clean and CVO-Final subsets, with the latter including motion blur.Source: §4.2.4 · p. 8, Table 4 · p. 8

4.3 Qualitative results

• On DAVIS, visualizations show trajectories on spaced keyframes and continuous tracks, with stable tracking of static backgrounds and dynamic objects.The continuous tracks show motion consistency and trajectory completeness.Source: §4.3 · p. 8, Figure 3 · p. 9

4.4 Ablation study

• Ablations identify every-frame tracking, pose initialization, dynamic-track refinement, inlier filtering, background-motion modeling, and depth consistency as useful design choices.Without every-frame tracking, pose cues are lost; omitting dynamic-track refinement leaves depths at UniDepth predictions, and depth consistency helps suppress abnormal estimates.Source: §4.4 · p. 9
• The pipeline improves over raw depth predictions across three depth models, performs comparably across dynamic-mask sources, and the upsampler improves accuracy while cutting runtime by approximately 12×.The depth-model evaluation fixes tracking to DELTA on Sintel; the upsampler runtime comparison uses CoTrackerV3 on CVO-Clean.Source: §4.4 · pp. 9–10, Table 8 · p. 10, Table 6 · p. 10

• TrackingWorld targets dense 3D tracking of almost all pixels across all frames in a world-centric coordinate system.Source: §5 · p. 10
• A tracking upsampler densifies sparse 2D tracks and is applied to capture newly emerging objects.Source: §5 · p. 10
• An efficient optimization-based framework lifts dense 2D tracks into consistent world-centric 3D trajectories while disentangling camera motion from foreground dynamic motion.The paper reports that evaluations across multiple dimensions demonstrate the system's effectiveness.Source: §5 · p. 10

• The stride s is set to 4, following the same setting as DELTA.Source: §A.1 · p. 15
• The temperature τ and perturbation parameter ε are each set to 0.1.Source: §A.1 · p. 15
• Each video clip contains 5 frames.Source: §A.1 · p. 15

• The depth-consistency loss uses depth values for static points after transformation into the camera coordinate system at each timestep.The passage identifies the transformed static-point depth and the corresponding depth-map value.Source: §A.2 · p. 15
• For the stated datasets, the weights λ_ba, λ_dc, and λ_asap are 1, 1, and 5, respectively.Source: §A.2 · p. 15
• The as-rigid-as-possible loss constrains geometric deformation by keeping neighboring dynamic control points' relative distances consistent over time.K-nearest neighbors are used to identify the neighboring tracks.Source: §A.2 · p. 15
• The temporal-smoothness loss penalizes abrupt changes across frames and enforces first-order smoothness in dynamic control-point trajectories.Source: §A.2 · p. 15
• The dynamic-point objective combines bundle adjustment, depth consistency, as-rigid-as-possible, and temporal-smoothness losses, weighted 1, 1, 100, and 10, respectively.Source: §A.2 · p. 15

• Static points are downsampled while dynamic points remain intact, reducing optimization cost without discarding fine-grained dynamic motion details.Static-point optimization jointly estimates camera poses and static trajectories, whereas dynamic-point optimization only recovers dynamic trajectories.Source: §A.3 · p. 15
• Static tracks are downsampled on the image plane by dividing both coordinate components by ϖ and rounding to the nearest integers.Source: §A.3 · pp. 15–16
• The method forms spatiotemporal keys from each frame and downsampled coordinate, removes duplicate keys, and optimizes the resulting reduced set.Source: §A.3 · p. 16
• After optimization, visible downsampled 3D points provide nearest neighbors for inverse-distance weighted interpolation of full-resolution trajectories.The neighbor search is restricted to points visible in the current frame.Source: §A.3 · p. 16
• On Sintel, the speed-up strategy achieves similar accuracy with reduced optimization time.The quantitative evaluation is reported in Table 9.Source: Table 9 · p. 16, §A.3 · p. 16

• For each frame, raw monocular depth is aligned to tracking-based depth using per-point local scale ratios rather than a global frame-wise scale.Source: §A.4 · p. 17
• At each pixel, the method propagates scale from its k nearest tracking points, weighting neighbors inversely by their raw-depth-lifted 3D distance.Source: §A.4 · p. 17
• Dense 3D tracks propagate accurate scale information across the image, producing a temporally consistent and spatially coherent video depth sequence.The method uses an interpolation-based propagation step.Source: §A.4 · p. 17

• Uni4D estimates poses sequentially between consecutive frames, whereas TrackingWorld uses clip-to-global parallel pose estimation to improve computational efficiency.Source: §A.5 · p. 17
• TrackingWorld reports higher 3D tracking precision by distinguishing static and dynamic regions with masks and jointly optimizing camera poses and 3D trajectories.The joint optimization is reported to improve geometric consistency and reconstruction quality.Source: §A.5 · p. 17
• On Sintel (30–50 frames) and ADT (first 64 frames), TrackingWorld achieves higher accuracy and lower runtime than Uni4D.Source: §A.5 · p. 18, Table 11 · p. 18

A.6 More ablation study

• Filtering removes previously visible trajectories and isolated small components; the ablation reports improved camera-pose estimation and tracking stability.The component-size threshold is fixed at τ = 50 across experiments.Source: §A.6 · p. 18, Table 12 · p. 18

A.7.1 Comparison on camera pose estimation

• On Sintel, Bonn, and TUM-D, camera trajectories are more stable and better aligned with ground truth than DUSt3R and MonST3R.Source: §A.7.1 · p. 19, Figure 6 · p. 20

A.7.2 World-centric dense tracking results

• The displayed point clouds use temporally spaced keyframes, while the shown trajectories connect 3D tracks across all frames.Source: §A.7.2 · p. 19, Figure 7 · p. 21

A.8 Limitation and future work

• The method relies on auxiliary models for 2D tracks, monocular depths, and dynamic masks, adding computational overhead and stringent quality requirements.Source: §A.8 · p. 19

A.9 Assets availability

• The listed assets include Sintel, Bonn RGB-D, TUM RGB-D, DAVIS, ADT, Panoptic Studio, and CVO, with license information or terms noted.Source: §A.9 · p. 19