PaperlayerView exact revision on arXiv

Source-linked AI summary

ReLift: Dense 3D Tracking as Lifted Trajectory Repair

Assaf Singer, Maroun Khriesh, Noam Rotstein, Yoni Kasten, Or Litany

arXiv:2610.42136v1cs.CV

TL;DR

Dense monocular 3D tracking must recover point correspondences, depth, and motion through occlusion, while joint learning of these tasks faces scarce 4D supervision. ReLift combines pretrained 2D tracking and 3D geometry with learned trajectory repair and completion, trained on simulated lifting errors. It achieves state-of-the-art mean accuracy on WorldTrack and improves lifted trajectories across evaluated providers without retraining.

  • Problem

    Dense monocular 3D tracking requires correspondence, geometry, and motion inference through occlusion, while learning these tasks jointly is demanding given scarce 4D supervision.

    Source: §1 · p. 2

  • Method

    ReLift combines pretrained 2D correspondences and scene geometry, then uses a learned corrector trained on randomized geometric corruptions of clean trajectories to repair and complete lifted tracks.

    Source: §1 · p. 2

  • Results

    ReLift achieves state-of-the-art mean accuracy on WorldTrack; with 4RC geometry, its CoWTracker variant improves mean accuracy over 4RC by 2.1 (All) and 5.4 (Dyn) points.

    Source: §4.2 · p. 7, §5 · p. 10

  • Takeaways & Limitations

    The same corrector improves all six evaluated tracker–geometry combinations without retraining, and its weights can incorporate future advances in either component.

    Source: §1 · p. 2

Abstract

from arXiv · show

Dense 3D tracking from monocular video requires establishing point correspondences over time, estimating their depth, and inferring their trajectories during occlusion. Existing methods typically learn these tasks jointly, a demanding objective given the scarcity of 4D supervision. We show that strong pretrained 2D trackers and 3D geometry models provide an effective foundation: lifted trajectories, with occluded segments simply interpolated, already rival dedicated 3D trackers using the same geometry. However, lifted trajectories inherit correspondence and depth errors, and under occlusion, their depth comes from the occluder rather than the tracked point. We introduce ReLift, an efficient feed-forward framework that recasts dense 3D tracking as geometric trajectory repair and completion. At its core, a trajectory corrector alternates attention along and across trajectories to preserve reliable observations, repair corrupted positions, and complete missing motion. Using only geometric inputs allows us to train directly on 3D trajectories with simulated lifting errors, decoupling training from images and from any specific tracker or geometry model. Concentrating 4D supervision on trajectory refinement yields a reusable model that consistently improves lifted trajectories across trackers and geometry providers without retraining, so it benefits directly from future advances in either component. On WorldTrack, ReLift achieves state-of-the-art mean accuracy, improving on the strongest baselines by 2.1 APD on all points and 3.0 on dynamic points, with faster configurations remaining competitive with leading 3D trackers while running faster than 4RC. Visit our supplementry page for more examples and code.

1 INTRODUCTION

ReLift decomposes dense monocular 3D tracking by combining pretrained 2D correspondences and scene geometry, then repairing and completing the resulting trajectories. Training on simulated lifting errors supports transfer across providers, and ReLift achieves state-of-the-art mean accuracy on WorldTrack.

1 INTRODUCTION

  • Monocular 3D tracking jointly requires correspondence, scene geometry, and motion inference during occlusion, although pretrained trackers and geometry models already provide strong estimates for two of these tasks.ReLift focuses scarce 4D supervision on repairing errors in the combined estimates and completing occluded motion.Source: §1 · p. 2
  • The same corrector improves all six evaluated tracker–geometry combinations without retraining and exceeds 4RC’s mean tracking accuracy when using the same geometry inputs.Source: §1 · p. 2
  • Pretrained 2D tracks lifted with estimated geometry and simple gap initialization can match or exceed dedicated 3D trackers using the same geometry.Source: §1 · p. 2
  • ReLift combines pretrained 2D correspondences and scene geometry in a feed-forward framework that repairs corrupted positions and completes missing motion.Alternating attention along and across tracks shares temporal and spatial evidence; processing track groups independently keeps correction efficient.Source: §1 · p. 2
  • Training uses randomized geometric corruptions of clean PointOdyssey trajectories, without running tracking or geometry models, and the same weights transfer across providers without retraining.The training strategy concentrates supervision on trajectory refinement and can incorporate future advances in either component.Source: §1 · p. 2
  • ReLift achieves state-of-the-art mean dense 3D tracking accuracy on WorldTrack, while faster configurations retain competitive accuracy at substantially lower runtime than comparably accurate 3D trackers.Source: §1 · p. 2

2 RELATED WORK

Related work combines tracking and geometry for dynamic 3D reconstruction and tracking, using both per-scene optimization and feed-forward approaches. ReLift builds on these cues by correcting lifted trajectories with a feed-forward model.

2 RELATED WORK

  • Per-scene reconstruction methods combine pretrained depth and tracking cues or optimize camera poses and dense trajectories, but require lengthy per-scene optimization.ReLift pursues the combination of 2D motion and 3D geometry with a feed-forward model.Source: §2 · p. 3
  • Geometry models supply depth and cameras in ReLift, while 2D trackers establish point identities across frames, despite overlap between geometry and tracking capabilities.Source: §2 · p. 3
  • Strong 2D tracking methods establish correspondences without annotated 3D motion, and native CoWTracker surpasses evaluated 3D methods in image-space accuracy.Source: §2 · p. 3
  • TracksTo4D and Seurat use 2D trajectories to recover dynamic 3D structure; ReLift instead starts from tracks lifted with pretrained depth and cameras and learns 3D residuals.Source: §2 · p. 3
  • Feed-forward 3D trackers predict trajectories jointly with geometry or condition on supplied depth, while ReLift uses PointOdyssey trajectories to learn repair and completion.Under matched geometry, lifted tracks already rival dedicated 3D trackers, and correction further improves them.Source: §2 · p. 3

3 METHOD

ReLift starts from pretrained 2D tracks and scene geometry, then uses a geometry-only corrector to repair corrupted observations and complete missing motion. Its method combines visibility-aware lifting, trajectory tokens, alternating attention, and training on simulated lifts.

3.1 LIFTING TRACKS INTO 3D

  • 2D tracks and predicted visibility, together with geometry-derived depth, intrinsics, and camera poses, produce reference-frame 3D lifts; invalid or low-visibility samples are masked.The viewing-ray norm converts optical-axis depth into distance along the ray.Source: §3.1 · pp. 4–5

3.2 TRAJECTORY REPRESENTATION

  • After median-depth normalization, missing positions are interpolated and held after the last observation; ray–point tokens encode initialized and observed displacements, availability, nearest-observation distance, and viewing geometry.Tokens are grouped into non-overlapping frame chunks and projected into tracklet tokens for attention.Source: §3.2 · p. 5

3.3 TRAJECTORY CORRECTOR

  • Tracklet tokens receive temporal attention across each track and across-track attention within each chunk; a head predicts unrestricted XYZ residuals that correct initialized trajectories.Grouping reference pixels into independent track batches makes inference cost grow linearly with pixel count for fixed sequence length and group size.Source: §3.3 · p. 5

3.4 TRAINING FROM SIMULATED LIFTS

  • Training simulates lifting failures on clean PointOdyssey trajectories by perturbing rays and ranges, varying observation availability, and modeling occluder-depth samples without images or provider inference.Randomized corruption parameters encourage transfer, and the same corrector improves all six evaluated tracker–geometry combinations.Source: §3.4 · pp. 5–6
  • The weighted squared-error objective supervises positions and consecutive-frame displacements, including unavailable observations, and weights moving tracks and unavailable observations more heavily.The displacement term discourages jitter while preserving target motion; the loss weights moving tracks four times as much and unavailable observations twice as much as their counterparts.Source: §3.4 · p. 6

4 EXPERIMENTS

The experiments evaluate dense 3D tracking on WorldTrack and compare methods under matched geometry, then assess qualitative trajectories, training corruptions, and efficiency choices. ReLift improves accuracy across tracker–geometry combinations, with particularly large gains on occluded points.

4.2 QUANTITATIVE RESULTS

  • ReLift achieves state-of-the-art mean APD3D on all and dynamic points with CoWTracker and 4RC geometry, while all six configurations share one trained corrector.For both trackers, mean accuracy increases from DA3 to VGGT-Ω to 4RC geometry without retraining.Source: §4.2 · p. 7, Table 1 · p. 7
  • 84.9 mean all-point APD3D for the initialized CoWTracker–VGGT-Ω lift exceeds TrackCraft3R’s 83.9 and DELTAv2’s 81.1 with the same geometry.After correction, ReLift outperforms both methods in every APD3D column; with 4RC geometry, it improves over 4RC by 2.1 points on All and 5.4 on Dyn.Source: §4.2 · p. 7, Table 1 · p. 7
  • Correction adds 1.6–3.6 points to mean APD3D, with the largest gains on ground-truth-occluded points across datasets and all six provider combinations.With CoWTracker and VGGT-Ω, all-point APD3D rises from 84.9 to 87.5.Source: §4.2 · pp. 7–8, Figure 3 · p. 8
  • VGGT-Ω configurations run 2.7–4.4× faster than 4RC, including provider inference, while retaining competitive accuracy.Using 4RC geometry takes longer but gives ReLift its highest mean accuracy.Source: §4.2 · p. 8

4.3 QUALITATIVE RESULTS

  • Matched novel-view comparisons show coherent spin trajectories, car tracking through shadow changes, and dog-head recovery after occlusion.The comparisons use the same reference pixels; TrackCraft3R shares ReLift’s geometry, while 4RC and V-DPM use their own reconstructions.Source: §4.3 · p. 8, Figure 4 · p. 9

4.4 ABLATION STUDY

  • One simulated-lift model improves over initialized lifts on every dataset with both test geometry providers and outperforms provider-trained models overall.Provider-trained gains transfer inconsistently: DA3-lift training improves on held-out scenes but less than simulation, while VGGT-Ω-lift training reduces accuracy on every evaluated dataset; simulation’s advantage persists on matched training windows.Source: §4.4 · p. 8, Figure 5 · p. 9
  • Removing wrong-surface corruptions causes the largest accuracy loss; correlated noise gives smaller mean improvements, and synthetic gaps yield small occluded-point gains.Natural PointOdyssey occlusions remain in every training variant.Source: §4.4 · p. 9, Table 2 · p. 10
  • With C = 4 frames per token, correction is more accurate and about three times faster than C = 1; C = 8 reduces runtime further but slightly lowers accuracy.The chosen P = 1024 tracks per group balances accuracy against attention cost: smaller groups reduce accuracy, while larger groups offer little improvement.Source: §4.4 · p. 9

5 CONCLUSION

The conclusion presents ReLift as trajectory repair built on pretrained 2D tracks and estimated geometry, with simulated lifting errors enabling one corrector to transfer across providers. Its WorldTrack performance is strong, though it remains sensitive to input geometry.

5 CONCLUSION

  • ReLift combines pretrained 2D tracks and estimated geometry with feed-forward trajectory repair and completion, improving evaluated tracker–geometry combinations without retraining.The initialized lifts already rival dedicated 3D trackers using the same geometry; ReLift’s largest gains are on occluded points.Source: §5 · p. 10
  • ReLift achieves state-of-the-art mean accuracy on WorldTrack, while faster configurations retain competitive accuracy at substantially lower runtime than comparably accurate 3D trackers.Performance remains sensitive to input geometry, motivating future work on robustness to geometric and persistent correspondence errors.Source: §5 · p. 10

A.1 OBSERVATION PREPROCESSING

  • Depth is looked up at the nearest pixel, clamped to the image border, while the viewing ray uses the subpixel track position.Visibility thresholds are γ = 0.85 for AllTracker and γ = 0.65 for CoWTracker; the latter is calibrated to match AllTracker’s false-visible rate on 24 PointOdyssey training windows.Source: §A.1 · p. 15

A.2 CORRECTOR IMPLEMENTATION

The corrector alternates temporal and across-track attention before predicting residual corrections, with track-order equivariance but group-dependent predictions. Its inputs and residuals are scaled after scene normalization.

A.2 CORRECTOR IMPLEMENTATION

  • After scene normalization, initialized displacements, masked displacements, and camera centers are scaled by 10, while prediction-head outputs are divided by 10 to form residuals.The gap feature is min(g_ti, 40)/40; the loss uses scaled errors with unchanged range normalization.Source: §A.2 · p. 15
  • The anchor embedding maps concatenated positional encodings of the three anchor coordinates to 384 features, while the temporal embedding encodes the chunk index.The anchor encoding uses 96 features before its two linear layers; Table 3 specifies the architecture settings.Source: §A.2 · p. 15, Table 3 · p. 15
  • Each block applies temporal self-attention, across-track self-attention, and an MLP with pre-layer normalization and residual connections.There is no track-index or pixel embedding, so reordering tracks reorders outputs, although predictions can depend on the tracks included in the group.Source: §A.2 · p. 15

A.3 TRAINING DETAILS

Training uses sampled PointOdyssey trajectory windows with randomized geometric corruptions, including simulated occlusion and dropouts. Dense inference corrects eligible tracks in groups and handles short groups and temporal padding with repeated or held inputs.

A.3 TRAINING DETAILS

  • Training uses 100 PointOdyssey scenes for training and 8 for validation, caching 2,048 eligible tracks per window and sampling 512 tracks per training example.Windows contain 84 frames at strides 1 and 2; training samples retain 48, 68, or 84 frames and mirror x coordinates with probability one half.Source: §A.3 · p. 15
  • Simulated lifting errors combine frame-shared flicker, tilt, per-track bias, white noise, sparse outliers, and angular ray noise.The listed processes use autoregressive sequences for flicker, tilt, and bias; range perturbations are clipped before exponentiation, and ray noise is projected onto the ray’s orthogonal plane.Source: §A.3 · pp. 15–16
  • Synthetic occluders hide projected points behind moving rectangles, and selected hidden observations are retained with ranges replaced by occluder depth to simulate false visibility.The corruption distributions are fixed across training and are not fitted to evaluation sequences.Source: §A.3 · p. 16, Table 4
  • The loss weights moving tracks more heavily and adjusts weights by input availability, while optimization runs for 150,000 AdamW steps.The settings are ρ_min = 0.3, λ = 4, and w_ti = (1+3μ_i)(2−m_ti); training uses a 1,000-step warm-up and cosine decay.Source: §A.3 · p. 16

A.4 DENSE INFERENCE

  • Dense inference corrects tracks with positive reference depth, processes them in groups of 1,024, and leaves other pixels at their uncorrected lifts.A short final group is cyclically filled from its batch; temporal padding holds the camera at its last position, and padded predictions are discarded.Source: §A.4 · p. 16

B.1 EVALUATION PROTOCOL

The evaluation covers WorldTrack and dense Kubric tracking, with clip-level alignment and scoring protocols. On Kubric, performance depends on geometry provider and alignment: matched 4RC geometry yields strong results, while provider transfer remains sensitive to input geometry quality.

B.1 EVALUATION PROTOCOL

  • WorldTrack evaluation uses ground-truth points visible and inside the first-frame image, with one ADT clip excluded because it has no eligible queries, leaving 199 clips.Dynamic tracks are selected by cumulative ground-truth motion exceeding 0.01 metres on WorldTrack or scene units on Kubric.Source: §B.1 · pp. 16–17
  • RANSAC Sim(3) alignment fits scale, rotation, and translation per clip, while global-scale alignment fits only scale; AJ measures 3D Jaccard accuracy and OA measures visibility accuracy.Scores are averaged over clips, and the WorldTrack mean weights its four dataset scores equally.Source: §B.1 · p. 17

B.2 BASELINE SETTINGS

  • MotionCrafter is scored on 63.6% of ADT queries inside its crop, and affected ADT scores and benchmark means are marked with a ‡.Sim(3) handles MotionCrafter’s coordinate-frame mismatch with the first camera; scale-only alignment cannot correct that mismatch.Source: §B.2 · p. 17

C.1 DENSE TRACKING ON KUBRIC

  • With 4RC geometry, both tracker variants approach 4RC’s all-point accuracy and exceed its dynamic-point accuracy; under global-scale alignment, CoWTracker with 4RC geometry leads all three reported metrics.Under RANSAC Sim(3), DA3 and VGGT-Ω configurations trail 4RC but outperform methods using the same geometry on all-point APD3D; transfer remains sensitive to input geometry quality.Source: §C.1 · p. 17, Table 5 · p. 17

C.2 VISIBILITY METRICS AND END-POINT ERROR

ReLift’s CoWTracker variants lead mean visibility accuracy, and the 4RC-geometry variant has the highest mean AJ and lowest mean EPE. Under global-scale alignment, the reported mean-accuracy advantage and correction gains persist, though dataset rankings change.

C.2 VISIBILITY METRICS AND END-POINT ERROR

  • CoWTracker with 4RC geometry achieves the highest mean AJ and lowest mean EPE, while CoWTracker variants lead mean OA.CoWTracker with VGGT-Ω gives the lowest EPE on both real-world datasets, ADT and PS; OA is returned unchanged from the 2D tracker.Source: §C.2 · pp. 17–18, Table 7 · p. 18, Table 8 · p. 19
  • Table 6 reports dense Kubric APD3D on all and dynamic tracks and EPE under RANSAC Sim(3) and global-scale alignment.The supplied passage identifies the table’s metrics and alignment settings but provides no result values.Source: Table 6 · p. 18

C.3 GLOBAL-SCALE ALIGNMENT

  • Under global-scale alignment, 4RC-geometry configurations still lead mean APD3D on all and dynamic points, and correction adds 1.4–2.6 points on all points and 2.6–3.8 on dynamic points.Mean accuracy decreases across methods, with substantial drops on PS, where V-DPM leads; dataset rankings therefore depend on alignment.Source: §C.3 · p. 18, Table 9 · p. 19

C.4 IMAGE-PLANE ACCURACY

Image-plane evaluation favors strong native 2D tracking, while 3D-method rankings depend on tracker choice. APD2D evaluates projected trajectories with their cameras but cannot measure range errors along a valid viewing ray.

Evaluation protocol

  • APD2D averages accuracy over 1, 2, 4, 8, and 16-pixel thresholds after coordinates are rescaled to 256 × 256.Predictions are projected through associated cameras without ground-truth alignment; native AllTracker and CoWTracker predictions are scored directly in 2D.Source: §C.4 · pp. 19–20

Results

  • Native CoWTracker achieves the highest APD2D in every reported column; among 3D methods, CoWTracker with 4RC geometry leads mean All and ranks second on Dyn.Track4World ranks ahead of it on Dyn.Source: §C.4 · p. 20, Table 11 · p. 20
  • Switching from AllTracker to CoWTracker adds 1.86–1.92 points to mean All and 2.32–2.36 to mean Dyn across every dataset and geometry pairing.For either tracker, geometry-provider means vary by only 0.15 points on All and at most 0.57 on Dyn.Source: §C.4 · p. 20

Relation to 3D accuracy

  • All six 3D configurations remain below their native trackers in both means; with CoWTracker and 4RC geometry, the gaps are 1.48 points on All and 3.82 on Dyn.This complete-pipeline comparison does not isolate correction or separate visible from occluded points.Source: §C.4 · p. 21

Metric scope

  • APD2D cannot measure range errors along a valid viewing ray, so small variation across geometry providers does not imply similar 3D accuracy.Lifting and reprojecting through the same camera returns the input pixel; APD3D gains over initialized lifts provide complementary evidence.Source: §C.4 · p. 21

D.1 TRAINING ABLATION SETTINGS

The ablations test simulated-lift training, ray masking, temporal chunking, and group size, then report full-pipeline runtime. Simulated training transfers across providers, and inference settings trade accuracy against speed and memory.

D.1 TRAINING ABLATION SETTINGS

  • Training-window filtering aligns frame-0 ground-truth positions to lifted positions, refits to reduce outlier influence, and retains windows with at least 256 valid points and median residual below 0.2.One transform is applied to ground truth at every frame; there is no per-frame refitting, so estimated camera-motion errors remain part of correction.Source: §D.1 · p. 21
  • Simulated-lift training improves APD3D on every dataset and outperforms AllTracker+DA3-lift training with both test geometry providers, using the same 3,890 windows and training setup.The corresponding provider-lift models improve training-scene APD3D on all points by 14.14 and 8.33 percentage points over their lifts.Source: §D.1 · p. 21, Figure 6 · p. 21
  • Full corruption simulation yields slightly higher mean APD3D on ground-truth-occluded point-frames than removing synthetic gaps, supporting a modest benefit from those gaps.Natural occlusions remain in both variants, and removing synthetic occluders also removes false-visible observations at occluder depth, so the ablation does not isolate missing observations alone.Source: §D.1 · p. 22, Table 12 · p. 22

D.2 RAY MASKING

  • About 2.8 APD3D points overall and 8.1 on occluded points are gained by ray masking on ADT with either geometry provider; changes elsewhere are below one point and mixed..Source: §D.2 · p. 22, Table 13 · p. 22

D.3 CHUNK SIZE AND GROUP SIZE

  • C = 4 gives the highest accuracy; C = 8 cuts correction time by about a third at a cost of at most 0.3 APD3D points, while C = 1 is slower and less accurate.These chunk-size correctors are separately trained for 150,000 steps.Source: §D.3 · p. 22, Table 14 · p. 23
  • Increasing inference group size from P = 1024 to 4096 changes every score by at most 0.2 points, while smaller groups reduce accuracy, particularly on dynamic points.Across-track attention cost grows linearly with P; the selected setting is P = 1024.Source: §D.3 · p. 22, Figure 7 · p. 23

D.4 RUN TIME AND MEMORY

  • VGGT-Ω configurations offer competitive WorldTrack accuracy at substantially lower runtime than 4RC and V-DPM, while faster baselines are less accurate.AllTracker correction takes roughly 3 s at 512^2 and 12 s at 1024^2; its DA3 and VGGT-Ω configurations remain feasible at 1024^2, where 4RC, V-DPM, and DELTAv2 run out of memory.Source: §D.4 · p. 23, Table 14 · p. 23, Table 15 · p. 24

E ADDITIONAL QUALITATIVE RESULTS

Qualitative examples show trajectory correction making initialized lifts more coherent and provide comparisons with other methods across two DAVIS scenes.

Additional method comparisons

  • In soccerball, ReLift trajectories form cleaner arcs ending on the ball; in lady-running, they follow the body in an ordered band while TrackCraft3R’s scatter below it.4RC and V-DPM also produce compact trajectories in lady-running.Source: E ADDITIONAL QUALITATIVE RESULTS · p. 24, Figure 8 · p. 24

Isolating trajectory repair

  • In camel and skate-park, correction reduces trajectory outliers near the ground and produces more coherent paths along the body or within the trajectory bundle.The comparisons hold CoWTracker tracks and DA3 geometry fixed and use the same point cloud and novel viewpoint for both trajectory panels.Source: E ADDITIONAL QUALITATIVE RESULTS · p. 24, Figure 9 · p. 25

Visualization details

  • The visualizations use queries spaced 16 pixels apart, show current and preceding 20-frame trails, and mark current positions with dots.Colors follow query raster order.Source: E ADDITIONAL QUALITATIVE RESULTS · p. 25
Loading 2610.42136v1…