PaperlayerView exact revision on arXiv

Source-linked AI summary

TrackEverything: Long Horizon Dense Tracking via De-Duplicating 3D Scene Representations

Ayush Jain, Sreeharsha Paruchuri, Ishita Gupta, Fan Zhang, Tanner Schmidt, Jakob Engel, Katerina Fragkiadaki, Adam W. Harley

arXiv:2609.30222v1cs.CVcs.AIcs.RO

TL;DR

Existing point trackers trade long-horizon sparse tracking against dense tracking on short clips, and frame-based processing scales with video duration. TrackEverything represents videos as persistent world-coordinate 3D tracks, de-duplicates them across sliding windows, and reports all-point tracking beyond 1000 frames within 40 GB of GPU memory. On short TAPVid-3D clips, it outperforms open-source all-frame dense 3D trackers by more than 20% APD.

  • Problem

    Existing methods either track sparse query points over long horizons or all points only over short clips, while frame-based processing repeatedly encodes previously observed surfaces.

    Source: §1 · p. 2

  • Method

    TrackEverything tracks persistent world-coordinate 3D scene points, de-duplicates co-located tracks across sliding windows, and decodes dense trajectories only for dynamic points.

    Source: §1 · p. 2

  • Results

    More than 20% APD improvement over open-source all-frame dense 3D trackers on short TAPVid-3D clips, with all-point tracking across 1000+ frames within 40 GB of GPU memory.

    Source: §1 · p. 2, §6 · p. 11

  • Takeaways & Limitations

    TrackEverything tracks all visible points across long videos and remains competitive with state-of-the-art sparse trackers on long sequences despite tracking far more points.

    Source: §1 · pp. 2–3

    Active memory continues to expand during perpetual open-world exploration, and performance remains coupled to the fidelity of external geometry.

    Source: §5 · p. 10

Abstract

from arXiv · show

Existing point tracking models face a fundamental tradeoff: they can either track a sparse set of query points over long horizons, or track all points across only short clips. We introduce TrackEverything, a 3D point tracker that breaks this trade-off by representing videos as persistent 3D scene tracks in world coordinates. Grounded in the insight that videos are 2D projections of an underlying 3D world, TrackEverything decouples model complexity from video duration, allowing it to scale with unique physical scene geometry instead. Our approach introduces three key innovations. First, we employ a voxelization-based de-duplication mechanism at sliding-window boundaries to merge co-located tracks, preventing repeated observations of the same surface from redundantly accumulating. Second, we decompose tracking into an endpoint refiner that predicts each point's destination and static-versus-dynamic classification, followed by a lightweight trajectory refiner that decodes dense trajectories exclusively for dynamic points. Third, we propose 3D WAFT, replacing memory-prohibitive 4D correlation volumes with efficient feature sampling in the scene cloud. To the best of our knowledge, TrackEverything is the first 3D tracker capable of tracking all visible points across videos exceeding 1000 frames within 40 GB of GPU memory. On TAPVid-3D, TrackEverything outperforms all open-source all-frame dense 3D trackers by more than 20% APD on short clips, while remaining competitive with state-of-the-art sparse trackers on long sequences, despite tracking far more points.

1 INTRODUCTION

Frame-based dense tracking repeatedly processes already-seen surfaces, while existing methods trade long horizons for sparse queries or all-point coverage for short clips. TrackEverything instead represents videos as persistent 3D scene tracks, and reports dense tracking beyond 1000 frames within 40 GB of GPU memory.

1 INTRODUCTION

  • Existing methods either track sparse user queries over long horizons or track all points only in short clips, leaving full-length videos without all-visible-point tracking.Frame-by-frame 2D processing scales in compute and memory with video duration because every new frame adds a full grid of visual tokens, including re-observed surfaces.Source: §1 · p. 2
  • The approach is motivated by treating video as 2D camera projections of a 3D world, where scene-centric representations can scale with unique content rather than video duration.The paper presents persistent de-duplicating 3D scene representations as a potential substrate for extending foundation models to dynamic environments.Source: §1 · pp. 2–3
  • TrackEverything represents videos as non-redundant world-coordinate 3D scene tracks, scaling with unique physical geometry rather than frame count.It voxelizes at sliding-window boundaries to merge co-located tracks, predicts endpoints and static-versus-dynamic status before decoding trajectories only for dynamic points, and uses 3D WAFT feature sampling instead of memory-intensive 4D correlation volumes.Source: §1 · p. 2
  • More than 20% APD improvement over open-source all-frame dense 3D trackers on short TAPVid-3D clips; TrackEverything also tracks all points across 1000+ frames within 40 GB of GPU memory.On long sequences it remains competitive with state-of-the-art sparse trackers and first-frame dense methods while providing dense tracks for the entire scene.Source: §1 · pp. 2–3

2 RELATED WORK

Prior point trackers face limits from sparse query sets, first-frame-only visibility, or short clip lengths. TrackEverything instead tracks all points over long videos with persistent 3D scene representations and cross-window de-duplication.

2 RELATED WORK

  • Sparse 3D trackers remain limited by query-point counts or frame-wise feature clouds, while dense trackers either miss newly disoccluded surfaces or are confined to short clips.The related-work discussion notes that tracking all points in a 512×512, 200-frame video can require on the order of 10^10 predictions when surfaces are represented independently in every frame.Source: §2 · p. 3
  • TrackEverything differs by growing its world-coordinate representation only when new scene content appears, replacing 4D correlations with 3D WAFT and tracking all points across long videos.Its voxelization-based de-duplication collapses co-located points across sliding windows.Source: §2 · p. 3
  • Figure 2 depicts the pipeline from unprojected 3D feature clouds through endpoint and dynamic-point trajectory refinement to voxel merging at window boundaries.Only dynamic points receive dense trajectory decoding; co-located tracks are mean-pooled before the next window.Source: Figure 2 · p. 4

3 METHOD

TrackEverything tracks persistent world-coordinate scene points through sliding windows, separating endpoint estimation from dense trajectory decoding. Per-window encoding and boundary de-duplication support a compact scene representation, while the method also accommodates optional subpixel queries.

3 METHOD

  • TrackEverything predicts 3D trajectories, visibility, and static/dynamic status for unique scene points, using sliding windows and cross-window voxelization to merge tracks at shared positions.Dense within-window paths are decoded only for dynamic points; static points retain their positions.Source: §3 · p. 4, §3.2 · p. 5

3.1 ENCODING

  • Per-frame tokens are quantized into voxels and mean-pooled within each frame, then concatenated into a space-time feature cloud; repeated surfaces remain represented until window processing ends.Per-frame rather than cross-frame voxelization avoids merging distinct surfaces that occupy the same region at different times.Source: §3.1 · pp. 4–5

3.2 ITERATIVE REFINEMENT

  • Because full trajectories for every point are wasteful and boundary handoff needs only destinations, tracking is split into endpoint estimation and dynamic-only trajectory decoding.The method begins from zero-velocity initializations for newly disoccluded and carried-over points.Source: §3.2 · p. 5

3.2.1 ENDPOINT REFINER

  • The endpoint refiner updates each point’s estimated target-time position while predicting visibility and static-versus-dynamic status.It uses source and target position and time embeddings, plus sampled source and target image features as a template-matching signal for correspondence refinement.Source: §3.2.1 · p. 5

3.2.2 TRAJECTORY REFINER

  • The trajectory refiner initializes dynamic-point paths by constant-velocity interpolation and refines them with within-track temporal attention and scene-context cross-attention.Track summary tokens attend to the full feature cloud, while tracks do not attend to one another.Source: §3.2.2 · p. 6

3.3 SLIDING WINDOW INFERENCE AND DE-DUPLICATION

  • At each window boundary, the method merges points across origin frames after tracking them to the same target timestep, using voxelization at predicted target positions.The passage justifies merging on the premise that two entities cannot occupy the same voxel at the same time.Source: §3.3 · p. 6, §3.2.2 · p. 6

3.4 IMPLEMENTATION DETAILS

  • Training supervises positions and trajectories with 3D L2 loss and visibility and static/dynamic labels with cross-entropy, using labelled points and a subset of dynamic trajectories.TrackEverything has 41M trainable parameters and trains on Kubric, PointOdyssey, and Dynamic Replica; optional subpixel query features bypass voxelization.Source: §3.4 · p. 6

4 EXPERIMENTS

The experiments compare TrackEverything with sparse and dense trackers on TAPVid-3D and PointOdyssey, examining benchmark performance, scaling, and design choices. Results show competitive tracking across long sequences and favorable memory scaling, alongside gains from geometry inputs and selected architectural choices.

4.1 3D POINT TRACKING OVERVIEW

  • TAPVid-3D evaluates 50 minival clips from each of three datasets, with sparse query points enabling comparisons across sparse and dense tracking methods.The experiments report both APD-P and APD-M, which use different threshold conventions; the full-length comparison additionally includes sparse trackers, DeltaV2, and TrackEverything.Source: §4.1 · p. 7

4.2 MAIN RESULTS

  • TrackEverything beats VDPM by over 20% APD-P on average among all-frame dense trackers and remains competitive with sparse 3D trackers on full-length videos while tracking orders more points.D4RT scores higher on 48-frame DriveTrack and PStudio clips but trails on ADT, uses ∼20× the parameters, and does not track beyond 48 frames.Source: §4.2 · pp. 7–8, Table 1 · p. 8

4.3 COMPLEXITY ANALYSIS OF DENSE AND SPARSE POINT TRACKING

  • TrackEverything stays under 30 GB at 900 frames, while the other tested methods exceed memory between 96 and approximately 600 frames.Figure 3 compares accuracy, latency, and peak memory against video length at 384 query points; the experiment focuses on latency and memory trends rather than absolute accuracy.Source: Figure 3 · p. 8, §4.3 · p. 9
  • At 120 frames, TrackEverything maintains near-constant latency as query count grows; DeltaV2-sparse and CoTracker3 become slower at approximately 750 and 2,000 points, respectively.CoTracker3 and TrackEverything both remain below 30 GB, while SpatialTracker-v2 exceeds memory at 1,000 points.Source: Figure 3 · p. 8, §4.3 · p. 9

4.4 ADDITIONAL EXPERIMENTS AND ABLATION STUDIES

  • Sensor geometry raises TrackEverything's APD-P to 71.7 on PStudio and 46.5 on ADT, versus D4RT's 49.6 and 40.8, without retraining.Replacing Pi3 with VGGT-Ω improves APD-P by +6.6 on PStudio (30.1) and +6.4 on ADT (45.7).Source: §4.4 · p. 9
  • Forcing all points through the dynamic trajectory decoder leaves tracking accuracy at 31.2 APD-P but increases latency by 4.8× and peak GPU memory by 2.5×.The static/dynamic classifier achieves 93.1 and 92.5 F1 on Dynamic Replica and PointOdyssey, respectively.Source: §4.4 · p. 9
  • Removing iterative trajectory refinement reduces APD-P from 31.2 to 26.4 (−4.8), while removing 3D WAFT feature sampling reduces it to 29.3 (−1.9).Figure 4 shows dense tracks over visible points, including regions appearing after the first frame, while baselines leave large areas untracked or track only queried or first-frame points.Source: §4.4 · pp. 9–10
  • Voxel sizes from 0.005 to 0.02 retain approximately 31 APD-P, with v=0.02 selected; the chosen window length is L=16, with 31.2 APD-P.At voxel size 0.2, inference takes 0.03 s and uses 10.7 GB; L=8 and L=16 achieve 31.3 and 31.2 APD-P, while L=24 achieves 29.8.Source: §4.4 · p. 10

5 LIMITATIONS

TrackEverything’s de-duplication bounds memory growth with scene content, but memory still expands during open-world exploration. Its performance depends on external geometry, and irreversible proximity-based merging can collapse distinct surfaces.

5 LIMITATIONS

  • Memory continues to expand during perpetual open-world exploration, even though voxelized de-duplication bounds growth with scene content.The paper contrasts scaling to 1000+ frames with prior dense trackers failing beyond roughly 96 frames, but identifies hour-long trajectories as a next-step challenge.Source: §5 · p. 10
  • TrackEverything relies on external geometry, so performance remains coupled to pointmap fidelity despite sensor integration and reconstruction advances without retraining.Source: §5 · p. 10
  • Irreversible mean-pooling by spatial proximity can permanently merge distinct surfaces after tracking errors, leaving them impossible to disentangle in later frames.Users can protect important points through the non-voxelized query-point branch.Source: §5 · p. 10
  • The authors identify scaling model capacity and broadening pretraining data as likely ways to gain further on complex real-world dynamics.Source: §5 · p. 10

6 CONCLUSION

TrackEverything represents videos as persistent 3D scene tracks in world coordinates and uses de-duplication, endpoint-then-trajectory refinement, and 3D WAFT. It tracks all visible points across 1000+ frame sequences within 40 GB and outperforms open-source all-frame dense baselines on TAPVid-3D by over 20% APD.

6 CONCLUSION

  • Figure 4 shows TrackEverything tracking content that appears after the first frame, while sparse and first-frame dense baselines have a first-frame blind spot.The visualizations show dynamic-point trails only; static points are tracked but their trails are omitted for clarity.Source: Figure 4 · p. 11
  • 1000+ frame sequences can be tracked across all visible points within 40 GB of GPU memory, while TAPVid-3D performance exceeds open-source all-frame dense baselines by over 20% APD.Source: §6 · p. 11
  • Persistent world-coordinate scene tracks use voxelized de-duplication across sliding windows, endpoint-then-trajectory refinement, and 3D WAFT without costly 4D correlations.Dense trajectory decoding focuses on moving surfaces.Source: §6 · pp. 10–11

GENERATIVE AI DISCLOSURE

The authors report using generative AI for coding assistance and for writing and table restructuring, while reviewing AI-assisted work and code.

GENERATIVE AI DISCLOSURE

  • Generative AI assistance was limited to coding implementation, writing refinement, and table restructuring, with the authors reviewing the AI-assisted work and code.Source: GENERATIVE AI DISCLOSURE · p. 12
  • The authors state that generative AI was not used to design or provide feedback on the research methodology or experiments, interpret results, or perform qualitative or thematic analysis.Source: GENERATIVE AI DISCLOSURE · p. 12
  • The authors take responsibility for the final content, including text, claims, and artifacts produced with AI assistance.Source: GENERATIVE AI DISCLOSURE · p. 12

A.1 DENSE TRACKING ON DYNAMIC REPLICA AND POINTODYSSEY

The authors evaluate dense tracking on Dynamic Replica and PointOdyssey with estimated and ground-truth geometry, including 48-frame clips and full-length PointOdyssey videos. TrackEverything is competitive with DeltaV2 under estimated geometry, with larger reported gaps under ground-truth geometry.

A.1 DENSE TRACKING ON DYNAMIC REPLICA AND POINTODYSSEY

  • TAPVid-3D annotations evaluate a sparse set of query points, while the additional Dynamic Replica and PointOdyssey evaluation uses 48-frame clips.Source: §A.1 · p. 16, Table 3 · p. 16
  • 85.2 vs. 67.3 APD-P on Dynamic Replica and 73.2 vs. 33.2 on PointOdyssey are the reported TrackEverything-versus-DeltaV2 results with ground-truth geometry.Source: §A.1 · p. 16
  • 36.6 vs. 36.2 APD-P on Dynamic Replica and 24.9 vs. 23.8 on PointOdyssey show TrackEverything competitive with DeltaV2 under estimated Pi3 geometry.Source: §A.1 · p. 16
  • 31.8 APD-P, 74.5 APD-M, and 91.2% static/dynamic accuracy are reported on full-length 1000+ frame PointOdyssey videos with 10k+ labelled points.The reported F1 scores are 78.5% dynamic and 92.5% static; no quantitative baseline comparison is reported for this setting.Source: §A.1 · p. 16
  • Additional dense tracking results on diverse real-world scenes are available on the project page.Source: §A.1 · p. 16

A.2 INFERENCE SPEED BREAKDOWN

Optional static-only early merging removes redundant tokens before later refinement, nearly doubling tracker-only throughput while leaving full-length tracking accuracy unchanged.

A.2 INFERENCE SPEED BREAKDOWN

  • Early merging voxelizes static points after the endpoint refiner’s first iteration, preserving dynamic points and evaluated query cells for later trajectory refinement.The method relies on static/dynamic classification converging within one iteration; merged static tokens bypass trajectory refinement.Source: §A.2 · p. 16
  • 1.8× tracker-only throughput (12.6 to 23.1 FPS) with unchanged accuracy (31.3 vs. 31.2 APD-P) on full-length TAPVid-3D; end-to-end throughput rises 1.3×.With sensor or precomputed geometry, tracker-only throughput averages 23.1 FPS; the end-to-end rate includes VGGT-Ω geometry estimation.Source: Table 4 · p. 16, §A.2 · p. 17
  • Tracker-only throughput ranges from 8.4 FPS on expansive ADT environments to 41.5 FPS on focused PStudio scenes, with DriveTrack at 19.5 FPS.The reported per-dataset speeds reflect differing physical scene extents and unique-surface accumulation.Source: §A.2 · p. 17

A.3 TRAINING DATA AND MODEL SIZES

TrackEverything uses 61M parameters and is trained on Kubric, PointOdyssey, and Dynamic Replica, with a frozen 20M-parameter DINOv3-small backbone.

A.3 TRAINING DATA AND MODEL SIZES

  • TrackEverything has 61M parameters, including a frozen 20M DINOv3-small backbone, and is trained on Kubric, PointOdyssey, and Dynamic Replica.SpatialTracker-v2 uses those datasets plus 14 additional datasets; D4RT uses a 1B-parameter backbone and 144M-parameter decoder and trains on a broader set.Source: §A.3 · p. 17
  • Any4D trains on the three TrackEverything datasets plus Waymo-DriveTrack and four more datasets, while VDPM also uses a VGGT initialization and two static datasets.VDPM additionally trains on Kubric, PointOdyssey, and Waymo.Source: §A.3 · p. 17
  • TAPIP-3D and DeltaV2 train only on Kubric, whereas CoTracker3 uses Kubric and an additional 15k real-world videos.The passage reports training-data composition, not a controlled comparison of its effect on performance.Source: §A.3 · p. 17

A.4 ADDITIONAL METHOD AND IMPLEMENTATION DETAILS

These additional sections detail TrackEverything’s scene-memory and refinement implementation, PointOdyssey corrections, qualitative behavior, and supplementary benchmark and complexity analyses.

A.4 ADDITIONAL METHOD AND IMPLEMENTATION DETAILS

  • Sliding-window boundary voxelization merges co-located scene points to bound active tokens, while stored mappings and residuals support reconstruction of original dense trajectories.Earliest-observation birth metadata is retained, and completed windows release image features while keeping compact 3D tracks and mapping tables.Source: Figure 5 · p. 17, §A.4 · pp. 18, 20
  • Static/dynamic labels use world-space trajectory extent over each window, and classification errors can either add compute or hold a moving point stationary until a later window.A dynamic point is labeled when motion extent exceeds the stated threshold; a misclassified static point can pass through trajectory refinement as zero displacement.Source: §A.4 · p. 18
  • 3D WAFT projects candidate world-space points into image features; iterative updates use appearance mismatch to move dynamic points toward matching surface features.Zero-velocity projections can already align static points after accounting for camera motion; points outside the true field of view or occluded yield appearance mismatch for visibility prediction.Source: §A.4 · pp. 18–19
  • Training detaches recurrent state at window boundaries, accumulates local gradients, and performs one backward pass through the visual backbone after all windows.The unified baseline re-evaluation also found SpatialTracker-v2’s TAPVid-3D APD-M was over 20% higher than previously reported.Source: §A.4 · p. 20

A.5 FIXES TO THE POINTODYSSEY DATASET

  • PointOdyssey corrections reindexed extrinsics across 16 sequences, dropped point supervision on 502 frames with large track shifts, and excluded one jitter-corrupted sequence.Some noisy points remain uncorrected, and segmentation-mask bugs made mask-based label cleanup infeasible.Source: §A.5 · p. 20

A.6 QUALITATIVES

  • Qualitative examples show less coherent or drifting tracks without iterative refinement or WAFT, while static/dynamic labels can shift as objects start or stop moving.The figures also compare tracking on Breakdance, Dance-jump, and hand-manipulation sequences.Source: §A.6 · p. 21, Figure 7 · pp. 21–23, Figure 8 · p. 24

A.7 TAPVID-3D DETAILED METRICS

  • Tables 5 and 6 provide AJ, OA, EPE, APD-P, and APD-M metrics on TAPVid-3D for 48-frame and full-length settings, with conclusions similar to Table 1.The tables organize results across ADT, DriveTrack, PStudio, and average; full-length entries can be N/A when a method cannot run.Source: §A.7 · pp. 25–26, Table 5 · p. 25, Table 6 · p. 26

A.8 COMPLEXITY ANALYSIS

  • The APD-P complexity analysis repeats Figure 3’s evaluation, and its conclusions remain similar across video-length and query-count settings.Figure 10 plots accuracy, latency, and peak GPU memory against video length and query count; out-of-memory cases are marked with an x.Source: §A.8 · p. 26, Figure 10 · p. 27

Related papers

Loading 2609.30222v1…