PaperlayerView exact revision on arXiv

Source-linked AI summary

TAPVid-MV: A Benchmark for Tracking Any Point in 3D Across Multiple Views

Skanda Koppula, Frano Rajic, Abdullah Faiz Ur Rahman, Yi Yang, Ignacio Rocco, Jeet Thakwani, Rishabh Kabra, Andrew Zisserman, Joao Carreira, Siyu Tang, Carl Doersch, Gabriel Brostow

arXiv:2609.01899v1cs.CV

TL;DR

Existing benchmarks do not test long-term 3D point tracking across several synchronized views with camera motion, nor cleanly separate correspondence from geometry errors. TAPVid-MV introduces such a benchmark with diverse, manually verified trajectories and evaluates both tracking and reconstruction; across more than 30 baselines, performance remains poor and recovered geometry is a major bottleneck.

  • Problem

    Existing benchmarks omit long-term 3D point tracking across several synchronized views under camera motion and cannot separate correspondence errors from geometry errors.

  • Method

    TAPVid-MV provides 284 sequences, 1,142 calibrated camera streams, and 109,769 verified 3D trajectories, evaluating tracking across camera and world frames alongside reconstruction quality.

  • Results

    Across more than 30 baselines, no method comes close to solving the task; multi-view trackers do not consistently outperform monocular trackers, and geometry recovery is a major error source.

  • Takeaways & Limitations

    Jointly measuring reconstruction and tracking lets TAPVid-MV diagnose geometry bottlenecks separately from correspondence errors and support related tracking and reconstruction tasks.

  • Takeaways & Limitations

    Long-horizon temporal drift and identity swaps during multiobject contact remain largely unsolved even with the most accurate reconstruction.

Abstract

from arXiv · show

Multi-camera systems are increasingly practical for robotics, AR/VR, and autonomous driving because complementary views reduce depth ambiguity and preserve visibility under occlusion. Existing point-tracking benchmarks, however, focus on a single video or static multi-camera rigs. None test long-term 3D point tracking across several synchronized views under camera motion. We introduce TAPVid-MV (Tracking Any Point in Video across Multiple Views), the first benchmark for this setting. It contains a curated set of 284 sequences, 1,142 calibrated camera streams, and 109,769 point tracks across seven subsets spanning indoor and outdoor domains, from robotics and human activity to driving and synthetic procedural scenes. We obtain these trajectories using dataset-specific auxiliary modalities: sensor depth, LiDAR, SLAM and SfM points, human meshes, posed object meshes, and simulation. Every sequence and trajectory is visually verified by human annotators. Across more than 30 baselines, no method comes close to solving the task. Surprisingly, existing multi-view point trackers do not consistently outperform monocular point trackers. By evaluating reconstruction and point tracking on the same datasets, TAPVid-MV helps distinguish errors in recovered geometry from errors in point correspondence. Through this joint analysis, we identify geometry recovery as a major bottleneck for accurate 3D point tracking. Beyond multi-view 3D point tracking, our released annotations support monocular 2D and 3D point tracking, future-trajectory prediction, and 4D reconstruction.

1. Introduction

TAPVid-MV addresses the missing benchmark for long-term 3D point tracking across synchronized, moving multi-camera views. Its evaluation and results show that geometry recovery remains a major obstacle, while current methods leave the task far from solved.

  • Existing benchmarks omit long-term 3D point tracking across several synchronized views with camera motion and changing layouts.
  • TAPVid-MV contains 284 sequences, 1,142 calibrated camera streams, and 109,769 ground-truth 3D trajectories across seven indoor and outdoor subsets.Trajectories use dataset-specific depth, LiDAR, reconstruction, mesh, or simulation signals and are manually validated.
  • The benchmark evaluates tracking in query-view, non-query-view, and world coordinate frames while measuring reconstruction quality on the same data.
  • More than 30 baselines remain far from solving the task, and multi-view trackers do not consistently outperform monocular trackers.Monocular TAPIP3D leads aggregate query-view and world-space accuracy, while performing comparably to MVTracker in non-query views at 20.5 versus 20.7.
  • Replacing SpatialTrackerV2’s native reconstruction with shared VGGT-Ω reconstruction nearly doubles aggregate query-view accuracy from 11.8 to 23.5.This isolates recovered geometry as a major source of tracking error.
  • Future work should improve reconstruction from few moving cameras, exploit cross-view information, and address drift and identity swaps during object contact.The released resources also support monocular tracking, future-trajectory prediction, and 4D reconstruction.

2. Related Work

Prior work covers monocular tracking, monocular metric 3D tracking, and several multi-view settings, but lacks a benchmark combining long-term 3D tracking with multiple independently moving cameras and measured reconstruction quality.

  • Tracking Any Point predicts the pixel trajectory of a queried surface point through monocular video, while TAP-3D extends this to metric 3D trajectories.
  • Multi-view TAP-3D requires long-term, pixel-level 3D trajectories across multiple views in a common world frame.
  • Trajectory scores alone cannot distinguish correspondence errors from inaccurate recovered geometry, although current 3D trackers use both.
  • Existing real-world multi-view benchmarks use static cameras or moving rigid rigs, and none combines more than two views with independently moving cameras.TAPVid-MV evaluates reconstruction and tracking on the same data.

3. Multi-View TAP-3D

Multi-View TAP-3D asks models to track queried physical points through synchronized RGB streams from moving cameras, producing visible 3D trajectories in a shared world frame without ground-truth depth or poses.

  • The task provides synchronized RGB streams of a dynamic scene captured by multiple cameras that may move, plus query points.
  • The task is designed for long-term, pixel-level 3D tracking of rigid or deformable objects from multiple RGB views.
  • Each query identifies a physical surface point in one view and frame, without corresponding queries supplied in the other views.This avoids assuming that the same physical point has already been identified across cameras.
  • Models must predict each query’s 3D trajectory across all frames in one shared world coordinate frame, together with visibility in every view and frame.
  • Evaluation uses average Jaccard AJ3D, average 3D location accuracy δ3D, and occlusion accuracy OA across query-view, non-query-view, and world-space regimes.World-space scoring penalizes cross-view misalignment and pose drift; predictions are rescaled and aligned before scoring.
  • Because ground-truth depth and camera poses are not inputs, predictions are evaluated up to a similarity transform.

4. The TAPVid-MV Benchmark

TAPVid-MV combines seven heterogeneous subsets spanning real, rendered, and synthetic scenes with moving multi-camera configurations. Tracks are constructed from reliable auxiliary signals and manually checked across views.

  • The benchmark contains 284 sequences and 1,142 views across seven subsets, with domains including robotics, human activity, driving, and procedural scenes.
  • Tracks are constructed from robot kinematics, stereo, human meshes, posed object meshes, LiDAR, SLAM points, or exact synthetic geometry, followed by filtering and manual inspection.
  • Each scene uses three to eight cameras, every subset includes camera motion, and three subsets include egocentric views.
  • DROID contributes 5,371 robotic training and pretraining episodes plus 50 evaluation episodes using stereo, robot kinematics, and camera calibration refinement.
  • Ego-Exo4D supplies 500 tracks jointly visible from three views for at least 50 frames, combining human fits, SLAM points, depth, and visibility masks.
  • PACE provides synchronized three-camera RGB-D recordings of 238 household objects from 43 categories, including touching, overlapping, and moved objects.

5. Baselines and Results

Across more than 30 baselines, monocular trackers on a shared reconstruction generally outperform purpose-built multi-view and joint models, while reconstruction quality and cross-view correspondence remain major bottlenecks.

  • Baseline results: No method exceeds 6.6 on Waymo or 5.9 on Hi4D, despite these trackers topping existing monocular 2D and 3D benchmarks.Query-view accuracy is substantially stronger than world-space performance on these challenging subsets.
  • Multi-view baselines: Existing multi-view trackers do not consistently beat monocular trackers: MVTracker reaches δ3D N of 20.8 versus 20.6 for monocular TAPIP3D, but trails it on δ3D Q at 24.0 versus 27.1.MV-TAP trails the best lifted monocular 2D trackers on every metric, while cross-view fusion yields almost nothing on real moving-camera footage.
  • Geometry and tracking: Changing geometry affects accuracy far more than changing the monocular tracker: twenty trackers on one reconstruction fall within 2.5 points on DROID, whereas ground-truth depth raises TAPIP3D from 14.0 to 40.3.On Hi4D, the corresponding ground-truth-depth improvement is from 5.9 to 62.9, although even perfect depth leaves a ceiling of 40.3 on DROID.
  • View scaling: Additional views improve query-view tracking when they improve reconstruction, but Harmony4D and Perpetua show mixed scaling and no model fully exploits cross-view evidence.The gains are therefore gated by geometry quality rather than guaranteed by adding cameras alone.
  • Cross-view tracking: Cross-view correspondence remains unsolved: the best lifted tracker drops from 26.7 δ3D Q to 17.8 δ3D N overall, with larger drops on Perpetua and Waymo.With ground-truth depth on Perpetua, it still falls from 64.1 δ3D Q to 20.5 δ3D N, indicating a correspondence problem beyond reconstruction.

6. Conclusions

TAPVid-MV is introduced as a curated benchmark for multiview 3D point tracking, with released artifacts intended to support diagnosis and future progress across related 3D tasks.

  • The benchmark helps diagnose deficiencies in current approaches to 3D tracking.
  • The benchmark supports multiple 3D tasks, including posing and reconstruction.
  • Released artifacts include baselines, data, data-generation tools, and evaluation protocols for advancing model capabilities.

Supplementary Material

The supplementary material contains additional results, formalizations, implementation details, dataset statistics, visualizations, and DROID generation information.

  • The supplementary material includes additional results and extended task and metric formalization.
  • It documents baseline implementation details and dataset statistics.
  • It provides per-sequence track visualizations and DROID generation pipeline and dataset details.

A. Additional Results

Additional analyses show that PointOdyssey training transfers to Perpetua, while comparisons require careful attention to reconstruction quality, training data, and baseline strength.

  • OmniX achieves Perpetua reconstruction scores of EC 92.8 and DC 28.0, indicating transfer across differing environments, motions, and actors.
  • Recent joint 4D models are often compared against weaker modular pipelines than those evaluated here.
  • Existing comparisons do not control for training data, whose mixtures differ widely across methods.

B. Extended Task and Metric Formalization

TAPVid-MV formalizes synchronized multi-view 3D point tracking as predicting world-frame trajectories and per-view visibility from RGB streams and pixel queries. Its metrics separately assess query-view, cross-view, and globally consistent tracking while accounting for scale and reconstruction quality.

  • Task inputs: The model receives V synchronized RGB streams and P queries, each anchored to one pixel, view, and timestep.Corresponding views are synchronized within 1/FPS, and each query specifies (x_p, y_p, t_p, v_p).
  • Task outputs: Each query requires a 3D trajectory in one shared world frame plus optional per-view visibility predictions.Without ground-truth depth or extrinsics, predictions are defined up to a similarity transform and aligned before scoring.
  • Query sampling: Queries are sampled uniformly from visible frame-view pairs, while tracks invisible in every view are excluded.Every retained track is therefore queryable from at least one observed view.
  • Accuracy diagnosis: Reconstruction quality strongly governs tracking accuracy, although accurate geometry alone is insufficient.Replacing a joint model’s geometry with VGGT-Ω improves δ3D Q from 11.7 to 23.5, while V-DPM matches VGGT-Ω pose accuracy but tracks worse on PACE.
  • Per-view metrics: Per-view metrics average AJ3D, δ3D, and OA across cameras using the TAP-3D depth-adaptive threshold δ3D = δ2D · Z/f.Per-view scaling and thresholds assess each camera frame independently.
  • World-space metrics: World-space metrics evaluate all queries in a common frame with a fixed sequence threshold and global rescaling or Sim(3) alignment.The protocol prevents independently flattering each camera or aligning directly to predicted tracks.

C. Baseline Implementation Details

The baseline pipeline standardizes video resolution, reconstruction inputs, query handling, and scoring across monocular, lifted, multi-view, and joint methods. Comparisons distinguish shared-reconstruction tracking from methods that estimate geometry themselves.

  • Shared inputs: Point-tracker baselines use a shared VGGT-Ω reconstruction supplying per-view depth maps and cameras at a common evaluation resolution.Sequences are resized with 512 pixels on the longer side, with selected square crops for some datasets.
  • Evaluation setup: The evaluation reports tracking results alongside camera and depth quality, including query-view, non-query-view, and visibility metrics.Appendix tables identify supported metrics, privileged oracle-query rows, and reconstruction quality for joint methods.
  • Tracking procedure: Each tracker runs on the original and time-reversed videos, merging the two half-tracks at the query timestep.Predicted trajectories are transformed into source-camera coordinates for camera-space evaluation.
  • Monocular baselines: Monocular adapters process only the query view, then evaluate the lifted trajectory across all views and in world space.This restriction directly affects non-query-view and world metrics, while visibility uses the query-view estimate.
  • 2D-to-3D lifting: 2D trajectories are lifted through source depth, falling back to the most recent or first valid depth when predictions leave the image or encounter holes.Tracks with no valid depth remain invalid and count as incorrect.
  • Multi-view 3D trackers: MVTracker jointly consumes all views’ RGB-D and cameras to track query points directly in world space, while LAPA uses cross-view geometric attention instead of triangulation.MVTracker additionally applies gravity-aligned scene normalization; LAPA is trained on static calibrated multi-camera data.

F.5. Quality Validation and Dataset Statistics

The validation pipeline measures physical consistency against sensor depth rather than relying on reprojection alone. It processes thousands of robotic episodes and filters the benchmark split using geometry-quality thresholds and stratified sampling.

  • Validation metric: Depth Residual compares projected 3D-track depth with raw stereo sensor depth across visible frames.Because tracks are generated by direct projection in this pipeline, 2D reprojection error is an identity check rather than a physical-quality measure.
  • Interpretation: Low residuals characterize static background points, while robot residuals reflect forward-kinematics precision and fast motion.The metric therefore captures agreement with raw sensor measurements under distinct static and dynamic conditions.
  • Dataset statistics: 5,371 episodes were successfully processed, a 96.3% completion rate from 5,580 attempts, yielding over 16,000 calibrated streams and millions of trajectories.The DROID-50 split applies Chamfer < 0.05 and Depth Residual < 20 mm thresholds before stratified sampling.

G. Dataset Generation: Additional Details

The dataset combines auxiliary geometric modalities with automated filtering and human review across diverse human, robotic, object, driving, and synthetic scenes. Construction addresses mesh misalignment, missing correspondences, camera-frame recovery, and visibility verification.

  • World-frame recovery: PACE requires recovering a static world frame because its released camera extrinsics are rig-relative despite motion of the hand-carried rig.Stationary objects provide the reference for representing the rig trajectory.
  • Verification and cleaning: Track cleaning removes reconstruction collapses and trajectories whose 2D evidence contradicts the 3D label.Mesh-based tracks are checked against image-tracking consensus, especially around articulated extremities where fitted meshes may misalign.
  • Hi4D: Hi4D tracks use persistent SMPL body-surface anchors, scan intersections, ray-cast visibility, and interactive human review.Candidates are filtered for sufficient scan validity and visibility before selecting spatially diverse tracks.
  • Hi4D review statistics: Hi4D review covered 12,857 of 28,800 candidates, rejected 7,629, and required 20.7 hours across 48 sequences.Rejection rates ranged from 27.7% to 74.0%, varying with occlusion and motion.
  • Procedural scenes: The procedural indoor subset jointly generates collision-aware actor routes, moving camera trajectories, synchronized renderings, and exact mesh-based tracks.Visibility uses rendered depth and object identity to account for self- and cross-object occlusion.
  • Multi-camera scenes: Soccer and driving tracks combine calibrated multi-camera views with human meshes, SLAM or VGGT-Ω points, LiDAR, and explicit visibility tests.Sampling requires joint visibility across views, including at least two views for driving tracks.
  • Companion split: The static-only Hi4D companion split reuses manually verified tracks with the original eight-view static RGB videos and matching calibration metadata.It preserves the same labels while evaluating the public static captures.

I. Raw File Specification

TAPVid-MV sequences are distributed in a per-sequence directory with shared track and query files plus one repeated directory for each camera view. Each view stores images, visibility, camera parameters, and optional auxiliary data.

  • Directory layout: Each sequence directory contains tracks_xyz.npy, queries_xytv.npy, and a repeated <view_id>/ directory for every camera view.The view directories contain the per-camera files listed separately.
  • Per-view files: Each camera view stores JPEG image bytes, intrinsics, world-to-camera extrinsics, and visibility data.The image array has one object per frame, visibility is indexed by frame and point track, intrinsics are constant across frames, and extrinsics are per-frame 4×4 matrices.
  • Optional data: Optional per-view files provide depth and foreground masks.These files are marked optional in the directory specification.

J. Limitations

The benchmark’s main limitations concern annotation precision, missing highly deformable objects, unisolated training-resource effects, and reliance on one scaling protocol.

  • Annotation precision and low-threshold metrics: Absolute label error is unavailable for several real-sequence subsets, so strict-threshold scores may partly reflect annotation noise.The affected subsets rely on estimated depth, calibration, fitted meshes, or SLAM/SfM geometry; low-threshold results should be read with larger-threshold and aggregate metrics.
  • Highly deformable objects: The benchmark lacks precise annotations for highly deformable objects such as cloth and other soft materials.Rigid poses and standard articulated meshes do not provide adequate supervision for persistent metric surface correspondences on these objects.
  • Training-data utility: The contribution of the released Perpetua and DROID training resources has not been isolated through controlled experiments.The authors report initial gains from a Perpetua-containing mixture but have not trained on released DROID trajectories or established transfer effects.
  • Scaling protocol: Reported comparisons use query-median scaling, while alternative scaling protocols and ranking stability remain untested.Thus, the comparisons are specifically conditioned on the chosen scale-alignment protocol.
Loading 2609.01899v1…