PaperlayerView exact revision on arXiv

Source-linked AI summary

ARROW: Arbitrary Reconstruction and Tracking of 4D Observations in the Wild

Ilya Fradlin, Christian Schmidt, Jens Piekenbrinck, Karim Knaebel, Gonzalo Martin Garcia, Bastian Leibe

arXiv:2610.01314v1cs.CV

TL;DR

Geometry and tracking must establish correspondence across viewpoints, capture times, and visibility changes, but prior tracking models can retain temporal scope. ARROW uses content-derived identities to query arbitrary observations through a shared reconstruction-and-tracking model. It reports state-of-the-art 3D tracking, competitive reconstruction, and gains from broader training.

  • Problem

    Unified geometry and tracking models can retain the temporal scope of conventional tracking, although scene points may need to be matched across arbitrary viewpoints and capture times.

    Source: §1 · p. 2

  • Method

    ARROW uses input-derived ID tokens to associate independently decoded queries with a shared scene representation across arbitrary observations.

    Source: §1 · p. 2

  • Results

    ARROW sets a new state of the art in 3D tracking benchmarks while remaining competitive in 3D reconstruction, and outperforms dedicated trackers on adapted RGB-only MVTracker.

    Source: §1 · pp. 2–3, §5 · p. 10

  • Takeaways & Limitations

    Broadening training beyond temporally local observations can improve temporal tracking while supporting correspondence across a wider range of scene observations.

    Source: §1 · p. 3

    Queries share image context during decoding but do not interact directly, which the authors hypothesize may weaken performance in far-away or low-confidence regions.

    Source: E LIMITATIONS · p. 24

Abstract

from arXiv · show

Dynamic scenes may be captured by a moving camera, multiple video streams, or images taken at different times. These observations reveal complementary aspects of scene geometry and motion, yet bringing them together requires establishing correspondence across viewpoints, capture times, and visibility changes. We introduce ARROW, a feed-forward model that unifies 3D reconstruction and 3D point tracking from arbitrary image sets. At its core is a novel order-invariant querying approach, which allows the association of queries with observations across arbitrary inputs. We show that exposing the model to more diverse sets of inputs during training results in improved task performance. Moreover, the resulting model is capable of generalization to a wider range of tasks including multi-view tracking. Trained with this strategy, ARROW establishes a new state of the art in 3D tracking on WorldTrack and TAPVid-3D and outperforms dedicated multi-view trackers on an adapted RGB-only MVTracker benchmark, while remaining competitive across 3D reconstruction tasks. Code and weights are publicly available.

1 INTRODUCTION

ARROW addresses the separation between geometry reconstruction and point tracking by supporting correspondence across arbitrary observations without imposing an order. It combines content-derived identities with broader training inputs, and reports state-of-the-art 3D tracking results alongside competitive reconstruction performance.

1 INTRODUCTION

  • Earlier unified geometry-and-tracking models retained temporal tracking scope, while arbitrary viewpoints and capture times require correspondence beyond temporal continuity.Geometry constrains correspondence, and correspondence relates structure across views and time, although the problems have often been treated separately.Source: §1 · pp. 1–2
  • Content-derived ID tokens let ARROW associate queries with observations through a shared embedding space, avoiding fixed temporal identities and order-dependent associations.The tokens are derived from the input images and identify the queried source, target, and reference observations.Source: §1 · pp. 2–3
  • Each query independently attends to the full encoded context, allowing one shared model to support sparse trajectories as well as dense geometry and motion.The query formulation selects a source pixel and specifies a target capture time and reference coordinate system.Source: §1 · p. 2
  • Training across static image collections, dynamic videos, and multi-view sequences broadens supervision beyond local temporal neighborhoods and supports varied observation settings.Different cameras and distant capture times expose more varied appearances and more challenging correspondence problems.Source: §1 · p. 2
  • ARROW sets a new state of the art in 3D tracking on WorldTrack and TAPVid-3D, outperforms dedicated trackers on adapted RGB-only MVTracker, and remains competitive in 3D reconstruction.The reported ablations attribute benefits to both content-derived addressing and a more challenging training distribution.Source: §1 · pp. 2–3

2 RELATED WORK

Prior work addresses geometry, point tracking, and multi-view tracking through distinct assumptions and representations. ARROW combines arbitrary image-set inputs with independently specified sparse queries and a representation free of explicit temporal or camera indexing.

2 RELATED WORK

  • Geometry transformers estimate scene geometry, while point trackers follow correspondence; recovering geometry at each observation alone does not establish how physical objects move between observations.Image-plane trackers cannot resolve depth or separate camera motion from scene motion without additional geometry or camera information.Source: §2 · p. 3
  • Video motion methods support temporal interpolation, but represent motion across a video timeline rather than only the requested point states.Examples include shared motion bases and per-pixel B-spline trajectories.Source: §2 · p. 3
  • Multi-view trackers can reduce depth ambiguity and recover occluded points, but commonly assume calibrated, synchronized streams or external geometry.Dedicated trackers also do not consistently outperform monocular methods on TAPVid-MV.Source: §2 · pp. 3–4
  • ARROW combines static images, videos, and multi-view sequences with sparse queries independently specifying source, target, and reference, without explicit temporal or camera indexing.Its shared encoder and pointwise decoder support the combination of these capabilities in one model.Source: §2 · p. 4, Figure 3 · p. 4

3 METHOD

ARROW represents arbitrary RGB observations as an unordered set and uses queries to predict 3D point positions across source, target, and reference observations. Its shared encoder and pointwise decoder support reconstruction and tracking, trained with complementary data regimes and losses.

3 METHOD

  • The input is an unordered set of RGB images of a 4D scene; observation indices identify images without implying temporal order or separation.Distinct observations may share a capture time.Source: §3 · p. 4
  • Queries specify a source pixel and source, target, and reference observations, yielding the point’s 3D position at the target time in the reference coordinate system.The formulation supports outputs including tracks, point clouds, and camera parameters.Source: §3 · p. 5

3.1 ARROW ARCHITECTURE

  • The multi-view encoder creates global scene context and observation ID tokens, which the query encoder combines with pixel coordinates and local appearance before independent queries attend to context.The model is trained end-to-end with per-query losses and no explicit observation-identity supervision.Source: §3.1 · p. 5
  • On four DAVIS sequences, ARROW’s visualizations show clean 3D reconstructions and consistent long-term tracks alongside trajectory overlays.The figure compares V-DPM, 4RC, OmniX, and ARROW, with input frames at left.Source: Figure 4 · p. 6

3.2 TRAINING

  • Training samples static image collections, dynamic videos, and multi-view sequences, using known geometry, annotated tracks, and static backgrounds where available.For dynamic scenes without tracks, queries use the same source and target time to avoid incorrect motion supervision while retaining geometric supervision; losses supervise valid 3D, confidence, image-coordinate, visibility, normal, and displacement targets.Source: §3.2 · p. 6

4 EXPERIMENTS

The experiments evaluate ARROW on monocular and multi-view 3D tracking, camera pose estimation, reconstruction, and video depth. Results show strong tracking performance alongside competitive geometry estimates, with ablations examining the effects of the design and training choices.

Monocular 4D reconstruction and tracking

  • ARROW achieves the highest APD and lowest EPE across all four WorldTrack datasets, for both overall and dynamic-only tracking.WorldTrack uses 64-frame sequences with queries initialized in the first frame and reports global median scale alignment.Source: Table 1 · p. 7, §4 · p. 7
  • ARROW leads in average camera-space AJ and APD, and world-space APD and L1 on TAPVid-3D, while remaining competitive in OA.It slightly trails the baselines in OA; evaluation covers bidirectional tracking from arbitrary query frames over sequences of up to 300 frames.Source: Table 2 · p. 7, §4 · pp. 7–8

Multi-view 4D reconstruction and tracking

  • On the adapted RGB-only multi-view benchmark, ARROW leads on 11 of 12 dataset–metric combinations; DexYCB EPE is the exception.ARROW has the highest DexYCB APD and AJ; the benchmark uses RGB from four synchronized static cameras and 2D queries in visible source views.Source: Table 3 · p. 8, §4 · p. 8

3D reconstruction

  • ARROW achieves the lowest ATE on Sintel, leads all three pose metrics on Panoptic Studio, and has the lowest Bonn relative translation error while tying for lowest relative rotation error.For reconstruction, it has the lowest Chamfer distance and highest normal consistency on NRGBD, remains competitive on 7-Scenes, and leads on both Bonn video-depth metrics, KITTI AbsRel, and 7-Scenes δ1.Source: Table 4 · p. 8, §4 · p. 9, Table 5 · p. 9

Cumulative Ablations

  • Cumulative ablations show that identity-based querying improves tracking, non-sequential sampling raises multi-view APD by 28.85 percentage points over S3, and adding four static multi-view datasets improves every reported metric.The S4 non-sequential sampling change also improves monocular tracking, pose estimation, and reconstruction; S5 expands the mixture from 15 to 19 datasets.Source: §4 · pp. 9–10, Table 6 · p. 9

5 CONCLUSION

ARROW unifies reconstruction and tracking across arbitrary observations through order-invariant querying, and the paper reports state-of-the-art tracking alongside competitive reconstruction. The extended related work compares this approach with dense prediction, query-based tracking, long-range tracking, and multi-view methods.

5 CONCLUSION

  • ARROW uses content-derived ID tokens instead of order-dependent learnable embeddings to reconstruct and track across arbitrary observations, with broader training improving generalization and task performance.The conclusion reports state-of-the-art results on multiple tracking benchmarks and competitive 3D reconstruction; code and model weights are publicly available.Source: §5 · p. 10

A EXTENDED RELATED WORK

  • Dense 4D methods differ in temporal conditioning but retain dense-map outputs; Any4D (Karhade et al., 2026) limits motion to one reference frame.V-DPM (Sucar et al., 2026) conditions decoding on target time, while 4RC (Luo et al., 2026) conditions it on a source–target pair.Source: A EXTENDED RELATED WORK · p. 16
  • Fast Spatial Tracking and UniQuery4R (Chen et al., 2026) sample multi-scale source features, whereas ARROW forms queries from image coordinates and content-derived source, target, and reference identities.The passage reports ablation support for this design over interpolated encoder features.Source: A EXTENDED RELATED WORK · p. 16
  • Point4D (Jeon et al., 2026) propagates 3D endpoints across video chunks for occlusion queries, but retains sinusoidal frame-position encodings; ARROW focuses on tracking within one context.ARROW's content-derived identities accommodate variable context lengths, and its non-sequential training exposes it to large viewpoint changes.Source: A EXTENDED RELATED WORK · p. 17
  • Multi-view trackers can reduce occlusions with additional viewpoints, but several methods assume calibrated, synchronized video; OmniX (Jiang et al., 2026a) uses a scene-wide trajectory field.The passage also notes that multi-view reconstruction can support consistent novel-view video generation.Source: A EXTENDED RELATED WORK · p. 17

B.1 IMPLEMENTATION DETAILS

ARROW combines an image-set encoder with a pointwise query decoder and trains on varied image sets using multiple task losses. Its implementation specifies the query inputs, training augmentations, optimization, and loss normalization.

Architecture

  • The DA3-Giant encoder supplies final-layer context to an eight-layer pointwise decoder, whose queries embed a 9 × 9 source-centered RGB crop, Fourier coordinates, and source–target–reference identity tokens.The decoder has hidden dimension 1280 and 16 attention heads; identity tokens are LayerNorm-normalized, concatenated in source–target–reference order, and projected by an MLP.Source: §B.1 · p. 17

Training inputs and augmentation

  • Training samples sets of 2–24 images, uses 1024 queries per input image, and applies perspective warping, color jitter, JPEG compression, and blur augmentations.Images are resized with a uniformly sampled aspect ratio from 0.75 to 3.0 and a longer dimension of 504 pixels.Source: §B.1 · p. 17

Optimization

  • The model is trained for 200k steps on 32 NVIDIA GH200 GPUs with AdamW, layer-wise backbone learning rates, warmup, cosine decay, and an exponential moving average.The backbone is frozen for the first 4000 steps; the layer-wise peak rate ranges from approximately 1 × 10^-5 in the first block to 5 × 10^-5 in the final block.Source: §B.1 · p. 17

Loss formulation

  • The loss combines 3D point, confidence, UV, visibility, normal, and displacement terms, with targets normalized by mean distance from their reference-camera origins.Independent normalization removes global scale ambiguity, while a signed logarithm reduces distant points' influence; UV and visibility targets remain unchanged when the reference observation changes.Source: §B.1 · p. 18
  • The displacement target is the source-to-target world vector rotated into the reference camera and divided by scene scale; its radial logarithm compresses magnitude while preserving direction.Scene scale is the mean distance of valid scene points from the first input camera.Source: §B.1 · p. 18

B.2 TRAINING DATA

ARROW’s training mixture balances dynamic and static scenes, while its query-sampling strategy combines relational supervision with dataset-specific observation selection. Sampling also targets geometric edges and motion boundaries to improve local and dynamic-scene supervision.

B.2 TRAINING DATA

  • The training mixture contains approximately 78K scenes and 22.8M camera frames, with 78% of sampling weight assigned to dynamic data and 22% to static data.A dataset is sampled according to its mixture weight, followed by uniform scene sampling within that dataset.Source: §B.2 · p. 18

B.3 SAMPLING STRATEGY

  • The relational query interface separates context observations from pixels or track identities used for supervision, allowing static geometry, dynamic trajectories, and scenes without tracks to share one objective.Training sets contain 2–24 images; selection uses local temporal windows, cross-camera samples, covisibility, or random sampling according to dataset.Source: §B.3 · p. 19, Table 8 · p. 19
  • Geometry queries oversample pixels near depth or normal discontinuities, which are sparse under uniform sampling and important for sharp transitions and thin structures.The method detects relative depth jumps above 5% and normal changes above 25°; up to 30% of the geometry budget is sampled within three pixels of an edge, while positive base weights preserve coverage elsewhere.Source: §B.3 · pp. 19–20, Figure 6 · p. 20
  • Omitting normal-edge supervision led to more rounded reconstructions at sharp transitions; ARROW accurately recovers such structures, which the authors attribute to pixel-level decoding and dedicated edge supervision.Source: §B.3 · p. 20
  • Motion and boundary sampling allocates queries to a nine-pixel band around moving-object boundaries, dynamic interiors, and background geometry; static-side sampling addresses a failure mode where nearby background can deform with the moving object.Dynamic queries include valid visible source states and valid target states, including occluded ones; ordinary dynamic-region queries are generally same-time, with specified exceptions for trusted static backgrounds and Waymo.Source: §B.3 · p. 20

C.1 EVALUATION PROTOCOLS

The protocols specify how tracking and multi-view comparisons are sampled, scored, and adapted, and describe a query-based procedure for recovering camera poses.

C.1 EVALUATION PROTOCOLS

  • WorldTrack queries start at visible positions in frame one and are evaluated over 64 frames; scoring uses global median scale alignment, APD and EPE, with dynamic-only evaluation on tracks moving over 1 cm.APD averages the share of valid positions within 0.1, 0.3, 0.5, and 1.0 meters; EPE is mean Euclidean error.Source: §C.1 · pp. 20–21
  • The RGB-only multi-view adaptation evaluates four static cameras over synchronized timesteps, selecting visible source views and scoring one 3D position per physical timestep in the first camera’s coordinates.Panoptic Studio uses every second frame from the first 48; the adaptation’s protocol and input assumptions preclude direct numerical comparison with official MVTracker results.Source: §C.1 · pp. 21–22

C.2 BASELINE ADAPTATION FOR MULTI-VIEW TRACKING

  • The adapted comparisons use predicted geometry to lift pixel queries for external-geometry trackers, while 4RC and OmniX are also evaluated under the shared adapted protocol.MVTracker, TAPIP3D, and LAPA use different geometry and inference configurations; AJ and OA are omitted for 4RC and OmniX because they lack native visibility predictions.Source: §C.2 · p. 22

C.3 CAMERA POSE RECOVERY

  • Camera pose recovery decodes corresponding source-grid queries in two cameras, filters invalid predictions, and uses an SVD solve to estimate rotation and translation without rescaling.The middle input is the reference camera and receives the identity pose.Source: §C.3 · p. 22

D.1 FURTHER ABLATIONS

The ablations examine query design, training choices, context length, and inference windows. They report benefits from fine-tuning and full-context inference, with Stage S5 strongest on average in short-context evaluation.

D.1 FURTHER ABLATIONS

  • Freezing the encoder and training only the decoder reduces mean APD by 8.71 percentage points overall.The ablation indicates that encoder fine-tuning is important for optimal tracking performance.Source: §D.1 · p. 23
  • EMA slightly improves average all-point APD and EPE, while multi-level decoding gains are small on average and vary across datasets.The final model retains EMA and shared final-layer context for simplicity and memory efficiency.Source: §D.1 · p. 23
  • Patch-feature query initialization converges faster initially but yields lower final tracking accuracy and more patch-like spatial predictions.The observations favor initializing query appearance and spatial components independently of encoder patch features.Source: §D.1 · p. 23
  • Stage S5 achieves the best average all-point APD and EPE on 24-frame WorldTrack sequences, although earlier stages lead on some individual datasets.All stages use full-context inference within the 2–24-image training range.Source: §D.1 · p. 23, Table 10 · p. 24

D.2 LONG-RANGE TRACKING

  • For ARROW, full-context inference yields higher APD than 48-frame windows at every evaluated prefix longer than one window, with the gap widening as sequences grow.The figure reports full-context and windowed inference; V-DPM’s 200-frame evaluation exceeded 80 GPU-hours and other missing points ran out of memory.Source: §D.2 · p. 23, Figure 8 · p. 24

E LIMITATIONS

The limitations identify boundaries in temporal outputs, dense prediction cost, query interactions, domain generalization, precision, and PointOdyssey supervision.

E LIMITATIONS

  • ARROW supports outputs only at times corresponding to its input observations, and removing explicit time encodings may limit temporal interpolation or extrapolation.The authors describe the interpolation and extrapolation effects as possible limitations.Source: E LIMITATIONS · p. 24
  • Dense decoding has higher computational costs, despite sparse decoding and query batching alleviating memory bottlenecks.Improving query-based dense prediction efficiency remains future work.Source: E LIMITATIONS · p. 24
  • Queries share image context but do not interact directly; the authors hypothesize this may weaken predictions in far-away or low-confidence regions.The proposed effect is stated as a hypothesis.Source: E LIMITATIONS · p. 24
  • PointOdyssey includes floating objects, flattened background geometry, and depth artifacts that fragment geometry, motivating manual review and exclusion of unreliable supervision.The examples include background images mapped to flat planes and noisy depth annotations.Source: Figure 9 · p. 25
  • In multi-view tasks, combining in-distribution views with out-of-distribution inputs such as egocentric views degrades performance.The authors identify addressing such domain gaps as future work.Source: E LIMITATIONS · p. 25
  • Pose and image-coordinate estimates are globally accurate, but fine-grained precision remains sensitive to local variations.The authors suggest feature matching or local optimization as possible ways to improve precision.Source: E LIMITATIONS · p. 25

F DATA PREPROCESSING

The preprocessing pipeline addresses dataset-specific annotation and geometry problems, then uses scene metadata and sampling rules to manage unreliable regions and correspondences.

F DATA PREPROCESSING

  • PointOdyssey scenes are manually reviewed for static, dynamic, or unreliable instances and geometry issues; mask refinement and scene-specific sampling profiles handle problematic regions.Unreliable regions receive an ignore label, and profiles distinguish standard, flat-background, and moving-background scenes.Source: F DATA PREPROCESSING · p. 25
  • Hypersim camera orientation matrices that are not proper rotations are corrected using SVD, and normal-map axes are reconciled and oriented toward the camera.The correction avoids treating a reflection as a camera rotation.Source: F DATA PREPROCESSING · p. 25
  • Syn4D correspondences are decoded into shared world-space trajectories, Dynamic Replica labels are mapped to persistent identities, and ParallelDomain-4D tracks are validated across modalities.Validation checks forward–backward, cross-view depth, flow, and instance consistency.Source: F DATA PREPROCESSING · p. 25
  • Other dataset-specific preparation removes simulated vignetting, rectifies ASE cameras, filters unreliable geometry or depth, and builds depth-reprojection covisibility graphs to guide sampling.Dynamic identities remain defined by each dataset’s track correspondences.Source: F DATA PREPROCESSING · p. 25

G ADDITIONAL QUALITATIVE RESULTS

The qualitative examples cover video depth, UV tracking, 3D reconstruction, multi-view 3D tracking, and varied dynamic, static, and monocular scenes. They illustrate fine-detail preservation and tracking through occlusion and motion.

G ADDITIONAL QUALITATIVE RESULTS

Related papers

Loading 2610.01314v1…