PaperlayerView exact revision on arXiv

Source-linked AI summary

Point4D: Long-range 4D Motion Reconstruction

Minsik Jeon, Jay Karhade, Deva Ramanan, Shubham Tulsiani

arXiv:2609.09145v1cs.CV

TL;DR

Existing feed-forward 4D methods are limited to short windows, making long-video reconstruction challenging when full-sequence processing is computationally infeasible and points become occluded. Point4D uses chunked prediction with 3D queries to reconstruct long-range motion, achieving state-of-the-art performance on long-video tracking benchmarks.

  • Problem

    Existing feed-forward 4D methods use short input windows, while processing long videos is computationally infeasible and tracked points often become occluded or leave the field of view.

    Source: §1 · p. 2

  • Method

    Point4D predicts motion from 3D-coordinate queries and chains predicted endpoints across overlapping video chunks, reusing a descriptor from a frame where each point is visible.

    Source: §1 · pp. 2–3

  • Results

    Point4D achieves state-of-the-art long-video 4D tracking, outperforming chaining-based feed-forward 4D methods and 3D point trackers.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The authors present Point4D as a foundation step toward reliable 4D reconstruction on in-the-wild videos of arbitrary length.

    Source: §5 · p. 9

    A point that remains occluded or out of frame for an entire chunk lacks supporting evidence, making its predicted position unreliable; errors in predicted depth can also compound across chunks.

    Source: §5 · p. 9

Abstract

from arXiv · show

We introduce Point4D, a feed-forward model for 4D reconstruction of long-range video sequences. Point4D is able to reliably infer dense per-point 3D trajectories across multi-hundred-frame videos, unlike existing 4D methods that are limited to short input windows of at most a few dozen frames. A key innovation that enables this is our flexible 3D query-based motion decoder that decouples trajectory prediction from image-plane visibility. The predicted 3D endpoints are then directly re-queried in the next chunk without re-projection or matching. Furthermore, we show that extracting and reusing a visual descriptor from an arbitrary frame where the point is visible leads to better performance than relying solely on the source patch. Overall, Point4D achieves state-of-the-art performance across diverse long-video tracking benchmarks spanning over 200 frames and largely outperforms previous feed-forward 4D method. Project page: https://point-4d.github.io

1 Introduction

Long-range 4D reconstruction must recover per-point 3D trajectories across hundreds of monocular-video frames despite computation limits and visibility changes. Point4D addresses this with chunked prediction and 3D queries that can be chained directly.

1 Introduction

  • Existing feed-forward 4D methods use short input windows, while long videos challenge both full-sequence transformer computation and tracking through occlusion or field-of-view changes.These challenges arise when jointly processing hundreds of frames and when points become unobservable between observations.Source: §1 · p. 2
  • 3D point queries allow direct cross-chunk re-querying without reprojection or matching, including when points are occluded or outside the field of view.Unlike image-plane queries, 3D queries decouple trajectory prediction from visibility.Source: §1 · pp. 2–3, Figure 2 · p. 2
  • Reusing a visual descriptor extracted from any prior frame where a point is visible improves performance over relying only on the source patch.The descriptor is reused while predicted endpoints are chained across chunks.Source: §1 · p. 2
  • A feed-forward pipeline chains per-point motion across video chunks to reconstruct trajectories spanning hundreds of frames.Predicted 3D endpoints are re-queried in the next chunk.Source: §1 · p. 3
  • Point4D achieves state-of-the-art long-video 4D tracking, outperforming chaining-based feed-forward 4D methods and 3D point trackers.The model also improves over its 2D-query counterpart for short-range motion reconstruction.Source: §1 · pp. 2–3

2 Related Work

Prior work addresses tracking, dynamic-scene reconstruction, and long-sequence 3D reconstruction through different representations and strategies. Point4D extends query-based motion decoding to 3D so tracks can be chained across longer videos.

3D Tracking

  • Long-range 2D trackers use correlation-based matching and iterative updates, while 3D trackers lift points using depth or camera-stabilized features; Point4D instead predicts dense trajectories in one forward pass.Point4D remains competitive with iterative trackers on sparse benchmarks.Source: §2 · p. 3

4D Reconstruction of Dynamic Scenes

  • Feed-forward 4D reconstruction predicts motion from short frame windows, but image-plane queries require reprojection to extend tracks and fail under occlusion; Point4D uses 3D queries for longer-range chaining.This extends D4RT’s query-based decoding from 2D to 3D.Source: §2 · p. 3

Long-range 3D reconstruction

  • Long-sequence 3D reconstruction methods chain static geometry across chunks; Point4D extends chunking to 4D by chaining motion with 3D point queries.The approach recovers dense scene flow and long-range 3D tracks in a feed-forward manner.Source: §2 · p. 3

3 Method

Point4D reconstructs long-range 3D trajectories by querying motion in 3D rather than from image pixels. It chains predictions across overlapping chunks, keeping point identities valid through occlusion and field-of-view exits.

3.1 Preliminaries

  • A shared video encoder produces a scene representation with per-frame patch, camera, and time tokens, as well as depth maps and camera poses.The scene representation supports cross-attention decoding; the paper describes prior D4RT queries as 2D pixels whose source-frame visibility is required.Source: §3 · p. 3, §3.1 · p. 4

3.2 3D Query-based Decoder

  • The decoder uses a 3D source-coordinate query and a visual patch from any frame where the point is visible, then predicts its 3D position at a target time.The descriptor is reused across chunks, so the query remains valid when the point is occluded or outside the current field of view.Source: §3.2 · p. 5
  • The training objective combines 3D point, confidence, 2D reprojection, and target-time visibility losses.The point loss compares signed-log-transformed predicted and target 3D positions; reprojection enforces consistency with camera geometry.Source: §3.2 · p. 5

3.3 Trajectory Chaining

  • Point4D chains long-range trajectories by re-querying predicted 3D endpoints across overlapping video chunks.It aligns chunk coordinate frames using depth predictions from shared overlap frames, then transforms each endpoint into the next chunk’s coordinates.Source: §3.3 · pp. 5–6

3.4 Implementation Details

  • Training initializes the encoder and geometry heads from Depth Anything 3 weights, trains the decoder from scratch, and uses sequences of 16 to 64 frames.The training mixture includes dynamic and static datasets, with static points treated as stationary trajectories.Source: §3.4 · p. 6

4 Experiments

The experiments evaluate long-video trajectory chaining and single-chunk tracking, then test query design and accuracy degradation across chunks. Point4D performs best on long-video tracking while remaining comparable to other methods on short sequences.

4.1 Experimental Setting

  • The evaluation covers long-video chaining on 200-frame PointOdyssey and Dynamic Replica sequences and 150-frame PStudio sequences, plus single-chunk sequences up to 64 frames.Long videos are divided into 48-frame chunks with 8-frame overlap; metrics include EPE, APD, and long-video survival rate.Source: §4.1 · pp. 6–7

4.2 4D Tracking

  • Point4D achieves the best average rank on long-video tracking, outperforming every feed-forward method and running much faster than iterative trackers.The evaluation uses 200-frame sequences, or 150 frames for PStudio, partitioned into 48-frame chunks with 8-frame overlap.Source: Table 1 · p. 7, §4.2 · pp. 7–8
  • On single-chunk sequences of up to 64 frames, Point4D performs comparably to other feed-forward methods and outperforms 2D-tracker-plus-reconstruction and iterative-refinement pipelines.The reported long-video advantage is attributed to reliable 3D-query chaining rather than stronger single-chunk decoding.Source: Table 2 · p. 8, §4.2 · p. 8

4.3 Ablations and Analysis

  • Across query variants, Point4D combines 3D coordinates with descriptors from arbitrary visible frames and outperforms 2D queries and 3D source-patch queries, especially on long videos.The 2D variant requires reprojection, while the source-patch variant expects a visible query and descriptor from the source frame.Source: §4.3 · pp. 8–9, Table 3 · p. 9
  • On PStudio, Point4D’s APD and EPE degrade more slowly across chunks, surpassing VDPM within a few chunks despite VDPM’s higher first-chunk accuracy.The reported gap widens as 2D-query methods compound reprojection error at chunk boundaries.Source: §4.3 · p. 9

5 Discussion

Point4D uses 3D coordinate queries and descriptors from arbitrary visible frames for long-range 4D reconstruction. Its limitations include evidence gaps during prolonged occlusion and accumulated depth or alignment error.

Discussion

  • Figure 6 shows Point4D’s tracking accuracy degrades more slowly across chunks than competing methods.The conclusion describes its design choices as enabling better performance on long-range videos.Source: Figure 6 · p. 9, §5 · p. 9
  • A point occluded or out of frame for an entire chunk lacks supporting evidence there, making its predicted position unreliable.The handoff retains only the query’s 3D coordinate and patch descriptor, not scene representation or feature memory.Source: §5 · p. 9
  • Errors in predicted depth can compound across chunks because the formulation relies on predicted depth and Sim(3) alignment between consecutive chunks.Source: §5 · p. 9

A Implementation Details

The implementation combines data sampling and augmentation with supervised training and separate evaluation protocols for long-video and single-chunk tracking.

Training Details

  • OmniWorld contributes dynamic content without trajectory ground truth, so its queries are restricted to same-time source and target frames.This reduces its task to depth and relative-pose estimation without cross-frame trajectory labels.Source: A Implementation Details · p. 13
  • Training samples 750 query pixels per frame, with 40% from Sobel-detected edges and 40% of queries using the target frame’s own coordinate system.The source, target, and camera frame indices are sampled uniformly, subject to the target-frame constraint.Source: A Implementation Details · p. 13
  • Training uses AdamW for 150 epochs on 8 H100 GPUs, with a 1 × 10^-4 peak learning rate, dynamic-point upweighting, and image augmentations.Models initialized from DepthAnything3 use a learning rate scaled by 0.1; the schedule warms up for 10 epochs before cosine decay.Source: A Implementation Details · p. 13

Evaluation Details

  • Long-video evaluation includes static and dynamic points with one global sequence scale, while single-chunk evaluation uses dynamic points and independently rescales each frame pair.Static points are included in long-video evaluation because they often leave the field of view during chaining.Source: A Implementation Details · p. 13

B Runtime Analysis

Runtime is measured while varying input frames or query points on one A6000. Point4D is fast at sparse queries, while its query-based runtime grows with query count.

Runtime Analysis

  • Runtime tests vary input frames from 16 to 64 with 100 queries, or query points from 100 to 5,000 with 48 frames.Measurements use a single A6000 with 48 GB memory at 294 × 518 input resolution.Source: B Runtime Analysis · p. 13
  • Point4D is the second-fastest method at every tested frame count, while its runtime grows with query count and SpatialTrackerV2 runs out of memory at 5,000 queries.Dense DPT-based feed-forward decoders predict every pixel regardless of the number of requested points; Point4D’s query-dependent cost makes sparse tracking cheap.Source: B Runtime Analysis · p. 13, Figure 7 · p. 14
  • With dense queries, Point4D falls behind dense DPT-head methods but completes, whereas SpatialTrackerV2 runs out of memory.The runtime comparison plots inference time against input frames and query points.Source: B Runtime Analysis · p. 13, Figure 7 · p. 14

C Video depth and camera pose estimation.

Point4D performs on par with DA3 and outperforms other feed-forward 4D reconstruction methods on most video-depth and camera-pose metrics.

C Video depth and camera pose estimation.

D Chaining Accuracy by Visibility at the Handoff Frame

The section compares handoff accuracy by visibility and finds that 3D re-querying is especially advantageous for occluded points, while the strongest 2D handoff strategy is Selection.

Chaining Accuracy by Visibility at the Handoff Frame

Choosing the handoff strategy for 2D-query baselines

3D vs. 2D re-querying across visibility regimes

E Additional Quantitative Results on Long-Video Tracking

Point4D remains competitive as sequences lengthen and chaining continues, including on dynamic points and extended videos.

Additional chunk-wise tracking accuracy analysis

4D tracking results on dynamic points

  • On dynamic points, Point4D outperforms every feed-forward method on all metrics and has the best average rank among compared methods.The evaluation uses 200-frame sequences with 48-frame chunks and 8-frame overlap; PStudio is omitted because its queries are dynamic-only.Source: E Additional Quantitative Results on… · p. 15, Table 8 · p. 16

4D tracking results on longer sequences

  • Point4D ranks first on every metric on 500-frame PointOdyssey and 300-frame Dynamic Replica sequences, surpassing feed-forward methods and iterative trackers.The evaluation uses 48-frame chunks with 8-frame overlap; the longer sequences require 12 handoffs on PointOdyssey and 7 on Dynamic Replica.Source: E Additional Quantitative Results on… · p. 15, Table 9 · p. 16

F Additional Qualitative Results on Long-Video Tracking

Additional results show consistent long-video trajectories and slower accuracy degradation across chained chunks.

F Additional Qualitative Results on Long-Video Tracking

Related papers

Loading 2609.09145v1…