Source-linked AI summary
Dyna3: VLM-Guided Training-Free 4D Reconstruction via Depth Foundation Models
Xinhao Xiang, Weiyang Li, Zhijie Zheng, Abhijeet Rastogi, Jiawei Zhang
TL;DR
Depth foundation models such as DA3 are efficient but assume static scenes, while existing training-free 4D methods rely on correspondence-trained backbones. Dyna3 uses DA3 feature matching, VLM-guided instance segmentation, and decoupled static and dynamic reconstruction. It reports a +5.5 percentage-point J-Mean advantage over VGGT4D on DAVIS-2016 alongside efficiency gains.
Problem
Depth-only DA3 lacks the correspondence-based motion cues used by existing training-free 4D methods, leaving its use for dynamic-scene reconstruction an open question.
Source: §1 · p. 1Method
Dyna3 combines best-match search over DA3 features with VLM-guided instance segmentation and separate static-background and per-frame dynamic point-cloud reconstruction.
Source: §1 · p. 2, §3.3 · p. 3Results
Takeaways & Limitations
Dyna3’s efficiency gains allow denser temporal sampling than prior methods support.
Source: §1 · p. 2, §5 · p. 9Per-frame dynamic reconstruction lacks cross-frame correspondence, limiting temporal coherence, but avoids propagating dynamic-region matching errors into the static background.
Source: §4.4 · p. 7
Abstract
from arXiv · showhide
Recent depth foundation models like Depth Anything 3 (DA3) achieve remarkable multi-view depth estimation but assume static 3D scenes, limiting their applicability to real-world dynamic environments. Existing training-free 4D methods like Easi3R and VGGT4D rely on correspondence-trained backbones whose attention encodes cross-frame matching, a property absent in depth-only models like DA3. We present Dyna3, a training-free framework that extends DA3 for 4D dynamic scene reconstruction without any fine-tuning. Our key insight is that DA3's cross-view features, though trained only for depth consistency, implicitly encode motion-discriminative signals when combined with best-match feature search across frames. Its static surfaces find consistent matches globally, while dynamic objects cannot. We further adopt vision-language models (VLM) to automatically generate scene-specific semantic prompts for SAM 3, enabling precise instance-level segmentation that distinguishes which objects move from what objects exist. For reconstruction, we decouple the scene into a cross-frame aligned static background and per-frame dynamic point clouds. Experiments on four datasets demonstrate that Dyna3 surpasses correspondence-trained methods with +5.5pp J-Mean over state-of-the-art VGGT4D on dynamic object segmentation, while achieving up to 13x faster pose estimation and 3x faster 4D reconstruction with 4 to 8x lower memory. Dyna3 could therefore enable much denser temporal sampling that prior methods cannot support.
1. Introduction
Dyna3 addresses the gap between efficient depth-only models and training-free dynamic reconstruction methods that depend on correspondence-trained backbones. It combines best-match feature search with VLM-guided instance segmentation and separates static background reconstruction from per-frame dynamic points.
1. Introduction
- DA3’s depth-consistency training does not provide correspondence-based motion cues, so directly applying attention-based motion detection yields only ∼8% J-Mean.Without adaptation, moving objects are treated as static geometry, corrupting scene structure and camera trajectory.Source: §1 · p. 1
- Best-match feature search uses DA3 patch appearance features: static surfaces match across frames despite camera motion, while dynamic objects tend to score lower.Feature signals alone can confuse motion with texture variance, motivating semantic guidance.Source: §1 · pp. 1–2
- VLM-generated scene-specific prompts guide SAM3[3] to identify instances, while motion cues determine which objects move; reconstruction separates an aligned static background from per-frame dynamic point clouds.The pipeline requires no changes to DA3’s weights or architecture.Source: §1 · p. 2
- 65.04% J-Mean on DAVIS-2016 surpasses VGGT4D by +5.5 percentage points, while pose estimation runs up to 13× faster with 8× less memory.For final 4D reconstruction, Dyna3 achieves comparable quality with 3× faster inference and 4× less memory.Source: §1 · p. 2
- Dyna3 extends DA3 to training-free 4D reconstruction, addressing the gap between static-world assumptions and dynamic real-world scenes.The paper identifies avoiding both per-scene optimization and motion supervision as an open problem.Source: §1 · pp. 1–2
2. Related Work
Feed-forward multi-view reconstruction has advanced toward joint geometry and camera prediction, while DA3 offers a smaller depth-focused alternative without correspondence supervision. For dynamic scenes, prior methods either rely on expensive correspondence-trained backbones, fine-tune on dynamic data, or use per-scene optimization; Dyna3 targets efficient training-free reconstruction with DA3.
2.1. Feed-Forward Multi-View 3D Reconstruction
- DA3 unifies multi-view depth, camera rays, and pose in a feed-forward model with about 135M parameters, unlike larger correspondence-based models and iterative pipelines.The related-work discussion contrasts DA3 with DUSt3R [24] and VGGT [20], which use 571M–1.3B parameters.Source: §2.1 · p. 2
2.2. Dynamic and 4D Scene Reconstruction
- Dynamic reconstruction includes high-fidelity per-scene optimization methods, fine-tuned feed-forward models, and training-free methods that mine motion from correspondence-trained attention.MonST3R [38] fine-tunes DUSt3R; Easi3R [4] is limited to two-view inputs, while VGGT4D [8] supports multi-view reconstruction but remains expensive.Source: §2.2 · p. 3
- The related-work gap is that depth-only DA3 offers faster inference but lacks the attention-based motion cues available in correspondence-trained backbones.Dyna3 addresses this gap with best-match feature search and VLM-based instance segmentation.Source: §2.2 · p. 3
3. Method
Dyna3 uses pretrained DA3 depth, pose, and feature outputs to reconstruct monocular video as a persistent background, per-frame dynamic objects, and camera poses. It combines feature-based motion cues with VLM-guided instance segmentation, then separates static and dynamic content for reconstruction.
3.1. Overview
- Dyna3 processes monocular video at inference time through geometry extraction, motion scoring, VLM-guided segmentation, motion-aware object selection, and decoupled 4D reconstruction.It uses pretrained DA3 without fine-tuning or additional training.Source: §3.1 · p. 3
3.2. Geometry Extraction from DA3
- DA3 supplies per-frame depth, ray maps, and appearance-rich decoder features; RANSAC decomposes the ray maps into camera poses and intrinsics for lifting pixels into a unified world frame.The authors choose RANSAC decomposition over DA3’s camera-token head because they find it more reliable across diverse scenes.Source: §3.2 · p. 3
3.3. Motion Feature via Best-Match Search
- Best-match search compares each patch’s feature vector with locations across neighboring frames: static surfaces match despite camera motion, while dynamic objects receive lower similarity scores.The normalized score is thresholded and its largest connected component retained to form a coarse motion prior; appearance-based motion cues do not provide precise object boundaries.Source: §3.3 · pp. 3–4
3.4. VLM-Guided Instance Segmentation
- A VLM generates scene-specific noun phrases from a motion-region crop, and SAM3 uses those prompts to produce candidate instance masks across frames.Caching each clip’s video representation and reusing it for prompts yields a 4× speedup over per-prompt re-encoding with equivalent results.Source: §3.4 · p. 4
3.5. Motion-Aware Dynamic Instance Selection
- Motion-aware selection resolves cases where semantic categories include both moving and stationary instances by combining pixel motion scores with precise instance boundaries.It averages motion values within each candidate mask, selects instances above a threshold, and chooses the highest-scoring instance if none exceed it.Source: §3.5 · pp. 4–5
3.6. Decoupled 4D Scene Reconstruction
- Dyna3 aggregates non-dynamic pixels across frames into a voxel-consolidated static point cloud, while retaining each frame’s dynamic content as a separate point cloud.Mask dilation provides a safety margin against boundary leakage and ghost artifacts; multi-view aggregation improves coverage, and camera poses complete the reconstruction.Source: §3.6 · p. 5
4. Experiment
The experiments evaluate Dyna3 on dynamic-object segmentation, camera-pose estimation, and 4D reconstruction, alongside component ablations and an analysis of DA3 attention. Results show strong segmentation and competitive pose and reconstruction quality, with efficiency gains that support denser temporal sampling.
4.1. Experimental Setup
- The evaluation covers segmentation, pose estimation, and 4D reconstruction, comparing Dyna3 with training-free and fine-tuned baselines across the corresponding datasets.The experiments use DAVIS-2016/2017, TUM-dynamics, Sintel, and DyCheck; Dyna3 uses DA3-BASE, Qwen2-VL-2B, and SAM 3 on one NVIDIA A100 GPU.Source: §4.1 · p. 5, §4 · p. 5
4.2. Dynamic Object Segmentation
- 65.04% J-Mean on DAVIS-2016 beats VGGT4D by +5.5 percentage points, and Dyna3 leads by +2.3pp J-Mean on DAVIS-2017.On DAVIS-2016, F-Mean improves by +11.5pp over VGGT4D; on DAVIS-2017, F-Mean improves by +11.4pp.Source: Table 1 · p. 6, §4.2 · p. 6
4.3. Camera Pose Estimation
- On Sintel, Dyna3 achieves the best relative translation error, while running 8.6× faster and using 4.4× less memory than VGGT4D.On TUM-dynamics it ranks second in RTE and RRE; denser sampling improves relative pose accuracy, and at stride 3 VGGT4D runs out of memory while Dyna3 completes.Source: Table 3 · p. 6, §4.3 · p. 6, Table 4 · p. 7
4.4. 4D Reconstruction
- On DyCheck, Dyna3 ranks second on Accuracy and Completeness, while processing videos in 0.70 seconds with 3.74GB memory.The decoupled per-frame dynamic representation trades some dynamic reconstruction quality for robustness and efficiency; errors in dynamic regions do not propagate to the static background.Source: §4.4 · p. 7, Table 6 · p. 8
4.5. Ablation Studies
- On DAVIS-2016, progressive additions raise J-Mean from 8.16% for the DA3 baseline to 65.04% for full Dyna3.Motion-prior extraction reaches 29.65%, fine instance selection 33.51%, and VLM prompting without SAM3 53.35%; SAM3 refinement yields 65.04% J-Mean and 66.93% F-Mean.Source: §4.5 · p. 8
4.6. Analysis: How DA3’s attention layers does not encode motion feature
- DA3 attention maps are uniformly distributed rather than aligned with dynamic regions, while explicit best-match search yields 29.65% J-Mean versus about 8% for attention-based detection.Static surfaces find consistent matches across frames despite camera motion, whereas dynamic objects have lower match scores.Source: Figure 3 · p. 8, §4.6 · p. 9
5. Conclusion
Dyna3 extends Depth Anything 3 to training-free 4D dynamic scene reconstruction. It achieves 65.04% J-Mean on DAVIS-2016, with faster inference and lower memory use than VGGT4D.
5. Conclusion
- Dyna3 extends Depth Anything 3 to training-free 4D dynamic scene reconstruction, combining best-match feature search with VLM-guided instance segmentation.It achieves 65.04% J-Mean on DAVIS-2016, outperforming VGGT4D by +5.5 percentage points while running up to 13× faster with 8× less memory.Source: §5 · p. 9
A. VLM Prompt Details
The VLM receives a motion-prior crop from the representative middle frame and generates a scene-specific text prompt for SAM 3 video segmentation.
A. VLM Prompt Details
- Qwen2-VL-7B receives a crop of the motion-prior bounding box from the sequence’s middle frame.The VLM is asked to identify the main moving foreground subject using one to three specific noun phrases.Source: A. VLM Prompt Details · p. 12
- The prompt requests specific descriptions of the moving foreground subject rather than generic labels.It asks for one to three noun phrases and prohibits generic terms such as “person” or “object.”Source: A. VLM Prompt Details · p. 12
- The VLM output is used directly as the text prompt for SAM 3 video segmentation.Source: A. VLM Prompt Details · p. 12
B. Progressive Dynamic Mask Refinement
Dyna3 progressively combines best-match motion scores with SAM3 instance regions, then selects instances by motion score to produce dynamic masks. The refinement reduces textured-background noise, while small or distant objects remain challenging.
B. Progressive Dynamic Mask Refinement
- Best-match cosine-similarity search across temporal neighbors yields low motion-map scores for static surfaces and high scores for dynamic objects, but textured areas add background noise.The search compares each pixel’s DA3 patch feature with locations in neighboring frames.Source: B. Progressive Dynamic Mask Refinement · p. 12
- Masking motion scores to SAM3 instance regions removes background noise, and selecting instances above a motion-score threshold separates moving objects from static instances of the same category.The instance score is the mean motion score within that instance’s region.Source: B. Progressive Dynamic Mask Refinement · p. 12
- Camel, dog, and blackswan sequences score 97.4%, 93.6%, and 90.4%, respectively, while kite-surf scores 35.1% on small or distant objects.The text attributes the weaker kite-surf result to less reliable motion detection and instance segmentation on small or distant objects.Source: B. Progressive Dynamic Mask Refinement · p. 12
C. Why Attention-Based Motion Detection Fails on DA3
Attention-based motion detection relies on correspondence-trained attention, which DA3 does not have because it is trained for depth consistency. Dyna3 instead uses explicit feature comparison to construct motion signals.
C. Why Attention-Based Motion Detection Fails on DA3
- DUSt3R and VGGT use correspondence supervision that makes attention reflect cross-frame matching, enabling Easi3R [4] and VGGT4D [8] to detect when dynamic objects disrupt those correspondences.DUSt3R’s tracking loss trains attention to focus on corresponding locations; VGGT’s tracking-supervised attention represents cross-frame correspondences.Source: C. Why Attention-Based Motion… · pp. 13–14
- DA3’s depth-consistency losses do not supervise attention to focus on correspondences, so its attention encodes depth relationships and produces responses unrelated to dynamic content.Applying Easi3R’s formulation to DA3 yields uniformly distributed responses across the scene.Source: Figure 5 · p. 13, C. Why Attention-Based Motion… · p. 14
- Dyna3 avoids attention analysis by explicitly comparing DA3 patch features to construct motion signals without correspondence-trained representations.The passage identifies best-match search as the approach motivated by DA3’s attention failure.Source: C. Why Attention-Based Motion… · p. 14
D. Additional Implementation Details
The implementation uses four temporal neighbors, a 0.3 dynamic-score threshold, and 504-pixel processing resolution.
D. Additional Implementation Details
- SAM3 session reuse caches video features across text prompts, yielding a 4× speedup over per-prompt re-encoding.Motion scores use |Ni| = 4 frames, instance selection uses θ = 0.3, and the longer image side is processed at 504 pixels.Source: D. Additional Implementation Details · p. 14