PaperlayerView exact revision on arXiv

Source-linked AI summary

Depth Anything 3: Recovering the Visual Space from Any Views

Haotong Lin, Sili Chen, Junhao Liew, Donny Y. Chen, Zhenyu Li, Guang Shi, Jiashi Feng, Bingyi Kang

arXiv:2511.10647v1cs.CV

TL;DR

Existing 3D vision systems often specialize architectures and tasks, while DA3 seeks consistent geometry from arbitrary visual inputs with or without camera poses. It uses a plain transformer with depth-ray targets and teacher-student supervision, then evaluates geometry, pose, rendering, and monocular depth. DA3 sets state-of-the-art results across the proposed tasks and also supports efficient novel-view synthesis.

  • Problem

    Prior 3D vision models commonly use specialized architectures and task-specific optimization, motivating a unified approach to geometry from arbitrary visual inputs.

  • Method

    DA3 uses a single pretrained transformer, depth-and-ray prediction, optional camera conditioning, and teacher-student supervision to reconstruct geometry from any number of images.

  • Results

    DA3 sets new state-of-the-art performance across camera pose, any-view geometry, and visual rendering tasks, with 25.1% geometric improvement over VGGT and 21.5% over Pi3.

  • Takeaways & Limitations

    A minimal geometry foundation model can unify arbitrary-view reconstruction and serve as a strong backbone for feed-forward novel-view synthesis.

  • Takeaways & Limitations

    Future work must extend DA3 toward dynamic scenes, language and interaction cues, and larger-scale pretraining.

Abstract

from arXiv · show

We present Depth Anything 3 (DA3), a model that predicts spatially consistent geometry from an arbitrary number of visual inputs, with or without known camera poses. In pursuit of minimal modeling, DA3 yields two key insights: a single plain transformer (e.g., vanilla DINO encoder) is sufficient as a backbone without architectural specialization, and a singular depth-ray prediction target obviates the need for complex multi-task learning. Through our teacher-student training paradigm, the model achieves a level of detail and generalization on par with Depth Anything 2 (DA2). We establish a new visual geometry benchmark covering camera pose estimation, any-view geometry and visual rendering. On this benchmark, DA3 sets a new state-of-the-art across all tasks, surpassing prior SOTA VGGT by an average of 44.3% in camera pose accuracy and 25.1% in geometric accuracy. Moreover, it outperforms DA2 in monocular depth estimation. All models are trained exclusively on public academic datasets.

1 Introduction

Depth Anything 3 targets consistent 3D reconstruction from arbitrary visual inputs while minimizing architectural and prediction-target complexity. It combines a plain transformer, depth-ray representation, teacher-student training, and broad benchmarks to unify geometry estimation and visual rendering.

  • 3D vision tasks share strong conceptual overlap, but prior systems commonly use specialized architectures and joint task optimization that limit pretrained-model reuse.
  • DA3 asks whether a single plain transformer and minimal prediction targets can recover 3D structure from arbitrary visual inputs.
  • DA3 reconstructs visual space from any number of images, with or without known camera poses, using joint any-view depth and pose estimation through a ray representation.
  • The model uses teacher-student training to unify real-world depth captures, 3D reconstructions, and synthetic data despite noisy or incomplete real-world supervision.
  • DA3 establishes visual-geometry and feed-forward novel-view-synthesis benchmarks, reporting state-of-the-art performance across 18 of 20 geometry and pose settings and over 160 NVS scenes.

2 Related Work

Related work spans modular and learned multi-view geometry, monocular depth estimation, and feed-forward novel-view synthesis. DA3 is presented against efforts to replace specialized pipelines with scalable transformer-based and pretrained geometry models.

  • Multi-view visual geometry estimation: Traditional multi-view geometry decomposes reconstruction into matching, pose estimation, SfM, and dense stereo, remaining strong on textured scenes but brittle under difficult visual conditions.
  • Depth Anything 3: DA3’s pipeline uses a single transformer, adaptive cross-view attention, a Dual-DPT depth-and-ray head, and optional camera-token conditioning.
  • Multi-view visual geometry estimation: Learned component and feed-forward approaches reduce engineering complexity but often struggle with scalability, generalization, or arbitrary input cardinalities.
  • Multi-view visual geometry estimation: DUSt3R established transformer-based feed-forward multi-view geometry by directly predicting point maps and relative pose, motivating extensions to more views, video, camera conditioning, and view synthesis.
  • Monocular depth estimation: Monocular depth research moved from domain-specific supervised models toward generalist systems using massive multi-dataset training and vision transformers.
  • Feed-Forward Novel View Synthesis: Feed-forward novel-view synthesis avoids per-scene optimization by predicting 3D representations in one pass, with recent work favoring explicit 3D Gaussian representations.

3 Depth Anything 3

DA3 formulates arbitrary-view geometry as dense depth-and-ray prediction using a minimal transformer-centered architecture. Its design includes implicit pose representation, optional camera conditioning, shared depth-ray processing, teacher-generated supervision, and normalized multi-term training.

  • 3.1 Formulation: DA3 handles single images, multi-view collections, and videos, optionally incorporating known camera poses.
  • 3.1 Formulation: Each input image is paired with a depth map, camera extrinsics, and related geometric quantities for dense visual-space reconstruction.
  • 3.1 Formulation: A per-pixel ray stores camera origin and world-frame direction, providing an implicit pose representation aligned with the input image and depth map.
  • 3.1 Formulation: Depth-ray targets provide a minimal sufficient representation for scene structure and camera motion, while a lightweight camera head avoids costly pose recovery from ray maps.
  • 3.2 Architecture: The architecture combines a single transformer backbone, optional camera encoder, and Dual-DPT head; adaptive attention supports cross-view reasoning without architectural changes.
  • 3.2 Architecture: The Dual-DPT head shares reassembly modules before separate depth and ray fusion branches, encouraging interaction while avoiding redundant intermediate representations.
  • 3.3 Training: Teacher-student training uses synthetic-data pseudo-depth aligned to noisy or sparse real-world measurements through RANSAC least squares.
  • 3.3 Training: The training objective normalizes ground-truth signals and combines depth, ray, point-map, camera, and gradient losses with α = 1 and β = 1.

4 Teacher-Student Learning

Depth Anything 3 builds teacher supervision from synthetic-data relative depth and noisy real-world measurements, extending depth representation and geometric training objectives. The aligned supervision supports joint depth–ray learning and downstream monocular and metric depth models.

  • 4.1 Constructing the Teacher Model: Synthetic-only teacher training supplies supervision for poor-quality real-world datasets with noisy or incomplete depth.The teacher generates high-quality pseudo-depth, while noisy measurements provide scale and shift for alignment.
  • 4.1 Constructing the Teacher Model: DA3 expands the synthetic training corpus to improve depth estimation and recover finer geometric detail.The expanded corpus includes diverse synthetic datasets beyond those used in DA2.
  • 4.1 Constructing the Teacher Model: Scale–shift-invariant depth replaces DA2’s disparity target, with exponential depth improving discrimination in near-camera regions.Depth is selected for downstream metric-depth and multiview-geometry tasks that operate directly in depth space.
  • 4.1 Constructing the Teacher Model: Distance-weighted surface-normal supervision refines local geometry by downweighting neighbors farther from each center pixel.The objective also includes depth-gradient and global–local losses.
  • 4.1 Constructing the Teacher Model: Joint sky and object masks prevent regions without defined ground truth from degrading depth prediction and support downstream use.The masks are aligned with the depth output and supervised with MSE loss.
  • 4.2 Teaching Depth Anything 3: RANSAC scale–shift alignment converts teacher relative depth and sparse metric measurements into scale-consistent, pose–depth coherent supervision.The aligned supervision complements joint depth–ray objectives and improves real-world generalization.
  • 4.3 Teaching Monocular Model: The monocular student predicts relative depth from teacher-generated pseudo-labels and achieves state-of-the-art performance on standard monocular depth evaluation.Unlike DA2, the student predicts depth maps rather than disparity.
  • 4.4 Teaching Metric Model: Metric-depth training aligns teacher predictions with ground-truth metric depth after canonical camera-space transformation for varying focal lengths.Teacher predictions are used as labels to preserve sharp boundaries.

5 Application: Feed-Forward 3D Gaussian Splattings

The paper extends DA3 to feed-forward 3D Gaussian splatting by fine-tuning a DPT head that predicts pixel-aligned Gaussian primitives from visual tokens. Pose-adaptive design, rendering-oriented objectives, and varied training conditions support synthesis with or without camera poses.

  • 5 Application: Feed-Forward 3D Gaussian Splattings: FF-NVS fine-tunes DA3 with an added DPT head to infer pixel-aligned 3D Gaussians from visual inputs.The application demonstrates the paper’s minimal modeling strategy for novel-view synthesis.
  • 5.1 Pose-Conditioned Feed-Forward 3DGS: GS-DPT predicts opacity, rotation quaternion, scale, and RGB color for each camera-space Gaussian.Depth is unprojected into world coordinates to obtain Gaussian positions, which are rasterized into novel views.
  • 5.1 Pose-Conditioned Feed-Forward 3DGS: The NVS model combines rendered-view photometric losses with scale-shift-invariant depth loss on observed views.This follows the teacher–student learning paradigm.
  • 5.2 In-the-Wild Feed-Forward 3DGS: A pose-adaptive version accepts posed and unposed inputs and supports varying resolutions and input view counts with identical pretrained weights.The design predicts 3DGS parameters in local camera space and integrates with the DA3 backbone.
  • 5.2 In-the-Wild Feed-Forward 3DGS: An additional depth offset reduces the trade-off between surface geometry and rendering quality, while spherical harmonics model view-dependent color.These changes target greater robustness for in-the-wild use.
  • 5.2 In-the-Wild Feed-Forward 3DGS: Training freezes the pretrained DA3 backbone and varies image resolution and context-view count to stabilize training across evaluation scenarios.Higher-resolution inputs use fewer context views, while lower-resolution inputs use more.
  • 5.2 In-the-Wild Feed-Forward 3DGS: The NVS model trains on 10,015 DL3DV scenes, with 140 benchmark scenes kept disjoint from training to prevent data leakage.DL3DV supplies diverse real-world scenes with COLMAP-estimated camera poses.

6 Visual Geometry Benchmark

The benchmark evaluates pose, reconstructed geometry, and visual rendering across diverse scenes and datasets. Its protocols generate consistent pose-depth estimates, align and fuse reconstructions, and assess novel-view rendering against ground truth.

  • The benchmark directly evaluates pose accuracy, reconstruction accuracy from predicted pose and depth, and visual rendering quality.
  • Pose estimation: Pose evaluation processes selected images with a feed-forward model, then computes accuracy from the resulting consistent pose and depth estimates.When scenes contain more than the image limit, 100 images are randomly sampled with a fixed seed.
  • Geometry estimation: Geometry evaluation aligns predicted poses to ground truth, applies RANSAC-based transformation selection, fuses point clouds with TSDF, and compares the reconstruction with ground truth.The selected transformation maximizes the inlier pose set, with inliers defined by translation errors below the median pose deviation.
  • Metrics: Pose accuracy is reported with AUC derived from relative rotation and translation accuracy, primarily at thresholds 3 and 30.AUC integrates the accuracy-threshold curve determined by the smaller of the rotation and translation accuracies.
  • Metrics: Reconstruction quality uses accuracy, completeness, Chamfer Distance, precision, recall, and F1-score based on distances between reconstructed and ground-truth point sets.
  • Visual rendering: The visual-rendering benchmark uses three datasets, COLMAP ground-truth poses, 12 context views selected by farthest-point sampling, and novel views sampled every eight frames.It reports PSNR, SSIM, and LPIPS on rendered novel views.

7 Experiments

Experiments compare DA3 with feed-forward geometry and rendering baselines, then test its representation and architecture through ablations. DA3 achieves strong pose, reconstruction, monocular-depth, and rendering results, while depth plus ray and a single pretrained transformer are supported by the ablations.

  • Comparison with State of the Art: DA3 is compared with VGGT, Pi3, MapAnything, Fast3R, and specialized feed-forward 3DGS models under benchmark and unified training protocols.
  • Comparison with State of the Art: DA3-Giant attains the best pose-estimation performance on nearly all metrics, with at least an 8% relative improvement over competitors on Auc3.On ScanNet++, it achieves a 33% relative gain over the second-best model.
  • Comparison with State of the Art: 25.1% relative improvement over VGGT and 21.5% over Pi3 is achieved by DA3-Giant in reconstruction geometry.It outperforms all competitors in the five pose-free settings and establishes new state-of-the-art performance in nearly all scenarios.
  • Comparison with State of the Art: With pose conditioning, DA3 improves on most datasets, but gains from model scaling are smaller than in pose-free models.The reported exception is 7Scenes, where the limited video setting already saturates performance.
  • Comparison with State of the Art: DA3 outperforms VGGT and Depth Anything 2 on standard monocular-depth benchmarks.
  • Visual rendering: Geometry-model-based rendering frameworks consistently outperform specialized feed-forward models, while performance is substantially better on DL3DV than on the other datasets.The comparison uses PSNR, SSIM, and LPIPS under a unified protocol with 12 input context views.
  • Sufficiency of the Depth-Ray Representation: The minimal depth + ray configuration consistently outperforms depth + pcd + cam and depth + cam, achieving nearly 100% relative gain in Auc3 over depth + cam.Adding an auxiliary camera head provides no further benefit.
  • Sufficiency of a Single Plain Transformer: A VGGT-style architecture reaches 79.8% of baseline performance at similar scale, while full attention alternation degrades nearly all metrics.The comparison supports the single-transformer design and partial alternation strategy.

7.3 Analysis for Depth-Anything-3-Monocular

The monocular analysis studies teacher supervision and the resulting student model across standard depth benchmarks. The new teacher generally exceeds DA2, and the monocular student surpasses the DA2 student, including a gain above 10% on ETH3D.

  • Teacher supervision: Teacher-label supervision yields depth maps with substantially richer detail and finer structures.
  • Teacher model: Training with V3 datasets and multi-resolution yields the best teacher performance, while depth-based geometry gives the best AbsRel and SqRel.The full teacher loss outperforms the other evaluated variants.
  • Teacher model: The new teacher consistently outperforms DA2 across datasets except NYU, where performance is on par with DA2.
  • Monocular student: The monocular student with a ViT-L backbone outperforms the DA2 student across all evaluation datasets.On ETH3D, the improvement over DA2 exceeds 10%; on SINTEL, the gain is +5.1%.

7.4 Analysis for Depth-Anything-3-Metric

DA3-metric performs strongly across metric-depth benchmarks, achieving leading results on ETH3D and SUN-RGBD while remaining competitive elsewhere. Teacher supervision improves visual sharpness and fine-detail quality despite mixed standard-metric effects.

  • DA3-metric achieves state-of-the-art performance on ETH3D, with δ1 = 0.917 and AbsRel = 0.104.
  • DA3-metric outperforms UniDepthv2 on ETH3D δ1 by 0.054, achieving 0.917 versus 0.863.
  • DA3-metric achieves best AbsRel on SUN-RGBD and second-best performance on DIODE, with δ1 = 0.838 and AbsRel = 0.128.
  • DA3-metric demonstrates strong generalization across all benchmarks, particularly on diverse outdoor scenes such as ETH3D.
  • Teacher supervision significantly improves metric-depth sharpness and fine-detail quality, while its removal slightly improves metrics on NYUv2 and KITTI.
  • The comparison reports results at 270 × 480 resolution after retraining feed-forward 3DGS baselines with matched testing configurations.
  • Adding a 3D Gaussian DPT head improves rendering quality, especially for thin structures and wide-baseline outdoor scenes.

8 Conclusion and Discussion

DA3 demonstrates that a plain transformer with depth-and-ray targets and teacher–student supervision can unify any-view geometry without ornate architectures. Its geometry backbone also supports efficient feed-forward novel view synthesis, while future work targets broader world-model capabilities.

  • A plain transformer trained on depth-and-ray targets with teacher–student supervision unifies any-view geometry without ornate architectures.
  • DA3 sets new pose and reconstruction records on the proposed visual geometry benchmark, with both giant and compact variants surpassing prior models.
  • Future work includes dynamic-scene reasoning, language and interaction cues, and larger-scale pretraining toward actionable world models.

Appendix

The appendix notes that synthetic datasets provide large-scale depth supervision but contain quality problems that require preprocessing before teacher-model training.

  • Synthetic datasets offer large-scale ground-truth depth annotations but may contain invalid backgrounds, spatial misalignments, clipping artifacts, and erroneous depth values.
  • The preprocessing filters problematic samples and clips unrealistic depth ranges to provide higher-quality supervision for the teacher model.

A Data Processing

The data-processing pipeline removes corrupted samples and clips implausible depth values across training datasets. These steps target invalid backgrounds, image-depth misalignment, missing depth, clipping, incorrect ground depth, and textureless samples.

  • TartanAir: TartanAir preprocessing removes the amusement scene for its invalid skybox background and clips maximum depths for selected scenes.
  • IRS: IRS preprocessing filters samples with spatial misalignment between images and depth maps using boundary-overlap analysis.
  • UnrealStereo4K: UnrealStereo4K preprocessing removes scenes with erroneous regions or clipping and samples where the sea lacks depth values.
  • Depth-range clipping: GTA-SfM and Kenburns preprocessing clips maximum or extreme depth values at 1000 and 50,000, respectively.
  • Sample removal: PointOdyssey removes scenes with incorrect ground depth, while TRELLIS removes samples lacking texture.
  • Depth-range clipping: OmniObject3D preprocessing clips the maximum depth value at 10.
Loading 2511.10647v1…