PaperlayerView exact revision on arXiv

Source-linked AI summary

Amodal Depth Anything: Amodal Depth Estimation in the Wild

Zhenyu Li, Mykola Lavreniuk, Jian Shi, Shariq Farooq Bhat, Peter Wonka

arXiv:2412.02336v1cs.CV

TL;DR

Amodal depth estimation asks whether models can infer the geometry of occluded object regions, where real-world ground-truth depth is difficult to collect at scale. The paper introduces relative-depth estimation in the wild, ADIW, and two complementary frameworks; experiments report state-of-the-art ADIW performance and benefits from object-level supervision and guidance. The work also identifies dependence on the input amodal mask as a limitation, since inaccurate or ambiguous masks can propagate errors.

  • Problem

    Amodal depth estimation seeks depth values for occluded object parts, but scalable real-world ground-truth depth for those regions is unavailable.

    Source: §1 · p. 2

  • Method

    The paper builds ADIW from segmentation datasets and composited scenes with aligned depth maps, and proposes Amodal-DAV2 and Amodal-DepthFM for relative-depth prediction.

    Source: §1 · p. 2

  • Results

    Experiments report state-of-the-art amodal depth estimation on ADIW, with both frameworks benefiting from object-level supervision and guidance signals.

    Source: §4.4 · p. 7, §5 · p. 8

  • Takeaways & Limitations

    The models can generate diverse plausible occluded-region structures, and the work points to applications such as inpainting and scene reconstruction.

    Source: §5 · p. 8

    Dependence on the input amodal mask means inaccurate or ambiguous masks can propagate errors and cause cascading failures in amodal depth estimation.

    Source: C. Limitation and Future Work · p. 9

Abstract

from arXiv · show

Amodal depth estimation aims to predict the depth of occluded (invisible) parts of objects in a scene. This task addresses the question of whether models can effectively perceive the geometry of occluded regions based on visible cues. Prior methods primarily rely on synthetic datasets and focus on metric depth estimation, limiting their generalization to real-world settings due to domain shifts and scalability challenges. In this paper, we propose a novel formulation of amodal depth estimation in the wild, focusing on relative depth prediction to improve model generalization across diverse natural images. We introduce a new large-scale dataset, Amodal Depth In the Wild (ADIW), created using a scalable pipeline that leverages segmentation datasets and compositing techniques. Depth maps are generated using large pre-trained depth models, and a scale-and-shift alignment strategy is employed to refine and blend depth predictions, ensuring consistency in ground-truth annotations. To tackle the amodal depth task, we present two complementary frameworks: Amodal-DAV2, a deterministic model based on Depth Anything V2, and Amodal-DepthFM, a generative model that integrates conditional flow matching principles. Our proposed frameworks effectively leverage the capabilities of large pre-trained models with minimal modifications to achieve high-quality amodal depth predictions. Experiments validate our design choices, demonstrating the flexibility of our models in generating diverse, plausible depth structures for occluded regions. Our method achieves a 69.5% improvement in accuracy over the previous SoTA on the ADIW dataset.

1. Introduction

Amodal depth estimation asks whether models can infer the geometry of occluded object parts, a task that lacks scalable real-world ground-truth depth and is distinct from image inpainting. The paper introduces relative-depth estimation in the wild, the ADIW dataset, and deterministic and generative frameworks; experiments report state-of-the-art ADIW results and strong zero-shot performance on real-world images.

1. Introduction

  • Amodal depth estimation targets the depth of invisible object parts, but real-world ground-truth depth behind occlusions cannot be collected at scale.Unlike inpainting, the task infers depth values for occluded object regions given an input image and target amodal mask.Source: §1 · p. 2
  • Prior approaches rely on difficult-to-scale synthetic datasets and metric depth, which is associated with a domain gap and poor zero-shot performance on real-world images.Synthetic data lacks the complexity and diversity of real-world scenes.Source: §1 · p. 2
  • The paper formulates amodal depth estimation in the wild as relative-depth prediction, enabling real-world training without precise metric measurements.It leverages depth relationships within scenes to improve generalization.Source: §1 · p. 2
  • ADIW is a large-scale real-world dataset built from segmentation datasets and composited training pairs, with scale-and-shift alignment used to refine and blend depth predictions.Depth maps are estimated using a large depth foundation model to create consistent training ground truth.Source: §1 · p. 2
  • Amodal-DAV2 extends Depth Anything V2 with guidance layers, while Amodal-DepthFM adapts DepthFM using conditional flow matching; both build on large pre-trained models with minimal modifications.The frameworks target deterministic and generative model classes, respectively.Source: §1 · p. 2
  • Experiments report state-of-the-art amodal depth estimation on ADIW and strong zero-shot performance on real-world images.Ablations also support object-level supervision and guidance signals, while the models can generate diverse plausible structures for occluded regions.Source: §1 · p. 2

2. Related Work

Related work spans monocular depth estimation, amodal perception, and depth inpainting. The paper distinguishes its focus on directly estimating depth for occluded object regions from methods centered on visible pixels, synthetic-scene metric depth, or visual completion.

2.1. Monocular Depth Estimation

  • Monocular depth research has shifted from domain-specific metric depth toward cross-domain relative depth, but existing methods still focus on visible pixels rather than occluded regions.Examples include MiDaS, ZoeDepth, Depth Anything, Marigold, and DepthFM.Source: §2.1 · p. 3

2.2. Amodal Perception

  • Prior amodal depth methods use synthetic datasets for driving or indoor scenes and metric-depth estimation, while this work adopts relative depth with large pre-trained models and realistic synthetic data.The cited prior datasets are AmodalSynthDrive and Amodal-3D-FRONT.Source: §2.2 · p. 3

2.3. Depth Inpainting

  • Depth-inpainting methods fill missing depth or image regions for completion and view synthesis, whereas this work directly estimates object-level occluded depth independently of color information or visual coherence.The stated distinction prioritizes depth accuracy over realistic view synthesis.Source: §2.3 · p. 3

3. Method

The method formulates amodal depth estimation in the wild as relative-depth prediction and builds ADIW from composited natural images with aligned depth labels. It adapts pretrained deterministic and generative depth models using guidance for occluded regions.

3.1. Task Definition

  • Amodal depth estimation predicts relative depth for occluded object regions from an observation image, its depth map, and a target amodal mask.Source: §3.1 · p. 3

3.2. Dataset Collection

  • ADIW's pipeline builds whole-object segmentation data, composites sampled occluders into natural images, aligns relative-depth maps on visible background pixels, and yields 564K labeled images.Source: §3.2 · p. 4

3.3. Amodal Depth Estimator

  • The estimator adapts pretrained deterministic DAV2 and generative DepthFM models with added guidance while retaining their pretrained capabilities.Source: §3.3 · p. 4

3.3.1 Amodal-DAV2

  • Amodal-DAV2 adds a zero-initialized guidance convolution for observation depth and amodal mask, and trains with scale-invariant loss over the whole object.Source: §3.3.1 · p. 5

3.3.2 Amodal-DepthFM

  • Amodal-DepthFM fine-tunes conditional flow matching with image, observation-depth, and amodal-mask guidance, then aligns its decoded depth prediction with observed depth over shared visible regions.Source: §3.3.2 · p. 5

4. Experiments

The experiments assess amodal-depth predictions with metrics stratified by object visibility, compare methods on ADIW, and examine guidance, supervision, inference alignment, and prediction flexibility.

4.1. Metrics

  • Evaluation reports RMSE, log10 error, and accuracy under threshold δ on invisible object regions, separately for easy, medium, and hard visibility levels.The visible-ratio bands are (0.75, 1], (0.5, 0.75], and (0, 0.5], respectively; RMSE is scaled by 100 for illustration.Source: §4.1 · pp. 5–6

4.3. Main Results

  • On the ADIW validation set, Amodal-DAV2-L has the most consistent scale, accurate shape, and best-preserved details among the qualitative comparisons.Invisible Stitch and Jo et al. use metric-depth models, so their depth maps are shifted relative to methods using relative-depth estimators.Source: Figure 6 · p. 6

4.2. Implementation Details

  • Training uses approximately 559K samples and a 4K-sample validation set; both models start from depth-pretrained parameters and are fine-tuned for amodal depth.Amodal-DAV2 uses 50K iterations; Amodal-DepthFM uses 15K iterations.Source: §4.2 · p. 6

4.3. Main Results

  • 27.4% lower RMSE than the previous state of the art: Amodal-DAV2-L sets a new ADIW benchmark after prior methods are retrained and outputs aligned for comparison.Amodal-DepthFM also outperforms prior approaches without alignment.Source: Table 1 · p. 6, §4.3 · p. 7
  • Inpainting-based alternatives exhibit ghosting or cascade errors, while the proposed models directly predict occluded-region depth without RGB priors, using the amodal mask as guidance.The comparison includes Invisible Stitch and pix2gestalt.Source: §4.3 · p. 7

4.4. Ablation Studies and Discussion

  • Observation-depth and amodal-mask guidance are crucial, and both frameworks benefit from supervision that includes visible object regions.Inference alignment improves Amodal-DepthFM but decreases Amodal-DAV2 performance.Source: §4.4 · p. 7, Table 3 · p. 7
  • Different amodal masks produce corresponding plausible depth predictions; Amodal-DepthFM adds finer detail, sharper boundaries, and multiple plausible structures, while Amodal-DAV2 performs better overall.The generative model can produce multiple structures, unlike the deterministic Amodal-DAV2.Source: §4.4 · pp. 7–8, Figure 8 · p. 8

5. Conclusion

The conclusion presents ADIW and two complementary frameworks for estimating occluded-object depth in natural scenes. Amodal-DAV2 achieves state-of-the-art relative-depth performance, while Amodal-DepthFM produces finer details and sharper boundaries.

5. Conclusion

  • The work introduces ADIW, a large-scale dataset built with segmentation datasets and a compositing pipeline, alongside Amodal-DAV2 and Amodal-DepthFM.The dataset provides real-world amodal-depth annotations.Source: §5 · p. 8
  • Amodal-DAV2 achieves state-of-the-art relative-depth performance, whereas generative Amodal-DepthFM produces finer details and sharper boundaries in occluded regions.Amodal-DepthFM can synthesize multiple plausible structures; Amodal-DAV2 is deterministic.Source: §5 · p. 8, Figure 9 · p. 8
  • Both models benefit from object-level supervision and guidance signals; the paper identifies occluded-geometry understanding, inpainting, and scene reconstruction as future-research and application directions.Source: §5 · p. 8

A. Baseline

The baseline section adapts two inpainting-based methods as candidate solutions for amodal depth estimation.

A. Baseline

  • Invisible Stitch and pix2gestalt are adapted as potential inpainting-based solutions to amodal depth estimation.Source: A. Baseline · p. 9

A.1. Invisible Stitch

The Invisible Stitch adaptation uses image inpainting before estimating depth for the occluded region.

A.1. Invisible Stitch

  • The pipeline inpaints the target using SD-XL and a textual description, extracts an object mask with RMBG-1.4, then uses Invisible Stitch to estimate observation and amodal depth.The authors note that satisfactory inpainting requires an accurate textual description.Source: §A.1 · p. 9

A.2. pix2gestalt

The pix2gestalt pipeline estimates amodal depth by inpainting the target’s invisible parts, estimating depth for the original and inpainted images, and aligning the results.

A.2. pix2gestalt

  • The input amodal mask is preprocessed into a visible mask for the target object.Source: §A.2 · p. 9
  • pix2gestalt first inpaints the target object’s invisible parts, then Depth-Anything V2 ViT-G estimates depth for the original image and the inpainting result.Source: §A.2 · p. 9
  • The depth maps are aligned using shared visible regions to produce the final amodal depth prediction.Source: §A.2 · p. 9

B. Amodal 3D Reconstruction

Amodal depth supplies geometry for occluded regions that ordinary depth methods leave unrepresented, supporting reconstruction of more complete 3D meshes.

B. Amodal 3D Reconstruction

C. Limitation and Future Work

The authors identify input-mask dependence and reduced fine-detail capture as limitations, and propose video handling and a unified multimodal framework as future directions.

C. Limitation and Future Work

Related papers

Loading 2412.02336v1…