PaperlayerView exact revision on arXiv

Source-linked AI summary

OpenDepth: Repurposing Image Editing Priors for Robust Open-World Monocular Depth Estimation

Mengke Zhang, Tianyi Yang, Xinzhuo Li, Anlong Ming

arXiv:2610.17610v2cs.AI

TL;DR

Open-world depth estimation remains challenging under weather and illumination shifts, where T2I priors can produce structural hallucinations. OpenDepth uses I2I editing priors with shallow feature alignment and joint RGB latent reconstruction, and reports robust zero-shot SOTA across standard and degraded benchmarks.

  • Problem

    T2I-based depth models can suffer structural hallucinations under open-world perturbations, while severe shifts challenge geometric estimation.

    Source: §1 · pp. 1–2

  • Method

    OpenDepth uses I2I editing priors, aligning shallow features and jointly reconstructing clean RGB latents to map degraded inputs into a unified latent domain.

    Source: §3.3 · p. 4, §3.4 · p. 5

  • Results

    OpenDepth reports new SOTA records for robust zero-shot depth estimation across standard and degraded benchmarks, with exceptional detail preservation using 79K training samples.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The reported evaluations support I2I editing priors as a robust foundation for geometric estimation across environmental degradations.

    Source: §1 · p. 2, §5 · p. 8

Abstract

from arXiv · show

Recently, the integration of generative priors has revolutionized Monocular Depth Estimation (MDE), achieving state-of-the-art (SOTA) performance with remarkably high data efficiency. However, although methods based on Text-to-Image (T2I) models excel in general scenes, their efficacy degrades sharply when confronted with open-world environmental perturbations, such as illumination variations and adverse weather. Since T2I priors are optimized for semantic synthesis rather than preserving pixel-level structure, they struggle to decouple geometry from appearance under severe visual shifts, easily triggering structural hallucinations. Therefore, our core insight is to model open-world depth estimation as consistent geometric inference across multi-style images, fully leveraging the powerful structure-preserving editing priors of Image-to-Image (I2I) models. Based on this, we propose OpenDepth, a robust open-world MDE framework that introduces "Intra-Domain Awareness" -- the capability to map diverse stylistic inputs sharing identical geometry into a unified latent space. To achieve this, we first align shallow features within the Diffusion Transformer (DiT) to anchor degraded observations to unified geometric representations. Additionally, we introduce a joint RGB latent reconstruction task, requiring the model to recover clean RGB information from degraded inputs. This multi-stage strategy effectively decouples environmental style interference, compelling the model to focus strictly on underlying geometric recovery. Extensive evaluations show that OpenDepth sets new SOTA records for robust zero-shot depth estimation across standard and extreme weather benchmarks, demonstrating exceptional structural integrity and adaptability.

1 Introduction

OpenDepth reframes robust depth estimation as geometric inference across visual styles, using I2I editing priors and latent alignment to address perturbation-related structural failures. It reports new zero-shot SOTA records across standard and degraded benchmarks with 79K training samples.

1 Introduction

  • In a matched-data comparison using 59K samples, the I2I baseline preserves scene geometry under perturbations better than Lotus-D, which exhibits structural failures.The comparison uses the same Hypersim and Virtual KITTI samples for both models.Source: Figure 1 · p. 1, §1 · p. 2
  • OpenDepth frames open-world depth estimation as unified geometric inference across visual styles, requiring diverse observations to map into a clean latent domain before geometric reasoning.The paper calls this capability “Intra-Domain Awareness.”Source: §1 · p. 2
  • Early Single Stream feature alignment and joint RGB latent reconstruction map perturbed inputs toward a unified latent domain for high-fidelity geometry recovery.The reconstruction task uses diverse conditional inputs to recover unperturbed RGB information.Source: §1 · p. 2
  • 79K training samples yield new SOTA records for robust zero-shot depth estimation across standard and degraded benchmarks, with exceptional detail preservation.The reported scope includes complex open-world scenes.Source: §1 · p. 2

2 Related Work

Prior MDE work has advanced through discriminative and generative models, while I2I estimators offer structural fidelity but remain vulnerable to severe input shifts. OpenDepth addresses that gap with latent alignment and unified RGB reconstruction for robust open-world perception.

2.1 Foundation MDE Model

  • I2I estimators such as FE2E and Edit2Perceive have shown stronger structural fidelity than T2I counterparts, while OpenDepth targets their remaining feature-misalignment gap under severe shifts.The related-work overview traces MDE from discriminative methods to generative priors and image-editing models.Source: §2.1 · p. 2

2.2 Robust Monocular Depth Estimation

  • Robustness research spans adaptation, self-supervision, and foundation-model approaches, while current open-world models such as DepthAnything-AC rely mainly on output consistency and spatial-distance constraints.The cited approaches include nighttime enhancement, frequency-aware regularization, contrastive learning, and parameter-efficient adaptation.Source: §2.2 · pp. 2–3
  • OpenDepth combines I2I editing priors with latent alignment and unified RGB reconstruction to project degraded inputs into a shared, style-invariant latent domain.The framework overview describes these as dual constraints for decoupling environmental style from geometric structure.Source: §2.2 · p. 3, Figure 2 · p. 3

3 Methodology

OpenDepth builds robust monocular depth estimation around I2I editing priors, using latent feature alignment and RGB reconstruction to unify geometry across environmental styles.

3.1 Preliminaries: Multi-Modal Diffusion Transformer

  • MMDiT processes joint RGB and depth latents with text through double-stream and single-stream blocks, then outputs predicted depth and reconstructed RGB latents.The RGB image and target depth are VAE-encoded, patchified, and concatenated as spatial input.Source: §3.1 · p. 3

3.2 Advantages of I2I Editing Priors: A Motivational Analysis

  • I2I editing priors preserve source geometry under perturbations, whereas T2I priors can hallucinate structure; I2I features show higher, more stable clean-to-perturbed similarity across layers.The comparison evaluates Lotus-D and an I2I baseline without robustness fine-tuning.Source: §3.2 · p. 4, Figure 3 · p. 4

3.3 Latent Feature Alignment

  • To counter feature drift after cross-modal fusion, OpenDepth aligns image tokens in early Single Stream blocks against clean features, which remain fixed through stop-gradient.In practice, L=4; alignment improves feature similarity throughout the network, especially in deeper layers.Source: §3.3 · p. 4

3.4 Unified RGB Reconstruction in Latent Space

  • Unified RGB reconstruction supervises the perturbed branch’s reconstructed RGB latent against the clean latent, with stop-gradient keeping the clean target fixed.Across 20 scenes and six perturbations, reconstruction aggregates deep-feature clusters that otherwise scatter across styles.Source: Figure 4 · p. 4, §3.4 · p. 5

3.5 Overall Objective

  • Training weights clean and perturbed depth losses, feature alignment, and cross-domain RGB reconstruction; each batch pairs clean and randomly perturbed images with shared ground-truth depth.The loss weights balance perturbation robustness, feature consistency, and latent reconstruction.Source: §3.5 · p. 5

4 Experiments

OpenDepth is evaluated zero-shot on standard and adverse-condition depth benchmarks, where it reports leading results across the tested settings. Ablations examine the discriminative formulation, perturbation training, feature alignment, and RGB reconstruction.

4.1 Experiment Setup

  • Training combines Hypersim, Virtual KITTI, and Cityscapes with compounded perturbations, and evaluation spans adverse-condition, synthetic-corruption, and standard benchmarks.The experiment setup also specifies the optimizer, training duration, and evaluation metrics and protocol.Source: §4.1 · p. 5

4.2 Zero-Shot Relative Depth Estimation on General Benchmarks

  • OpenDepth achieves a top average rank of 1.1 on general benchmarks, using 79K labeled samples.The authors report that I2I-based FE2E and OpenDepth consistently surpass Stable Diffusion-based T2I models.Source: §4.2 · pp. 5–6, Table 1 · p. 6

4.3 Zero-Shot Relative Depth Estimation on Open-World Benchmarks

  • OpenDepth reports state-of-the-art results across real-world adverse benchmarks, with qualitative comparisons showing stronger detail preservation and structural consistency.The comparison covers general and open-world conditions; Table 2 reports open-world benchmark results.Source: §4.3 · p. 6, Table 2 · p. 6, Figure 5 · p. 7
  • OpenDepth achieves the best performance in three of four KITTI-C corruption categories: Dark, Motion, and Gaussian noise.Table 3 reports zero-shot results on synthetic KITTI-C benchmarks.Source: §4.3 · p. 6, Table 3 · p. 7

4.4 Ablation Study

  • The discriminative formulation outperforms the generative formulation across ablation benchmarks; perturbation training reduces NuScenes-night AbsRel from 18.0 to 16.6.The main comparison tables use the discriminative formulation; applying the same perturbation strategy to Lotus-D improves it only marginally.Source: Table 4 · p. 7, §4.4 · p. 8
  • Feature alignment reduces AbsRel from 16.6 to 14.8 on NuScenes-night and from 19.6 to 18.5 on RobotCar-night.The authors describe alignment as anchoring perturbed inputs to the same fine-grained geometric representations as clean images.Source: §4.4 · p. 8
  • RGB reconstruction reduces AbsRel from 9.4 to 8.5 on DS-fog and from 18.5 to 17.0 on RobotCar-night.The authors report that it projects degraded observations into a unified latent domain.Source: §4.4 · p. 8

4.5 Qualitative Analysis

  • Under severe low light, edge responses progress from blurred and noisy contours in the baseline to sharper structures with feature alignment and RGB reconstruction.The full model recovers fine structures, including chair-leg intersections.Source: §4.5 · p. 8, Figure 6 · p. 8

5 Conclusion

OpenDepth combines shallow feature alignment and joint RGB latent reconstruction to map degraded inputs into a unified latent space. The paper reports robust zero-shot generalization and state-of-the-art results on standard and extreme-weather benchmarks.

5 Conclusion

  • OpenDepth uses shallow feature alignment and joint RGB latent reconstruction to map degraded inputs into a unified latent space and reports state-of-the-art zero-shot results across standard and extreme-weather benchmarks.The conclusion describes this strategy as decoupling stylistic interference from geometric perception.Source: §5 · p. 8

Supplementary Material

The supplement provides experimental protocols and datasets, ablations, inference-efficiency benchmarks, and additional qualitative visualizations.

Supplementary Material

  • The supplementary material covers protocols, datasets, robust training, architectural and loss ablations, inference efficiency, and expanded qualitative visualizations.The stated purpose is to further validate robustness and generalization.Source: Supplementary Material · p. 11

A.1 Evaluation Datasets

The supplement describes real-world, synthetic-corruption, and general-purpose evaluation datasets, along with the evaluation protocol and training perturbations. Metrics use aligned predictions evaluated on valid pixels.

A.1 Evaluation Datasets

  • Evaluation covers real-world adverse-condition datasets, four KITTI-C corruption types, and standard benchmarks, with baselines re-evaluated under standardized protocols.The adverse datasets include nighttime and weather subsets; the general benchmarks span indoor, outdoor, and synthetic scenes.Source: §A.1 · p. 11

A.2 Evaluation Metrics

  • AbsRel and threshold accuracy δ1 are computed on valid pixels after scale-and-shift alignment in disparity space, which removes global scale ambiguity while preserving relative depth structure.The aligned predictions are converted back to depth before metric computation.Source: §A.2 · p. 11

B.1 Perturbations

  • Training perturbations follow DepthAnything-AC, and Table B1 summarizes their categories, probabilities, severity ranges, and generation processes.The table caption notes mutually exclusive groups within weather and blur categories.Source: §B.1 · p. 11, Table B1 · p. 12

B.2 Effect of Training Sampling Probabilities

The sampling ablations favor an asymmetric 8:1:1 mixture, with Hypersim dominant but outdoor sources retained. The training pipeline combines clean and perturbed depth supervision with feature alignment and RGB reconstruction, while balanced auxiliary-loss weights yield the best overall trade-off.

B.2 Effect of Training Sampling Probabilities

  • The balanced 1:1:1 mixture performs weakest overall, whereas increasing Hypersim to 4:1:1 consistently improves all four benchmarks.Sampling is batch-level, and epoch length follows the largest dataloader, so ratios redistribute the effective optimization budget.Source: §B.2 · p. 12
  • The 8:1:1 sampling ratio gives the best trade-off across adverse-weather and nighttime benchmarks, while 12:1:1 and outdoor-heavy alternatives do not improve it.Hypersim supplies geometry-rich supervision, while smaller Virtual KITTI and Cityscapes shares provide outdoor-domain anchors.Source: §B.2 · p. 12, Table B2 · p. 13

B.3 Full Training Pipeline

  • The full training pipeline supervises clean and perturbed inputs with shared depth targets, shallow-feature alignment, and reconstruction toward the clean RGB latent.The Step1X-Edit DiT backbone, VAE, and text encoder are frozen; trainable parameters are introduced through LoRA adapters.Source: §B.3 · p. 13

C.1 Sensitivity to Loss Weights

  • Equal auxiliary-loss weights, α=β=γ=1, provide the best overall trade-off across all four robustness benchmarks.Perturbed-input depth supervision is especially important on nighttime benchmarks; feature alignment and cross-image reconstruction address complementary robustness needs.Source: §C.1 · p. 13

C.2 Analysis of the Perturbation Branch Supervision

OpenDepth generalizes to unseen rain, cloud, and Gaussian-noise perturbations, and its ablations favor ground-truth supervision for perturbed inputs and alignment of the first four Single Stream blocks. Discriminative depth regression also outperforms the generative variant, particularly on outdoor KITTI.

C.2 Analysis of the Perturbation Branch Supervision

  • Fit GT outperforms Fit Clean on all four adverse-condition benchmarks, with the largest AbsRel gap on both nighttime sets.Fit GT reduces AbsRel by 0.5–0.8 and raises δ1 by 0.8–1.4 pp; on NuScenes-night and RobotCar-night, AbsRel drops by 0.8 in both cases.Source: §C.2 · pp. 13–14, Table C2 · p. 14

C.3 Sensitivity to the Number of Feature Alignment Layers

  • Aligning the first four Single Stream blocks outperforms aligning one layer, eight layers, or four middle or late layers.First 4 reduces AbsRel versus First 1 on DS-rain (10.0 to 9.7), DS-fog (8.8 to 8.5), NuScenes-night (14.9 to 14.5), and RobotCar-night (17.7 to 17.0); extending to First 8 degrades results, and First 4 also beats Middle 4 and Last 4.Source: §C.3 · p. 14, Table C3 · p. 15

C.4 Analysis of Generative and Discriminative Models

  • On KITTI, the discriminative model records AbsRel 6.30 and RMSE 3.1318, versus generative-model AbsRel 6.54 and RMSE 3.3800.On NYUv2, discriminative AbsRel is 3.88 versus generative AbsRel 3.90; the gap is described as marginal indoors.Source: §C.4 · p. 15, Table C4 · p. 15

C.5 Generalization to Unseen Perturbations

  • OpenDepth outperforms DepthAnything-AC [50] on all three unseen degradation types, including DS-rain AbsRel 12.5 to 9.7 and KITTI-C Gaussian AbsRel 15.3 to 14.1.Neither model was exposed to rain, cloud, or Gaussian noise during training.Source: §C.5 · p. 15, Table C5 · p. 16

D Failure Case Analysis

OpenDepth preserves sharper local geometry in near-field nighttime scenes, but its disparity-based representation can make distant structures less distinct. Extremely dark samples also sometimes assign sky regions less than the farthest depth, yielding a favorable near-range trade-off with room for far-range calibration and sky handling.

D Failure Case Analysis

  • OpenDepth preserves sharper near-field building contours, roadside structures, and local scene grouping in the RobotCar-night example.The authors connect this tendency to shared clean/perturbed depth targets, early feature alignment, and auxiliary RGB latent reconstruction.Source: D Failure Case Analysis · p. 16
  • Disparity-style latent supervision and scale-and-shift fitting under an 80 m cap leave distant structures less separated in some nighttime scenes.The representation allocates stronger effective resolution to near- and mid-range geometry than to far-field content.Source: D Failure Case Analysis · p. 16
  • In a small number of extremely dark samples, sky regions are not mapped to the farthest depth, though the 80 m mask limits their aggregate-metric influence.The model regresses normalized disparity rather than an explicit infinite-depth label and has no dedicated sky prior.Source: D Failure Case Analysis · p. 16
  • The reported nighttime trade-off favors near-range structural fidelity and robustness while leaving room to improve far-range calibration and sky handling.Source: D Failure Case Analysis · p. 16

E Inference Efficiency Analysis

Cached empty-prompt conditioning bypasses the text encoder during inference because its embeddings are constant. Under the stated NYUv2 evaluation setup, OpenDepth is faster than FE2E while Lotus-D remains the fastest and lightest baseline.

E Inference Efficiency Analysis

  • Inference efficiency is benchmarked on NYUv2 at raw 640x480 resolution, batch size 1, and a shared internal resolution of 768x576.Source: E Inference Efficiency Analysis · p. 16
  • Caching deterministic empty-prompt embeddings bypasses Qwen2.5-VL [62] at runtime, reducing GPU memory use and inference latency.The paper states this does not require architectural distillation or sacrifice structural priors.Source: E Inference Efficiency Analysis · p. 16
  • Discriminative OpenDepth achieves 704.4 ms latency and 1.42 img/s throughput, with a 4.0× speedup over FE2E [58] and slightly lower memory use.Memory use is 25.05 GB for OpenDepth versus 25.20 GB for FE2E; Lotus-D [21] remains fastest and lightest overall.Source: E Inference Efficiency Analysis · p. 16

F More Visual Results

The visual results highlight OpenDepth’s nighttime depth structure and include comparisons with DepthAnything-AC, alongside an NYUv2 efficiency and performance comparison.

F More Visual Results

  • On RobotCar-night, OpenDepth preserves clearer near-range boundaries and more coherent local structure, while very distant regions remain less separated.In a few extremely dark scenes, sky regions may receive finite rather than farthest depth.Source: Figure D1 · p. 17
  • The NYUv2 comparison covers the whole dataset at 768x576 internal resolution, with batch size 1, measured on one NVIDIA GeForce RTX 4090 GPU.The caption identifies an inference-efficiency and performance comparison but gives no values.Source: Table D1 · p. 17
  • Figures F1 and F2 compare DepthAnything-AC with OpenDepth.Source: Figure F1 · p. 17, Figure F2 · p. 18

Related papers

Loading 2610.17610v2…