PaperlayerView exact revision on arXiv

Source-linked AI summary

PXDepth: Pixel-Space Modeling for Structure Preserving Monocular Depth Estimation

Zhiyuan Yuan, Guanying Chen, Lingteng Qiu, Ruimao Zhang, Shuguang Cui, Xiaochun Cao

arXiv:2608.16984v1cs.CVcs.AIcs.GR

TL;DR

Monocular depth estimators can lose fine-grained structures and object-boundary fidelity despite strong zero-shot generalization. PXDepth separates global context modeling from pixel-space depth prediction, achieving improved local geometric fidelity with competitive global accuracy and efficient single-pass inference.

  • Problem

    Large-patch ViT tokenization can weaken fine-grained spatial cues that convolutional upsampling may not fully recover, limiting structural fidelity in monocular depth estimation.

  • Method

    PXDepth combines a Global Context Encoder with a full-resolution Pixel-Space Depth Predictor using context-modulated pixel transformers.

  • Results

    Across multiple benchmarks, PXDepth improves local geometric fidelity while preserving competitive global depth accuracy and remaining more efficient than multi-step generative methods.

  • Takeaways & Limitations

    PXDepth provides a single-pass approach for preserving fine-grained geometry without sacrificing global depth consistency.

  • Takeaways & Limitations

    Ambiguous or unreliable depth annotations can cause inaccurate predictions on transparent and reflective surfaces, and relative depth cannot recover metric scale without an external reference.

Abstract

from arXiv · show

Recent monocular depth estimators achieve strong zero-shot generalization, yet often struggle to preserve fine-grained structures and object boundaries. We attribute this limitation to the prevalent combination of large-patch ViT encoders and convolutional decoders, as coarse tokenization can weaken pixel-level cues that upsampling cannot fully recover. To address this issue, we propose PXDepth, a discriminative monocular depth model that separates global context modeling from pixel-level depth prediction. Specifically, a large-patch ViT captures global scene context, while a pixel-space predictor composed of Context-Modulated Pixel Transformer blocks maintains high-resolution spatial representations throughout depth estimation. This design preserves fine structures and sharp boundaries without sacrificing global depth consistency. Across diverse zero-shot benchmarks, PXDepth combines faithful local geometry with competitive global depth accuracy while remaining efficient at inference. Our code and model are available at https://yuanzhy29.github.io/PXDepth-Page/.

1 Introduction

PXDepth addresses structural distortions in monocular depth estimation by separating global context modeling from pixel-space depth prediction. Its architecture combines a Global Context Encoder with a Context-Modulated Pixel Transformer-based Pixel-Space Depth Predictor, achieving finer structural detail and competitive global accuracy while remaining efficient.

  • Motivation: Generative methods such as Pixel-Perfect Depth (PPD) [Xu et al., 2025] improve detail and structural fidelity through pixel-space diffusion, but multi-step denoising makes inference inefficient.PPD uses semantic guidance and a cascade Diffusion Transformer design to produce depth predictions with finer details and better structural fidelity.
  • Proposed method: PXDEPTH is a discriminative architecture that directly models depth in pixel space to recover fine-grained details and improve structural fidelity.It separates global context modeling from pixel-level prediction through its Global Context Encoder and Pixel-Space Depth Predictor.
  • Problem: Mainstream discriminative MDE models weaken fine-grained spatial cues because large-patch ViT tokenization produces low-resolution grids that convolutional upsampling cannot fully recover.These encoders capture strong semantic and scene-level context, but coarse tokenization can cause structural distortions despite visually plausible predictions.
  • Proposed method: PXDEPTH integrates a Global Context Encoder with a Pixel-Space Depth Predictor to preserve global structure and recover fine-grained structural details.The Pixel-Space Depth Predictor refines pixel-space features through Context-Modulated Pixel Transformer blocks, while the Global Context Encoder provides global context.
  • Results: PXDEPTH combines faithful local geometry with competitive global depth accuracy while remaining efficient at inference across multiple benchmarks.Experiments report finer details and better structural fidelity than strong discriminative baselines, while remaining more efficient than multi-step generative methods (see Figure 1).

2 Related Work

Related monocular depth estimation work spans early cue-based and graphical approaches, supervised CNN models, detail-focused discriminative methods, and generative priors. PXDepth differs by predicting depth in pixel space from global context in a single feed-forward pass.

  • Monocular Depth Estimation: Monocular depth estimation progressed from hand-crafted cues and graphical models to supervised dense prediction with CNN-based architectures.Representative CNN approaches include multi-scale prediction, fully convolutional residual networks, ordinal regression, and local planar guidance.
  • Depth Estimation with Detailed Geometry: Recent discriminative methods improve detailed geometry through local refinement, high-resolution prediction, or multi-scale decoding.DepthPro combines a high-resolution design with boundary-aware training for sharper metric depth, while MoGe targets improved local geometry through affine-invariant prediction.
  • Architecture Comparison: PXDepth predicts depth with a Pixel-Space Depth Predictor conditioned on global context in a single feed-forward pass, unlike low-resolution ViT decoding or iterative pixel-space denoising.The comparison contrasts conventional convolutional decoding from low-resolution ViT features and PPD's iterative pixel-space denoising with PXDepth's feed-forward design.
  • Generative Methods: Generative depth methods use diffusion or other generative priors to improve depth and structural details.Marigold adapts image diffusion models, Lotus-2 uses a deterministic two-stage framework, and Pixel-Perfect Depth performs pixel-space diffusion with semantic guidance and a cascade DiT design.

3 Motivation

Existing discriminative monocular depth estimators capture global structure with large-patch ViTs but weaken high-frequency cues through patch tokenization and convolutional upsampling. PXDepth instead separates global-context encoding from pixel-space prediction, using context guidance to preserve fine details while maintaining global geometric consistency.

  • Motivation: The task is to predict a pixel-aligned depth map ˆD ∈R^H×W from an input image I ∈R^H×W×3.
  • Motivation: Large-patch ViT encoders produce low-resolution features that convolutional decoders upsample, weakening high-frequency cues and making simultaneous handling of low- and high-frequency regions difficult.Self-attention remains effective for global-structure capture, but spatially shared convolutional kernels limit detail reconstruction.
  • Motivation: Diffusion-based alternatives leverage pretrained latent-diffusion priors but typically compress depth into VAE latents, which can weaken sharp depth discontinuities and structural fidelity.PPD [Xu et al., 2025] instead performs diffusion directly in pixel space with semantic guidance and a cascade DiT [Peebles and Xie, 2023] design.
  • Main Idea: PXDepth separates global-context encoding from pixel-space prediction, allowing the predictor to use the input image under context guidance rather than reconstructing depth solely from low-resolution features.This design preserves high-frequency cues while maintaining global geometric consistency.

4 Method

PXDepth separates global scene-context extraction from pixel-space depth prediction: a large-patch ViT supplies context, while a full-resolution predictor uses CM-PiT blocks to refine pixel features into depth. Context-guided modulation and temporary token compaction provide efficient attention while preserving pixel-space representations.

  • 4 Method: PXDepth combines a Global Context Encoder with a Pixel-Space Depth Predictor that maintains full-resolution features and maps refined representations to depth.The predictor uses CM-PiT blocks conditioned on global context and a lightweight depth head.
  • Global Context Encoder: The Global Context Encoder uses a large-patch ViT to tokenize the image into a low-resolution grid and aggregate global information with self-attention.It can inherit priors from pretrained vision foundation models including DINOv2, DINOv3 [Sim´eoni et al., 2025], and MoGe-2 [Wang et al., 2025b].
  • Pixel Embedding: A 1 × 1 convolution embeds each RGB pixel into a pixel-space feature stream whose spatial resolution remains preserved throughout depth prediction.This differs from conventional decoders that reconstruct depth solely from low-resolution ViT features.
  • Context-Modulated Pixel Transformer: CM-PiT blocks condition attention and feed-forward layers on context through Context-Guided Adaptive Normalization, projecting context into pixel-wise scale, shift, and residual gates.This strategy is contrasted with direct addition and cross-attention, and was originally introduced in PixelDiT [Yu et al., 2026b].
  • Coarse-to-Fine Pixel Compaction: Pixel token compaction reduces attention length from HW to L = (H/p)(W/p), a p^2-fold reduction, while expansion before residual updates preserves pixel-space features.Early blocks use p = P for coarse structure modeling, whereas later blocks use p = P/2 for finer-detail refinement.

5 Experiments

PXDepth is evaluated zero-shot across diverse benchmarks and achieves strong global depth accuracy, leading boundary fidelity, and competitive inference efficiency. Ablations show that global context, coarse-to-fine pixel compaction, and adaptive normalization each contribute to performance.

  • Global Depth Accuracy: PXDepth achieves better global depth accuracy than structure-aware methods while remaining comparable to methods focused on global prediction across the MoGe and MDA benchmarks.The remaining gap to MoGe-2 [Wang et al., 2025b] is attributed to MoGe-2’s substantially larger and more diverse training data.
  • Boundary Fidelity: PXDepth achieves the best mean CD and second-best mean Acc, preserving accurate boundary geometry with fewer flying points and less distortion than competing methods.PPD [Xu et al., 2025] preserves sharp boundaries but has boundary noise, while MDA [Bian et al., 2026] reduces flying points but produces more distorted structure.
  • Inference Efficiency: At 518×518, PXDepth is slower than MoGe-2, comparable to DA V2 and InfiniDepth, and approximately 3.7× faster than PPD.Its efficiency over generative methods comes from single-pass feed-forward prediction rather than iterative denoising.
  • Effect of Global Context Encoder: Removing the Global Context Encoder substantially degrades global and boundary metrics, while MoGe-2 initialization improves both over DINOv3.Ablation variants use the same training protocol on Hypersim [Roberts et al., 2021] and UrbanSyn [G´omez et al., 2025], then evaluate zero-shot on HiRoom [Lin et al., 2025].
  • Component Ablations: Coarse-to-fine pixel compaction improves boundary quality at comparable global accuracy, while adaptive normalization outperforms addition and cross-attention with only modest extra inference time.Reducing compaction size in later blocks refines local geometric details, and adaptive normalization provides the best combined ablation performance.

6 Conclusion · A Additional Implementation Details

PXDepth addresses coarse spatial cues in common monocular depth architectures by pairing global context modeling with a pixel-space depth predictor. The paper also identifies challenges with transparent or reflective surfaces and metric-scale recovery, while the appendix documents implementation details.

  • 6 Conclusion: PXDepth targets the loss of fine-grained spatial cues caused by large-patch ViT tokenization that convolutional upsampling may not fully recover.
  • 6 Conclusion: The model combines a Global Context Encoder with a Pixel-Space Depth Predictor to address this architectural limitation.
  • 6 Conclusion: The Global Context Encoder captures global structure for monocular depth estimation.
  • 6 Conclusion: The Pixel-Space Depth Predictor preserves fine-grained geometric information during prediction.
  • 6 Conclusion: Ambiguous or unreliable depth annotations can cause inaccurate predictions on transparent and reflective surfaces.
  • 6 Conclusion: The relative-depth formulation cannot recover metric scale without an external reference, motivating improved supervision and metric cues for scale-aware estimation.
  • A Additional Implementation Details: The appendix provides additional details about training configuration, model architecture, and evaluation protocol.

A.1 Training Data and Configuration · A.2 Training Objective

PXDepth is trained exclusively on synthetic RGB-D data using two stages: low-resolution pretraining followed by high-resolution fine-tuning. Its objective combines normalized-depth, multi-scale gradient, and binary-mask supervision, weighted to preserve accurate depth and sharp local structure.

  • A.1 Training Data and Configuration: Training uses exclusively synthetic RGB-D data, with Stage 1 for low-resolution pretraining and Stage 2 for high-resolution fine-tuning.Table 4 summarizes stage-specific sampling weights, native RGB resolutions, and scene distributions; the weights are relative mixture factors, not percentages or dataset sizes.
  • A.1 Training Data and Configuration: Optimization uses AdamW with β = (0.9, 0.95), weight decay 0.01, cosine OneCycle schedules, and 2% warm-up on NVIDIA RTX H100 GPUs.Per-GPU batch sizes are 8 and 4 in Stages 1 and 2, respectively.
  • A.2 Training Objective: Ground-truth depth is converted to log depth and normalized per map using the 2% and 98% percentiles of valid log-depth values.This normalization reduces depth’s dynamic range across scenes.
  • A.2 Training Objective: Normalized predictions are supervised with an ℓ1 loss over valid depth pixels.The prediction is denoted ˆd and the valid-pixel set is V.
  • A.2 Training Objective: A multi-scale depth-gradient loss supervises gradients of the depth error at strides s ∈{1, 2, 4, 8} to preserve local variations and sharp discontinuities.The loss uses stride-specific error maps and valid-pixel sets.
  • A.2 Training Objective: The mask head applies binary cross-entropy to sigmoid mask probabilities, retaining inference pixels whose predicted probability exceeds 0.5.Supervision is applied over the labeled pixel set M, with binary target m(u).
  • A.2 Training Objective: The final objective combines normalized-depth, gradient, and mask losses with λd = 1, λg = 0.5, and λm = 0.5.These three terms are combined into the final training loss.

B More Architecture Details … C.2 Evaluation Datasets and Resolution Handling

PXDepth combines a ViT-L/14 global context encoder with pixel-space depth prediction built from learned token compaction and context-guided adaptive normalization. The evaluation protocol standardizes baseline implementations, checkpoints, and dataset-specific resolutions for fair comparison.

  • B.1 Model Configuration: ViT-L/14, initialized from MoGe-2, extracts layers 5, 11, 17, and 23, projects them to 1,024 channels, and sums them into Fctx.The resulting global context feature is used by the pixel-space predictor.
  • B.1 Model Configuration: CM-PiT blocks use RMSNorm, SwiGLU, 2D RoPE, 1,536-dimensional attention, 24 heads, query/key normalization, and learned sigmoid output gates.Each attention output is modulated by a gate predicted from its input token.
  • B.2 Pixel Token Compaction: Pixel token compaction partitions features into non-overlapping p×p regions, applies learned projections and global attention, then restores attended tokens to original pixel locations.This replaces spatial pooling while preserving the pixel-space layout.
  • B.3 Context-Guided Adaptive Normalization: Context-guided normalization expands Fctx into pixel-wise shift, scale-residual, and residual-gate maps applied directly to pixel features without interpolation or learned upsampling.The six parameter maps each contain d_pix channels and modulate attention and feed-forward branches.
  • C.1 Baseline Model Configurations: Baselines are evaluated with official implementations and public checkpoints, including specified publicly released monocular or variant configurations for DA V2, DepthPro, InfiniDepth, MoGe-2, and PPD.The passage specifies model variants such as DA V2’s ViT-L relative-depth model and MoGe-2’s ViT-L geometry-and-normals model.
  • C.2 Evaluation Datasets and Resolution Handling: Inputs are resized to each method’s target inference resolution, then predicted depths are nearest-neighbor resized to the dataset’s evaluation resolution before metrics and visualizations.This places all methods at the same resolution for each dataset and supports fair comparison.

C.3 Boundary Evaluation Protocol

The MDA boundary protocol standardizes image preprocessing, depth-boundary extraction, and 3D point-cloud comparison across zero-shot benchmarks. It reports boundary accuracy, completeness, and Chamfer distance after ICP alignment, excluding samples with insufficient valid points.

  • C.3 Boundary Evaluation Protocol: Evaluation uses benchmark-specific resolutions from Table 5, with symmetric principal-point cropping, resizing and center cropping, updated intrinsics, and nearest-neighbor depth resampling.Table 5 summarizes configured resolutions and scene distributions for the zero-shot benchmarks.
  • C.3 Boundary Evaluation Protocol: Ground-truth depth is clipped to [0.1, 65] meters, normalized to 8-bit, processed by Canny with thresholds 100 and 200, and cleaned near invalid depth.Pixels adjacent to invalid depth are removed using one 2 × 2 dilation, and samples with fewer than ten valid boundary points are excluded.
  • C.3 Boundary Evaluation Protocol: Predicted and ground-truth depths at valid boundaries are back-projected into 3D using the ground-truth intrinsics and aligned by point-to-point ICP with a 0.1 m threshold.The protocol therefore compares corresponding predicted and ground-truth boundary point clouds after registration.
  • C.3 Boundary Evaluation Protocol: Accuracy measures predicted-to-ground-truth nearest-neighbor distance, completeness reverses that direction, and Chamfer distance averages the two; Acc and CD are reported in millimeters.Algorithm 1 summarizes the same sequence from boundary detection through point-set conversion, filtering, alignment, and metric computation.

D Additional Quantitative Results

Additional evaluations probe PXDepth’s zero-shot generalization on controlled synthetic scenes and real-world datasets, covering affine-invariant depth accuracy and 3D boundary quality. Results are reported on Synth4K, iBims-1, and HAMMER.

  • Evaluation results: The additional evaluation tests zero-shot generalization across synthetic and real-world scenes using affine-invariant depth accuracy and 3D boundary metrics.Synth4K provides controlled-geometry synthetic evaluation, while iBims-1 and HAMMER support 3D boundary assessment.
  • Synth4K: Synth4K evaluates zero-shot affine-invariant depth estimation across five indoor and outdoor game-scene subsets with dense synthetic depth.The benchmark was introduced by InfiniDepth and is used to assess performance under controlled geometry.
  • iBims-1 and HAMMER: iBims-1 and HAMMER provide quantitative comparisons using 3D boundary metrics, with Acc and CD reported in millimeters.These datasets extend the evaluation to real-world scenes and boundary-focused measurements.

E Additional Qualitative Results

Additional qualitative comparisons show that PXDepth preserves fine-detail structures.

  • E Additional Qualitative Results: Qualitative comparisons show that PXDepth preserves fine-detail structures.
Loading 2608.16984v1…