PaperlayerView exact revision on arXiv

Source-linked AI summary

Efficient and High-Quality Depth Estimation via Pixel-Space Diffusion with Linear Attention

Bingde Liu, Wu Ran, Jinglei Zhang, Huanhuan Yuan, Chao Ma

arXiv:2608.30129v1cs.CV

TL;DR

High-resolution monocular depth estimation needs generative detail without the quadratic attention and multi-step denoising costs of existing approaches. Lapis combines linear-attention pixel-space diffusion with patch-level consistency, pixel-level refinement, and direct x-prediction, achieving SOTA accuracy and boundary sharpness while reducing latency. Its current scope is single-image relative depth, and reduced-precision linear attention remains numerically unstable.

  • Problem

    High-resolution generative depth estimation is limited by quadratic attention, multi-step denoising, structural misalignment, pixel-detail loss, and one-step sampling instability.

  • Method

    Lapis uses linear-attention pixel-space diffusion with a Patch-level Consistency Module, a Pixel-level Refinement Module, and direct x-prediction for one-step inference.

  • Results

    Across 12 benchmarks, Lapis achieves SOTA accuracy and boundary sharpness across resolutions; at 1080P, it attains 5.1 average AbsRel and requires only 13% of the strongest competitor’s latency.

  • Takeaways & Limitations

    Lapis provides a scalable, computationally efficient foundation for high-resolution 3D perception that bridges generative quality and discriminative speed.

  • Takeaways & Limitations

    Lapis currently focuses on single-image relative depth, while linear attention remains numerically unstable in reduced-precision formats such as fp16.

Abstract

from arXiv · show

This work presents $\textbf{Lapis}$, a $\textbf{l}$inear-$\textbf{a}$ttention-based $\textbf{pi}$xel-$\textbf{s}$pace generative framework that achieves efficient and high-fidelity depth estimation with one-step diffusion. While generative frameworks have significantly advanced monocular depth estimation with superior detail fidelity, the $\mathcal{O}(N^2)$ complexity of standard attention and the multi-step denoising process introduce prohibitive computational costs when scaling them to high-resolution image applications. Although linear attention and one-step prediction are intuitively viable, directly applying them leads to poor structural consistency, detail loss, and noise. Lapis rectifies these limitations through a coarse-to-fine hierarchy. Specifically, a Patch-level Consistency Module restores structural coherence by integrating semantic and spatial priors. Subsequently, a Pixel-level Refinement Module recovers sharp geometric boundaries via skip-connection-based pixel correspondence. Furthermore, to mitigate sampling noise inherent in one-step diffusion, we leverage the manifold assumption and adopt a direct $\mathbf{x}$-prediction strategy to target the clean data manifold. Extensive evaluations on multiple benchmarks demonstrate that Lapis consistently achieves state-of-the-art (SOTA) accuracy and boundary sharpness across various resolutions, reducing inference latency by up to 7.6$\times$ at 1080P and 10.9$\times$ at 1440P resolution compared to previous SOTA generative models.

1 Introduction

Lapis addresses the efficiency–fidelity trade-off in high-resolution monocular depth estimation with linear attention, one-step pixel-space diffusion, and a coarse-to-fine design. Its modules restore global structure and pixel-level detail while direct x-prediction suppresses one-step sampling noise.

  • Motivation: High-resolution depth estimation exposes a fidelity–efficiency trade-off because generative models face quadratic attention costs, multi-step denoising, or information loss and weak global consistency.Latent-space models lose information through the VAE bottleneck, while pixel-space models with standard attention are expensive at high resolutions.
  • Method: Lapis introduces a Patch-level Consistency Module that integrates semantic and spatial priors to restore global structural coherence.The module addresses patch-level structural misalignment in linearized attention.
  • Method: The Pixel-level Refinement Module uses skip-connection-based pixel correspondence to recover sharp boundaries and fine-grained geometric details.It re-integrates pixel-level input cues during decoding to counteract detail loss.
  • Method: Direct pixel-space x-prediction targets the clean data manifold and suppresses the noise and instability of one-step diffusion.This replaces volatile v-prediction under the manifold assumption that natural data lie on a low-dimensional manifold.
  • Results: At 1080P, Lapis achieves a 5.1 average AbsRel, a 10% relative improvement over the strongest competitor, while requiring only 13% of its latency.Across 12 benchmarks, it reports SOTA accuracy and boundary sharpness across varying resolutions with competitive efficiency.

2 Related Work

Related work spans discriminative monocular depth estimation, generative diffusion models, and linear attention. Lapis applies linear attention to dense depth estimation while addressing its structural and pixel-level fidelity gaps.

  • Monocular Depth Estimation: Discriminative monocular depth models progressed from domain-specific CNN regression toward open-domain systems using diverse datasets, Vision Transformers, pre-trained encoders, and massive training data.Recent discriminative models achieve strong zero-shot robustness but may produce over-smoothed results.
  • Generative Depth Estimation: Early generative depth models use latent diffusion and ϵ-prediction with 10–50 sampling steps, while flow-matching v-prediction methods still require at least 4 steps to suppress noise artifacts.Existing one-step x-prediction models form another acceleration direction.
  • Linear Attention: Linear attention removes Softmax through kernel approximations to achieve O(N) complexity, but dense depth estimation is especially sensitive to representational decay from linear approximations.Prior visual applications mainly target categorical tasks or perceptual plausibility rather than metric accuracy.

3 Method

Lapis combines linearized pixel-space diffusion with a coarse-to-fine architecture to achieve efficient one-step depth estimation while preserving global structure and pixel-level detail.

  • 3.1 One-step Pixel-Space Diffusion: Lapis replaces standard DiT self-attention with linearized attention and directly predicts clean depth maps in pixel-space using one-step diffusion.The framework avoids latent autoencoders and multi-step sampling overhead.
  • 3.3 Patch-level Consistency Module: The Patch-level Consistency Module integrates semantic and spatial priors to enforce global structural alignment across the token grid.Global and local consistency mechanisms address structural misalignment and patch discontinuities.
  • 3.1 One-step Pixel-Space Diffusion: The training objective combines velocity-based flow supervision with multi-scale gradient matching in x-space to balance accuracy and boundary sharpness.The total loss is L_total = L_velocity + λ_grad L_grad, with λ_grad controlling the trade-off.
  • 3.3 Patch-level Consistency Module: The ConvFFN adds a residual 3 × 3 depth-wise convolution to preserve spatial continuity across adjacent tokens and suppress grid artifacts.Its local inductive bias complements the global structural rectification provided by the PCM.
  • 3.4 Pixel-Level Refinement Module: The Pixel-level Refinement Module reconnects pixel-level input cues with upsampled latent features to recover sharp geometric boundaries and fine details.The PRM uses long-range skip connections, positional encoding, projection, and PixelShuffle upsampling before depth prediction.

4 Experiments

Experiments evaluate Lapis across native and high-resolution settings, boundary quality, runtime, and component or prediction-target ablations. The results show strong accuracy, sharp boundaries, and efficient one-step inference across benchmarks.

  • Implementation Details: About 122K synthetic training samples from five datasets are resized to 1024 × 768 and concatenated without weighted sampling.The datasets are Hypersim, TartanAir, VKITTI2, UrbanSyn, and MatrixCity.
  • Implementation Details: 520M parameters and a frozen DINOv2-Large semantic encoder define the main architecture, while the PRM uses three blocks.The backbone preserves DiT-Large hidden dimensions and block count while replacing attention with linearized operations.
  • Native Resolution: Lapis achieves SOTA zero-shot relative depth performance on five native-resolution real-world datasets.Evaluation uses AbsRel and δ1 accuracy on NYUv2, KITTI, ETH3D, ScanNet, and DIODE.
  • Test-time Resolution Scaling: Approximately 10% and 19% AbsRel improvements are reported at 1080P and 1440P, respectively, over current SOTA models.These evaluations use five high-resolution datasets and assess preservation of global structure under resolution changes.
  • Boundary Sharpness: Lapis consistently outperforms discriminative and generative baselines in boundary-sharpness F1-score across test resolutions.The metric targets sharp occlusion boundaries and fine-grained structural transitions on Hypersim, Sintel, and Spring.
  • Ablation and Prediction Targets: One-step x-prediction is the optimal sampling choice, while PCM supports global coherence and PRM restores high-frequency geometry.Removing Global PCM sharply reduces accuracy, Local PCM raises AbsRel from 4.8 to 5.1, and PRM is essential for detailed geometry despite marginal accuracy effects.

5 Conclusion

The paper presents Lapis as a linear-attention, pixel-space generative model for efficient high-quality depth estimation at high resolution. Its dual-level refinement and direct x-prediction yield SOTA zero-shot performance with substantially lower inference latency.

  • Conclusion: Lapis decouples patch-level context modeling from pixel-level refinement and uses direct x-prediction to address linearized-backbone degradation and one-step diffusion noise.The model is designed for efficient high-resolution depth estimation.
  • Conclusion: Extensive experiments show SOTA zero-shot performance while significantly reducing inference latency.The paper positions Lapis as a scalable computational foundation for real-world 3D perception.

A More Implementation Details

Implementation details specify comparable ablation configurations, evaluation protocols, and preprocessing choices for relative-depth and boundary-sharpness experiments.

  • Architecture: Lapis and its ablation variant remain comparable with DiT-L and DiT-B in hidden dimensions and block counts.Table F lists the detailed architectural configurations.
  • Relative Depth Evaluation: Relative-depth evaluation follows Marigold preprocessing, least-squares alignment, and metric computation protocols.ETH3D is downsampled to 2048 × 1360, and unreliable DIODE transitions are masked using a Sobel threshold of 0.3.
  • Boundary Sharpness Evaluation: Boundary-sharpness evaluation follows the Depth Pro protocol with least-squares alignment and clipping to each dataset’s valid depth range.Predictions are resized to ground-truth resolution before final sharpness metrics are computed.

B.3 Runtime Analysis

The runtime analysis combines qualitative point-cloud inspection with controlled latency evaluation. Lapis is reported to suppress artifacts while outperforming generative baselines in efficiency.

  • Runtime: Runtime is measured on one NVIDIA RTX Pro 6000 Blackwell GPU with batch size 1.Results average 100 iterations after 20 warm-up rounds, with compilation and memory-efficient attention used to reduce overhead.
  • Runtime: Lapis significantly outperforms existing generative models in inference latency while achieving throughput competitive with regression-based architectures.Table 4 reports latency, throughput, and peak GPU memory across resolutions.
  • Qualitative Point Clouds: Lapis suppresses flying-pixel artifacts and produces more spatially consistent surfaces than compared discriminative and latent-space generative methods.The qualitative comparison focuses on point clouds without post-processing and highlights artifacts in red boxes.

C Edge-Aware Point Cloud Evaluation

Lapis is evaluated with edge-aware point clouds to assess 3D structural integrity on the official Hypersim test split. This provides a more rigorous perspective than boundary F1-score alone.

  • Edge-aware point clouds evaluate Lapis from the perspective of 3D structural integrity.The evaluation is conducted on the official Hypersim test split.
  • The evaluation is presented as more rigorous than boundary F1-score for assessing structural fidelity.
  • The setup addresses flying pixel artifacts associated with over-smoothing or lossy VAE compression in other paradigms.

D Extended Ablation Study

The extended ablations examine linear-attention kernels and semantic conditioning, finding competitive kernel performance and advantages for spatially corresponding conditioning strategies.

  • Kernel Ablation: Linearized configurations maintain performance parity with standard Softmax attention, while the ELU + 1 kernel achieves a slightly superior overall result.
  • Semantic Conditioning Strategies: Direct one-to-one spatial correspondence in MLP and AdaLN variants significantly outperforms global Linear Cross-Attention for preserving geometric layout.
  • Kernel Ablation: The kernel ablation compares ReLU, ELU+1, Focused ReLU, and standard Softmax attention within the complete Lapis architecture.
  • Qualitative Results: Qualitative comparisons on Hypersim, Sintel, and Spring show sharp structural boundaries across complex indoor and wide-scale outdoor scenes.
  • Semantic Conditioning Strategies: Interleaved AdaLN-Zero is selected as the final semantic-conditioning configuration because it offers slightly better average results without additional parameters.

E More Visual Results

Additional qualitative results on synthetic benchmarks and open-domain images demonstrate Lapis’s robustness and structural fidelity across diverse scenarios.

  • Lapis preserves structural fidelity across Hypersim, Sintel, Spring, and open-domain images from Pexels and Unsplash.
  • The visual comparisons are intended to demonstrate robustness across diverse scenarios.

F Limitations and Future Work

The paper identifies numerical instability in reduced-precision linear attention and limits the current framework to single-image relative depth estimation. Future work targets more stable formulations and extensions to metric depth and video consistency.

  • Linear attention currently exhibits numerical instability in reduced-precision formats such as fp16.
  • The instability is attributed to the lack of explicit normalization in attention-score computation.
  • Lapis currently focuses on single-image relative depth estimation rather than metric depth or video-consistent estimation.
  • Future extensions to metric depth and video consistency would broaden applications in robotics and temporally coherent scene reconstruction.
Loading 2608.30129v1…