Pixel-Perfect Depth with Semantics-Prompted Diffusion Transformers

• SP-DiT integrates semantic representations to preserve global consistency while enhancing fine-grained details, improving NYUv2 AbsRel by up to 78%.Semantic regularization stabilizes alignment between vision-foundation representations and DiT features.Source: §1 · pp. 2–3
• Cas-DiT uses larger patches early and smaller patches later, reducing token computation while improving global-structure modeling and fine-detail generation.This coarse-to-fine strategy progressively increases the number of tokens across DiT blocks.Source: §1 · pp. 2–3
• Pixel-space diffusion avoids VAE compression artifacts, targeting flying-pixel-free point clouds from estimated depth maps.Prior latent diffusion models compress depth maps through a VAE, which loses edge sharpness and structural fidelity.Source: §1 · pp. 2–3
• The model achieves the best performance among all published generative depth estimation models across five benchmarks.Source: §1 · p. 3
• The model significantly outperforms previous models on an edge-aware point-cloud metric designed to assess flying pixels at object boundaries.Source: §1 · p. 3

2.1 Monocular Depth Estimation

• Monocular depth estimation supports applications including 3D reconstruction, novel view synthesis, and robotic manipulation, but existing models can produce flying pixels at boundaries and fine details.Source: §2.1 · p. 3
• Recent generative depth methods fine-tune Stable Diffusion in latent space, requiring VAE compression that can introduce flying pixels in reconstructed point clouds.Source: §2.1 · p. 3, §2.2 · p. 3

2.2 Diffusion Generative Models

• Pixel-space diffusion avoids latent compression but is computationally intensive and slow to converge at high resolutions.Source: §2.2 · pp. 3–4
• Pixel-Perfect Depth uses semantic prompting and Cascade DiT to enable efficient high-resolution depth estimation without latent compression.The overview combines image-plus-noise input, vision-foundation semantic features, and direct pixel-space diffusion.Source: §2.2 · p. 4, Figure 3 · p. 4

3.1 Pixel-Perfect Depth

• The model performs diffusion directly in pixel space, allowing it to model depth discontinuities at object edges while avoiding latent-space VAE artifacts.The design targets flying-pixel-free point clouds from estimated depth maps.Source: §3.1 · p. 4

3.2 Generative Formulation

• Flow Matching learns a continuous transformation from Gaussian noise to depth samples, with the model predicting a velocity field from the noisy sample, time step, and input image.Inference iteratively solves the corresponding ODE from noise toward the depth sample.Source: §3.2 · pp. 4–5

3.3 Semantics-Prompted Diffusion Transformers

• SP-DiT extracts high-level semantic representations from the input image and integrates them into DiT tokens to preserve global semantics and fine-grained details.Semantic features are normalized, aligned to token resolution, and fused through an MLP before subsequent semantics-prompted blocks.Source: §3.3 · pp. 6–7
• Without SP-DiT, pixel-space diffusion struggles with global image structure and fine-grained details, whereas the proposed semantic prompting improves both aspects.Experiments use encoders from DINOv2 [44], VGGT [65], MAE [24], and Depth Anything v2 [82], all of which improve performance and training stability.Source: §3.3 · pp. 6–7, Figure 6 · p. 7

3.4 Cascade DiT Design

• The Cascaded DiT Design uses larger patches in early blocks and smaller patches later, reducing token processing costs while refining fine-grained spatial details.The first N/2 blocks form the coarse stage, and the remaining N/2 SP-DiT blocks form the fine stage.Source: §3.4 · p. 7

3.5 Implementation Details

• Table 1 reports that the model outperforms other generative depth models on five zero-shot benchmarks, with 512×512 and 1024×768 variants evaluated using AbsRel and δ1.The architecture uses 24 DiT blocks, transitions from (H/16) × (W/16) to (H/8) × (W/8) tokens, and outputs an H × W depth map.Source: Table 1 · p. 8, §3.5 · p. 8
• Depth values are log-transformed and normalized using the 2% and 98% depth percentiles before diffusion training.The transformation uses ˜d = log(d + ϵ), with a small positive ϵ for numerical stability.Source: §3.5 · p. 8
• The model is trained at 512×512 and 1024×768 resolutions with AdamW, a constant learning rate of 1 × 10−4, velocity MSE, and gradient matching loss.Training uses eight NVIDIA GPUs with a per-GPU batch size of four.Source: §3.5 · p. 8

4.1 Experimental Setup

• The 512 × 512 model is trained on approximately 54K Hypersim samples, while the 1024 × 768 model adds four datasets totaling 70.5K samples.The added datasets are UrbanSyn, UnrealStereo4K, VKITTI, and TartanAir.Source: §4.1 · p. 8

4.2 Ablations and Analysis

• SP-DiT improves NYUv2 AbsRel by 78%, while Cas-DiT reduces RTX 4090 inference time by 30% and improves accuracy.The ablation compares against vanilla DiT and attributes the gains to semantic prompting and progressive token expansion.Source: §4.2 · p. 9, Table 2 · p. 9
• Ablations show that pretrained encoders from MAE, DINOv2, Depth Anything v2, and VGGT all significantly improve SP-DiT performance.The comparison is reported in the VFM ablation study.Source: §4.2 · p. 9, Table 3 · p. 10

4.3 Zero-Shot Relative Depth Estimation

• The model outperforms all other generative depth models on every metric across five real-world benchmarks.Evaluation covers NYUv2, KITTI, ETH3D, ScanNet, and DIODE using AbsRel and δ1.Source: §4.3 · p. 9, §4.1 · p. 9

4.4 Edge-Aware Point Cloud Evaluation

• The edge-aware point-cloud evaluation shows the method performs best, while VAE reconstruction increases Chamfer Distance through flying pixels.The evaluation uses Hypersim ground-truth point clouds and measures Chamfer Distance near Canny-extracted edges.Source: §4.4 · pp. 9–10, Table 4 · p. 10

5 Conclusion

• Pixel-space diffusion with Semantics-Prompted DiT and Cascade DiT Design produces high-quality, flying-pixel-free point clouds and significantly outperforms prior models in edge-aware evaluation.The approach avoids flying pixels attributed to VAE compression in latent-space diffusion.Source: §5 · p. 10, Table 5 · p. 11
• Video applications lack temporal consistency, causing slight depth flickering across frames.The limitation is shared with most image-based diffusion models.Source: §5 · p. 10
• The multi-step diffusion process is slower than discriminative models such as Depth Anything v2.The paper suggests video-depth methods and DiT acceleration strategies as future directions.Source: §5 · p. 10

A Qualitative Comparisons with MoGe

• The proposed model produces significantly fewer flying pixels than MoGe at edges and fine structures.MoGe and other discriminative models are described as suffering from flying pixels in these regions.Source: A Qualitative Comparisons with MoGe · p. 11

B Additional Discussion with REPA

• Semantics-Prompted DiT significantly outperforms REPA when both methods use the same pretrained vision encoder.The paper attributes the difference to direct semantic integration during training and inference, unlike REPA’s implicit alignment and inference behavior.Source: B Additional Discussion with REPA · p. 11
• REPA cannot use the pretrained vision encoder to provide semantic prompts at inference, whereas Semantics-Prompted DiT incorporates high-level semantics during inference.The comparison is presented as one explanation for the proposed method’s superiority.Source: B Additional Discussion with REPA · p. 11
• Table 6 reports runtime measured on an RTX 4090 GPU using the 512 × 512 model with 4 denoising steps, while Table 7 compares PPD-Large and PPD-Small.Source: Table 6 · p. 12, Table 7 · p. 12

C Analysis of Flying Pixels in Different Types of VAEs

• Both VAE variants exhibit flying pixels at object edges and fine details.The analysis reconstructs ground-truth depth maps using VAEs with different latent dimensions.Source: C Analysis of Flying Pixels in… · p. 12
• The shared artifacts reveal a common weakness of VAE reconstructions in preserving precise geometric structures.Source: C Analysis of Flying Pixels in… · p. 12
• Increasing the latent dimension from 4 to 16 does not eliminate flying pixels in the tested VAE reconstructions.VAE-d4 uses Stable Diffusion 2, while VAE-d16 uses Stable Diffusion 3.5.Source: C Analysis of Flying Pixels in… · p. 12

D Efficiency and Lightweight Variant

• PPD is slower than Depth Anything v2 but has inference time comparable to Depth Pro.Its slower runtime results from the multi-step diffusion process.Source: D Efficiency and Lightweight Variant · p. 12
• PPD-Small achieves substantially faster runtime with only marginal accuracy loss than PPD-Large.PPD-Small uses DiT-Small with fewer parameters for efficient inference.Source: D Efficiency and Lightweight Variant · p. 12
• Using a larger VAE latent dimension still fails to eliminate flying pixels at object edges and details.Both VAE-d4 and VAE-d16 reconstructions show these artifacts.Source: Figure 8 · p. 12