PaperlayerView exact revision on arXiv

Source-linked AI summary

PointDiT: Pixel-Space Diffusion for Monocular Geometry Estimation

Haofei Xu, Rundi Wu, Philipp Henzler, Nikolai Kalischek, Michael Oechsle, Fabian Manhardt, Marc Pollefeys, Andreas Geiger, Federico Tombari, Michael Niemeyer

arXiv:2607.02515v1cs.CV

TL;DR

Monocular 3D reconstruction remains challenged by scale, depth, and geometric-detail ambiguities. PointDiT addresses this with a minimalist pixel-space diffusion transformer operating directly on point maps, surpassing complex alternatives while producing sharper boundaries and robust depth in ambiguous regions.

  • Problem

    Monocular 3D reconstruction remains ill-posed because perspective projection creates inherent scale and depth ambiguities, while complex regions can yield over-smoothed geometry lacking high-frequency detail.

  • Method

    PointDiT uses a plain ViT diffusion model trained directly on raw point-map patches and conditioned on frozen multi-layer DINOv3 image tokens.

  • Results

    PointDiT surpasses complex latent-based and hybrid alternatives; BF1 boundary sharpness improves from 9.41 to 10.50 over the best baseline.

  • Takeaways & Limitations

    Pixel-space diffusion can model dense 3D geometry effectively with sharper boundaries and supports both single-step and multi-step inference.

  • Takeaways & Limitations

    The model is trained at fixed resolutions and remains weaker on outdoor scenes because of limited training-data scale and diversity.

Abstract

from arXiv · show

State-of-the-art single-image 3D reconstruction methods often rely on complex hybrid architectures and loss functions, or compress geometry into latent spaces in order to leverage pre-trained latent diffusion models. In this work, we show that such architectural overhead and intricate loss formulations are unnecessary. We introduce a minimalist pixel-space Diffusion Transformer, built on a plain ViT, that operates directly on raw 3D point map patches and is conditioned on image tokens from a pre-trained DINOv3. Unlike existing latent diffusion approaches, we train our diffusion backbone entirely from scratch, eliminating the need for point map tokenizers. Despite its simplicity, our approach surpasses complex latent-based diffusion models while remaining significantly simpler than hybrid alternatives. Notably, it produces sharper geometric structure and is more robust in highly ambiguous regions, such as transparent objects.

1. Introduction

PointDiT introduces a minimalist pixel-space diffusion framework that predicts dense 3D point maps directly from RGB images using a plain ViT conditioned on DINOv3 tokens. It avoids VAE-induced noise and deterministic over-smoothing, achieving sharper geometry and strong performance in ambiguous regions such as transparent objects.

  • Motivation: Monocular geometry estimation predicts dense 3D point maps from single RGB images, but perspective projection leaves 3D representation inherently ambiguous in scale and depth.Point maps represent geometry directly in camera coordinates, unlike depth maps, which require camera intrinsics for 3D recovery.
  • Limitations of Existing Methods: Deterministic regressors often produce over-smoothed geometry, while latent diffusion models introduce reconstruction noise through point-map compression with a VAE.The paper identifies these as limitations of the two dominant existing paradigms.
  • Method: PointDiT trains a plain ViT directly on raw point-map patches in pixel space, using diffusion to model ambiguous regions without VAE signal degradation.The framework is designed to eliminate architectural overhead and intricate loss formulations while operating directly in raw point-map space.
  • Method: DINOv3 image tokens condition the plain ViT backbone, injecting strong representation-learning priors into diffusion-based geometry prediction.The model also works with naive linear patchification, but DINOv3 conditioning substantially improves performance.
  • Results: PointDiT surpasses complex latent-based diffusion models while remaining simpler than hybrid deterministic alternatives, producing sharper boundaries and resolving depth in transparent objects.It achieves competitive results with one-step diffusion, with structural details improving further as sampling steps increase.

2. Related Work

Prior work adapts latent diffusion, pixel-space diffusion, pretrained visual representations, and monocular depth or point-map estimation to geometric prediction. PointDiT is positioned at their intersection while avoiding latent-space information loss and architectural complexity.

  • Latent Diffusion Models: Latent diffusion models have become dominant for high-resolution image synthesis, motivating adaptations to geometric tasks such as GeometryCrafter and generative depth estimation.LDMs separate semantic content from perceptual detail in a compressed latent space.
  • Pixel-Space Diffusion Models: Early diffusion models operated in pixel space, while DiT replaced U-Net with a Vision Transformer and JiT later demonstrated direct pixel-space ViT training.These developments provide the broader precedent for pixel-space transformer diffusion.
  • Representation Learning and Generative Models: Representation-learning methods show that pretrained vision encoders can improve generative diffusion, and RAE replaces the VAE with pretrained representation autoencoders such as DINOv2.PointDiT follows this general connection between pretrained representations and generative models.
  • Monocular Depth Estimation: Traditional monocular depth estimation predicts scalar depth with CNNs or Transformers, but depth maps remain 2.5D representations requiring known camera information.The supplied passage contrasts scalar depth prediction with richer geometric representations.
  • Monocular Depth Estimation: PPD applies pixel-space diffusion to monocular depth estimation but uses v-prediction, which performs worse than x-prediction in the authors’ controlled comparisons.The comparison is reported in Table 3(a).
  • Monocular Point Map Estimation: Monocular point-map estimation predicts dense camera-coordinate xyz directly, enabling holistic 3D reconstruction without intrinsic calibration.Deterministic feed-forward models such as MoGe dominate this area and typically use complex hybrid architectures and intricate loss functions.

3. Approach

PointDiT formulates monocular dense point-map prediction as image-conditioned flow matching, transporting Gaussian noise to point-map data with a ViT-based pixel-space Diffusion Transformer. The method uses normalized point maps, frozen DINOv3 image tokens, clean point-map prediction, and primarily a flow-matching loss for end-to-end training.

  • Flow Matching: Flow matching learns an ODE that transforms a prior noise distribution into the dense point-map data distribution.A linear interpolation between noise and data induces constant-velocity, straight-line transport for each sample pair.
  • Image Conditioned Flow Matching: Image-conditioned flow matching predicts a conditional vector field that uses RGB-image spatial context to resolve geometric ambiguity.The conditioning image steers the flow toward the target dense point map.
  • Point Map Normalization: Point-map normalization matches variable coordinate scales to the Gaussian noise prior, producing affine-invariant predictions recovered up to unknown scale and shift.The model is trained in the normalized space rather than directly on scene-dependent coordinate ranges.
  • Diffusion Transformer: The ViT-based Diffusion Transformer patchifies noisy point maps, fuses them with frozen DINOv3 image tokens, and reconstructs the clean full-resolution point map.The network predicts clean point maps rather than velocities, while patchification reduces the sequence length from pixels to patches.
  • Training and Inference: The model is trained end-to-end mainly with flow-matching loss plus a single lightweight auxiliary term, and it supports competitive single-step inference with further gains from additional steps.Velocity-space optimization is used even though the network predicts the clean point map.

4. Experiments

Experiments train PointDiT entirely on synthetic data with a two-stage resolution curriculum and evaluate zero-shot generalization across seven real-world datasets. PointDiT-H achieves leading depth and point-map accuracy, while pixel-space generation improves boundary sharpness and supports robust single- and multi-step inference.

  • Training Setup: PointDiT uses synthetic-only training, pre-training at 256 × 256 and fine-tuning at 512 × 512 across a high-fidelity mixture of datasets.SceneNet-RGBD provides approximately 5.36 million photorealistic RGB-D samples for pre-training.
  • Evaluation Protocol: Zero-shot evaluation covers seven real-world datasets spanning indoor rooms and outdoor driving scenes, tested at both 256 × 256 and 512 × 512 resolutions.Inputs are standardized by rescaling each image so its shorter side matches the target resolution.
  • Inference Behavior: PointDiT-H maintains high-fidelity predictions with all-zeros input, and additional inference steps steadily improve BF1 while Rel and δ1 remain stable.The flow-matching formulation enables one network to support a variable number of inference steps.
  • Main Comparisons: PointDiT-H achieves the best depth accuracy and point-map δp 1, while PointDiT attains the highest boundary sharpness among evaluated methods.Evaluation aligns affine-invariant predictions using optimal scale and shift from a least-squares procedure.
  • Main Comparisons: 10.50 BF1 is achieved by PointDiT versus 9.41 for the best baseline, indicating sharper geometry; MoGe remains ahead on Relp, 4.21 vs. 4.40.PointDiT-L provides comparable boundary quality at lower cost, while PointDiT-B remains competitive with fewer parameters.
  • Ablations: 13.92 BF1 is obtained by the generative model versus 10.90 for the deterministic regressor, whose predictions are described as blurry and unable to recover thin structures and transparent objects.The experiments also identify x-prediction as crucial, report catastrophic failure for v-prediction, and improve quality by setting timestep 0 with 10% probability.

5. Conclusion · Appendix

PointDiT shows that a plain ViT can directly diffuse raw point maps conditioned on DINOv3 features, achieving accurate geometry with sharper boundaries while supporting single- and multi-step inference. The framework is limited by fixed training resolutions and weaker outdoor-scene performance, motivating mixed-resolution training and larger, more diverse datasets.

  • 5. Conclusion: PointDiT removes VAE and hybrid-network overhead by predicting monocular point maps directly in pixel space with a plain Vision Transformer conditioned on DINOv3 features.The model is trained directly on raw point maps rather than using point-map tokenizers.
  • 5. Conclusion: Patch size 16 recovers sharper boundaries and finer local structures than patch size 32 at 512 × 512 resolution.This comparison is reported in the patch-size ablation.
  • 5. Conclusion: The model produces accurate geometry with notably sharper boundaries and supports both single-step and multi-step inference.These properties follow from modeling dense geometry directly in pixel space.
  • 5. Conclusion: Pixel-space diffusion for dense geometry bridges standard image generation and 3D reconstruction without requiring a VAE.The authors identify this as a path toward end-to-end 3D and 4D generation using direct diffusion.
  • 5. Conclusion: The framework is currently trained only at fixed resolutions of 256 × 256 and 512 × 512, making mixed-resolution training a promising direction.Mixed-resolution training is proposed to generalize across image resolutions.
  • 5. Conclusion: Outdoor-scene performance still has room for improvement because the training data have relatively limited scale and diversity.The authors suggest scaling the dataset, particularly with more outdoor data.
  • 5. Conclusion: The authors acknowledge contributors for insightful discussions and support.The acknowledgements name Nando Metzger, Weirong Chen, Felix Wimbauer, Haiwen Huang, and other collaborators.

A. Experimental Details

PointDiT is trained on synthetic datasets with dense depth and known camera intrinsics, then evaluated zero-shot on unseen real-world benchmarks. Its two-stage training progresses from 256 × 256 pre-training on a large indoor dataset to 512 × 512 fine-tuning on diverse multi-dataset data.

  • Data and Evaluation: PointDiT trains on synthetic data with dense ground-truth depth and known camera intrinsics, while evaluation is zero-shot on unseen real-world benchmarks.Depth is back-projected through the intrinsics into per-pixel 3D point maps, which serve as prediction targets.
  • Training Procedure: Training uses 256 × 256 pre-training on a single large indoor dataset followed by 512 × 512 fine-tuning on diverse multi-dataset data.

A.1. Training Datasets

Training uses a two-stage curriculum: low-resolution pre-training on large-scale synthetic indoor data, followed by higher-resolution fine-tuning on a weighted mixture spanning broader geometric domains. Weighted sampling decouples the effective mixture from highly imbalanced dataset sizes.

  • Stage 1 (Pre-Training, 256 × 256): Stage 1 pre-training at 256 × 256 uses SceneNet-RGBD, a synthetic indoor dataset with ≈5.36M samples, to learn an image-to-point prior.Its scale and clean indoor geometry support inexpensive low-resolution pre-training.
  • Stage 2 (Fine-Tuning, 512 × 512): Stage 2 fine-tuning at 512 × 512 combines 11 synthetic datasets totaling ≈6.22M samples across indoor, outdoor ground-level, and aerial/diverse domains.This stage adds outdoor, large-scale, and high-detail geometry absent from Stage 1.
  • Weighted sampling: Each dataset d receives mixing weight w_d, and per-sample probability w_d/N_d, making its globally normalized sampling probability exactly w_d regardless of corpus size N_d.The datasets are combined through weighted sampling.
  • Weighted sampling: TartanGround contains 67.1% of all samples but is sampled 15% of the time, illustrating correction for raw corpus imbalance.The weighting scheme also accommodates small high-quality datasets such as Synscapes, which has 25k samples.

A.2. Evaluation Datasets

The evaluation measures zero-shot synthetic-to-real generalization across seven real-world depth benchmarks spanning indoor, outdoor, and mixed scenes. Three boundary-focused datasets additionally test performance on transparent or specular surfaces and sharp planar discontinuities.

  • Evaluation Datasets: Seven real-world depth benchmarks evaluate zero-shot synthetic-to-real generalization without overlap with either stage’s synthetic training datasets.The benchmarks span indoor, outdoor, and mixed scenes; full-set sample counts are reported in Table 5.
  • Evaluation Datasets: HAMMER, iBims-1, and Booster focus on transparent or specular surfaces and sharp planar discontinuities, with additional boundary-sharpness measurements.These three datasets are the challenging boundary-focused subsets of the evaluation suite.

A.3. Training Details

Training uses coordinated geometric and image-only photometric augmentation, followed by two-stage AdamW optimization in mixed precision. Stage 1 trains at 256×256, while Stage 2 fine-tunes at 512×512 from the Stage-1 checkpoint with interpolated positional embeddings.

  • Data Augmentation: Augmentations combine geometric transformations applied jointly to images and point maps with photometric transformations applied only to images.Geometric operations update camera intrinsics accordingly.
  • Data Augmentation: Stage 2 resizes images to height 512 with aspect-ratio preservation, then applies square random crops and horizontal flips with p=0.5.Stage 1 sets image height to 256; test-time cropping is centered, and flips negate the point X-coordinate.
  • Data Augmentation: Photometric augmentation applies color jitter to every sample and independently applies six appearance transformations with probability 0.2.The transformations are Gaussian blur, autocontrast, histogram equalization, channel permutation, JPEG compression, and grayscale conversion.
  • Optimization: Both stages use AdamW with β1=0.9, β2=0.95, zero weight decay, bf16 mixed precision, linear batch-size learning-rate scaling, and EMAs decaying at 0.9999 and 0.9996.No gradient clipping or accumulation is used, and the 0.9999 EMA supports all evaluations.
  • Optimization: Stage 1 trains at 256×256 with blr=5×10−5 for 30 epochs after a 5-epoch warmup, whereas Stage 2 uses blr=1×10−4 without warmup.Stage 2 is initialized from the Stage-1 checkpoint, with positional embeddings bicubically interpolated from a 16×16 to 32×32 grid.

B. More Evaluation Results · B.1. Per-Dataset Metrics

This section provides full per-dataset point-map, depth, and boundary evaluations, showing PointDiT-H’s best average depth performance and leadership on HAMMER, while revealing weaknesses on outdoor datasets.

  • B.1. Per-Dataset Metrics: The evaluation reports per-dataset point-map, depth, and boundary metrics, complementing the main paper’s averages over seven datasets.Boundary sharpness is reported only where ground truth supports boundary evaluation.
  • B.1. Per-Dataset Metrics: Depth results likewise cover all seven datasets at 512 × 512 resolution using sample-weighted averages over 3,444 samples.The depth metrics are presented separately for readability.
  • B.1. Per-Dataset Metrics: Boundary sharpness is evaluated only on HAMMER, iBims-1, and Booster.The reported boundary metric is BF1, with per-dataset sample counts provided separately.
  • B.1. Per-Dataset Metrics: PointDiT variants use 4 sampling steps, and the Avg column is the sample-weighted mean over evaluation samples.This averaging convention applies to the per-dataset tables.
  • B.1. Per-Dataset Metrics: PointDiT-H attains the best average depth boundaries, while PointDiT leads HAMMER across point-map, depth, and boundary metrics.The HAMMER result is identified as a notable per-dataset strength.
  • B.1. Per-Dataset Metrics: Point-map results cover all seven evaluation datasets at 512 × 512 resolution.The average is computed as a sample-weighted mean over 3,444 samples.
  • B.1. Per-Dataset Metrics: Across KITTI, DIODE, and ETH3D, PointDiT is inferior to stronger regression baselines such as MoGe and UniDepthV2.The gap is attributed mainly to limited outdoor-scene coverage in the synthetic training mixture, and adding outdoor training data is expected to narrow it.

B.2. More Visualizations

Additional depth visualizations show that PointDiT reconstructs thin structures and transparent objects more effectively while preserving more accurate relative scale across the global scene.

  • Depth comparisons: PointDiT significantly improves reconstruction of thin structures in the depth comparisons.This is illustrated in the first row of Figure 7.
  • Depth comparisons: PointDiT is significantly better at reconstructing transparent objects than the compared methods.The transparent-object comparisons appear in the second rows of Figure 7.
  • Depth comparisons: PointDiT maintains a more accurate relative scale across the global scene.Figure 7 presents this behavior in the third and fourth rows.
Loading 2607.02515v1…