PaperlayerView exact revision on arXiv

Source-linked AI summary

DDP: Diffusion Model for Dense Visual Prediction

Yuanfeng Ji, Zhe Chen, Enze Xie, Lanqing Hong, Xihui Liu, Zhaoqiang Liu, Tong Lu, Zhenguo Li, Ping Luo

arXiv:2303.17559v2cs.CV

TL;DR

Dense prediction needs a broadly applicable alternative to single-step discriminative methods and cumbersome task-specific diffusion designs. DDP formulates prediction as image-conditioned denoising from noise to maps, with efficient decoupling and iterative inference. It achieves state-of-the-art or competitive performance across three tasks and six benchmarks, while supporting dynamic inference and uncertainty awareness.

  • Problem

    Existing diffusion frameworks for dense prediction can be inefficient, slow to converge, and sub-optimal, while prior approaches often require cumbersome task-specific designs.

  • Method

    DDP uses conditional diffusion to denoise encoded prediction maps under image guidance, decoupling a once-run image encoder from a lightweight iterative map decoder.

  • Results

    DDP achieves state-of-the-art or competitive performance across semantic segmentation, BEV map segmentation, and depth estimation on six benchmarks, including 83.9 mIoU on Cityscapes.

  • Takeaways & Limitations

    DDP provides a general dense-prediction baseline with dynamic inference and natural prediction-uncertainty awareness.

  • Takeaways & Limitations

    Multi-step inference adds non-negligible computational cost, and DDP’s efficacy in other domains remains to be determined.

Abstract

from arXiv · show

We propose a simple, efficient, yet powerful framework for dense visual predictions based on the conditional diffusion pipeline. Our approach follows a "noise-to-map" generative paradigm for prediction by progressively removing noise from a random Gaussian distribution, guided by the image. The method, called DDP, efficiently extends the denoising diffusion process into the modern perception pipeline. Without task-specific design and architecture customization, DDP is easy to generalize to most dense prediction tasks, e.g., semantic segmentation and depth estimation. In addition, DDP shows attractive properties such as dynamic inference and uncertainty awareness, in contrast to previous single-step discriminative methods. We show top results on three representative tasks with six diverse benchmarks, without tricks, DDP achieves state-of-the-art or competitive performance on each task compared to the specialist counterparts. For example, semantic segmentation (83.9 mIoU on Cityscapes), BEV map segmentation (70.6 mIoU on nuScenes), and depth estimation (0.05 REL on KITTI). We hope that our approach will serve as a solid baseline and facilitate future research

1. Introduction

DDP formulates dense visual prediction as conditional denoising, using image guidance to iteratively transform noise into predictions while supporting efficient, dynamic, and uncertainty-aware inference. Across three tasks and six benchmarks, it achieves competitive or state-of-the-art results.

  • Dense prediction requires assigning discrete labels or continuous values to every image pixel for detailed scene understanding.
  • DDP extends conditional diffusion into dense prediction by learning to reverse noise added to encoded ground-truth maps under image-derived guidance.Training constructs noisy maps from encoded ground truth; inference reverses the learned diffusion process.
  • The decoupled architecture runs the image encoder once and performs iterative diffusion only in a lightweight map decoder.This design reduces the repeated computation required when diffusion models process the raw image at every step.
  • DDP supports dynamic inference and prediction-uncertainty awareness through iterative sampling with shared parameters.Multiple sampling steps trade computational cost against prediction quality.
  • 83.9 mIoU on Cityscapes, 70.6 mIoU on nuScenes, and 0.05 REL on KITTI demonstrate strong performance across representative dense prediction tasks.Experiments cover semantic segmentation, BEV map segmentation, and depth estimation across six diverse benchmarks.

2. Related Work

Diffusion and score-based models have broadened generative modeling, while dense prediction research has advanced through feature aggregation, stronger backbones, and decoder designs. DDP explores generative modeling for pixel-level prediction.

  • Diffusion and score-based models generate samples by reversing a process that gradually transforms data into noise.Their conceptual simplicity has enabled effective training and strong results across multiple generative modalities.
  • Dense prediction methods commonly advance through multi-scale feature aggregation, high-capacity backbones, and powerful decoder heads.These approaches target pixel-by-pixel classification or regression for tasks such as segmentation and depth estimation.

3. Methodology

DDP casts dense prediction as conditional denoising: noisy maps are progressively refined using image features, with a lightweight decoder separating repeated diffusion from one-time image encoding. Its training and sampling choices address label representation, task supervision, sampling drift, and compute–quality flexibility.

  • 3.1. Preliminaries: DDP trains a conditional denoising process that predicts clean maps from noisy maps guided by image features.The ground-truth map is progressively noised during training, and the model learns to reverse that process.
  • 3.2. Architecture: The image encoder runs once, while a lightweight map decoder performs the repeated diffusion steps, reducing inference overhead.The decoder receives noisy maps together with encoded image features and reuses shared parameters across iterations.
  • 3.2. Architecture: DDP supports modern perception backbones by producing multi-scale image features and decoding predictions from concatenated noisy maps and conditional features.The image encoder aggregates four resolutions, while the map decoder uses six deformable-attention layers for pixel-wise classification or regression.
  • 3.3. Training: Class embedding works best among the evaluated label encodings for discrete labels, while the cosine noise schedule usually performs best across benchmark tasks.The encoding strategies are normalized and scaled within [−scale, +scale], and scale controls the signal-to-noise ratio.
  • 3.3. Training: Task-specific supervision improves training over standard l2 loss, using cross-entropy for semantic segmentation and sigloss for depth estimation.The paper selects the loss according to the prediction task.
  • 3.4. Inference: Self-aligned denoising addresses sampling drift by constructing noisy maps from model predictions during the final 5K training iterations.This aligns training and testing data distributions and tends to replace performance degradation with saturation as sampling steps increase.

4. Experiment

DDP is evaluated across semantic segmentation, BEV map segmentation, and depth estimation, with ablations examining its inference properties and design choices. Across these settings, multi-step denoising generally improves performance while retaining practical efficiency, although segmentation remains slightly below Mask2Former.

  • Evaluation Setup: DDP is evaluated on semantic segmentation, BEV map segmentation, and depth estimation across six datasets.The benchmarks are ADE20K, Cityscapes, nuScenes, KITTI, NYU-DepthV2, and SUN RGB-D.
  • Main Properties: 82.60 mIoU is achieved by ConvNeXt-T DDP with three sampling steps, versus 82.33 mIoU with one step.Increasing sampling steps provides a controllable accuracy–computation trade-off without retraining the network, and stochastic sampling also yields pixel-wise uncertainty maps.
  • Semantic Segmentation: 46.1 mIoU on ADE20K with one step rises to 47.0 mIoU with three steps, while DDP surpasses the non-diffusion baseline by 1.2 points at one step.With Swin-L, three-step DDP reaches 53.2 mIoU, 1.1 points above UperNet at comparable FLOPs.
  • BEV Map Segmentation: 70.6 mIoU is reached on nuScenes multi-modality BEV segmentation after iterative denoising, compared with 70.3 mIoU for one-step DDP.In the camera-only setting, DDP reaches 59.3 mIoU with one step and 59.4 mIoU with three steps.
  • Depth Estimation: 0.05 REL is reported on KITTI with the Swin-L backbone, while DDP outperforms competitors with clear margins in most reported depth metrics.The comparison includes DepthFormer and DepthGen on a benchmark described as tending toward saturation.
  • Ablation Study: The map decoder uses six blocks and 8.4M parameters, compared with 41.5M for K-Net and 31.5M for UperNet.Ablations find class embedding preferable for discrete labels, a 0.01 scaling factor best, and cosine scheduling better than linear scheduling at 47.0 versus 45.1 mIoU.

5. Conclusion

DDP provides a general conditional-diffusion framework for dense visual prediction and achieves state-of-the-art or competitive results across three tasks and six benchmarks. Its main scope boundaries are extra multi-step inference cost and unverified efficacy beyond the evaluated domains.

  • DDP extends conditional diffusion into modern perception pipelines without architectural customization or task-specific design.
  • DDP achieves state-of-the-art or competitive performance on three representative dense prediction tasks across six diverse benchmarks.
  • Multi-step inference incurs non-negligible additional computational cost.
  • DDP’s efficacy in domains beyond the demonstrated dense visual prediction benchmarks requires further research.

A.1. Algorithm details

The supplementary algorithm details describe self-aligned denoising, which addresses sampling drift by illustrating the gap between training and inference denoising targets.

  • A.1. Algorithm details: Self-aligned denoising is used during the last 5K training iterations to address sampling drift.The procedure is presented as a supplement to the main-paper algorithms.
  • A.1. Algorithm details: Figure 4 illustrates the gap between the denoising targets used during training and inference.

A.2. More Discussions

DDP reaches an early performance saturation in perceptual tasks, while multi-step sampling supports computation–quality trade-offs and pixel-wise uncertainty estimation. The supplementary implementation uses self-aligned denoising to address sampling drift.

  • A.2. More Discussions: Perceptual diffusion models usually reach a saturation point within 3–5 sampling steps, making additional diffusion less advantageous.The paper contrasts this with image generation, where 10–50 iterations are often needed.
  • A.2. More Discussions: The self-aligned denoising procedure corrupts an encoded predicted map before the decoder predicts the map and computes the training loss.
  • A.2. More Discussions: Self-aligned denoising addresses sampling drift by aligning training and inference denoising targets.
  • A.2. More Discussions: DDP can achieve high accuracy in perception tasks with minimal computational cost.

B.1. Semantic Segmentation

This section documents the datasets and implementation settings used for semantic segmentation and cross-dataset depth evaluation. The supplied passages emphasize dataset composition and training/testing procedures rather than additional segmentation results.

  • B.1. Semantic Segmentation: ADE20K experiments use ImageNet-pretrained backbones, AdamW, a 6×10^-5 initial learning rate, polynomial decay, and 512×512 random resized crops.
  • B.1. Semantic Segmentation: Cityscapes contains 5,000 high-resolution images from 50 German cities, with 2,975 training, 500 validation, and 1,525 testing samples.
  • B.1. Semantic Segmentation: SUN RGB-D evaluation reports models trained on NYU-DepthV2 and tested without fine-tuning.
  • B.1. Semantic Segmentation: Cityscapes training uses random resizing and 512×1024 crops, while testing uses original 1024×2048 images.

B.2. BEV Map Segmentation

DDP is evaluated for BEV map segmentation on nuScenes, a large-scale multimodal benchmark with synchronized camera, LiDAR, and radar data.

  • nuScenes contains 700/150/150 scenes for training, validation, and testing.
  • The benchmark combines six cameras, one LiDAR, and five radars for multimodal scene perception.
  • Camera views are resized to 256×704, while point clouds are voxelized at 0.1m resolution.

B.3. Depth Estimation

DDP depth estimation is evaluated on outdoor KITTI and indoor NYU-DepthV2, with NYU performance reported using three diffusion steps.

  • KITTI: KITTI covers monocular depth estimation over a 0–80m depth range with sparse ground-truth maps.
  • KITTI: The KITTI setup follows the standard Eigen split, using approximately 26K training images and 697 test frames.
  • NYU-DepthV2: Table 7 reports NYU-DepthV2 validation performance for DDP with 3 diffusion steps, where lower or higher values depend on the metric.
  • NYU-DepthV2: NYU-DepthV2 contains over 1,449 aligned indoor RGB-depth pairs from 464 indoor areas.

C. Experimental Results

Across visualizations and controlled comparisons, DDP exhibits multi-step refinement, supports conditional generation, and applies its segmentation predictions as inputs to image synthesis.

  • Multiple inference: More sampling steps continuously improve DDP segmentation performance and produce smoother maps on Cityscapes and ADE20K.
  • ControlNet comparison: Combining DDP with ControlNet tests whether its pixel clustering property benefits segmentation-mask-conditioned image generation.
  • Qualitative results: Figures 5 and 6 visualize multiple inference on Cityscapes and ADE20K validation sets, while Figures 7–9 show BEV and depth predictions.
  • ControlNet comparison: Figure 10 compares Control Stable Diffusion, Uniformer-UnperNet, and DDP segmentation models using predicted segmentation maps as condition inputs.
Loading 2303.17559v2…