PaperlayerView exact revision on arXiv

Source-linked AI summary

DiffusionDet: Diffusion Model for Object Detection

Shoufa Chen, Peize Sun, Yibing Song, Ping Luo

arXiv:2211.09788v2cs.CV

TL;DR

Object detection traditionally relies on designed priors or fixed learnable queries, leaving open whether detection can begin from simpler candidates. DiffusionDet learns to denoise noisy boxes into object boxes and reuses one trained network across inference settings. It achieves favorable performance on standard benchmarks, including 45.8 AP on COCO and improved zero-shot transfer to crowded CrowdHuman scenes.

  • Problem

    Existing detectors depend on empirical object priors or fixed learnable queries, motivating a simpler detection approach without learnable-query surrogates.

  • Method

    DiffusionDet trains a decoder to recover clean ground-truth boxes from Gaussian-perturbed boxes and progressively denoises randomly sampled boxes at inference.

  • Results

    DiffusionDet achieves favorable performance against established detectors, including 45.8 AP on COCO with one sampling step and 300 random boxes.

  • Takeaways & Limitations

    Dynamic box counts and iterative evaluation let one trained network support flexible inference without retraining.

  • Takeaways & Limitations

    DiffusionDet may suffer from relatively slow sampling speed because it uses the fundamental DDIM diffusion strategy.

Abstract

from arXiv · show

We propose DiffusionDet, a new framework that formulates object detection as a denoising diffusion process from noisy boxes to object boxes. During the training stage, object boxes diffuse from ground-truth boxes to random distribution, and the model learns to reverse this noising process. In inference, the model refines a set of randomly generated boxes to the output results in a progressive way. Our work possesses an appealing property of flexibility, which enables the dynamic number of boxes and iterative evaluation. The extensive experiments on the standard benchmarks show that DiffusionDet achieves favorable performance compared to previous well-established detectors. For example, DiffusionDet achieves 5.3 AP and 4.8 AP gains when evaluated with more boxes and iteration steps, under a zero-shot transfer setting from COCO to CrowdHuman. Our code is available at https://github.com/ShoufaChen/DiffusionDet.

1. Introduction

DiffusionDet reformulates object detection as denoising from random boxes to object boxes, avoiding fixed learnable queries. Its flexible inference supports changing box counts and iteration steps, with favorable benchmark and zero-shot transfer results.

  • Object detection predicts bounding boxes and category labels, supporting tasks including instance segmentation, pose estimation, tracking, and visual relationship detection.
  • Modern detectors use empirical object priors or fixed learnable queries, motivating alternatives that avoid surrogate learnable queries.
  • DiffusionDet directly detects objects from random boxes and progressively refines their positions and sizes toward target objects.
  • During training, Gaussian noise perturbs ground-truth boxes, while the decoder learns to predict clean boxes from noisy RoI features.
  • A single trained network supports arbitrary evaluation box counts and repeated denoising iterations, enabling flexible speed-accuracy trade-offs without retraining.
  • On zero-shot COCO-to-CrowdHuman transfer, adjusting evaluation boxes and iteration steps yields significant gains in crowded scenes, unlike marginal or degraded gains from prior methods.
  • 45.8 AP with one sampling step and 300 random boxes on COCO exceeds Faster R-CNN and DETR, while increasing sampling steps and boxes reaches 46.8 AP.

2. Related Work

Prior object detectors rely on empirical priors or fixed learnable queries, while diffusion models have mainly demonstrated success in generation and segmentation. DiffusionDet extends diffusion modeling to object detection by progressively denoising random boxes.

  • Object detection: Most object detectors regress boxes and classify categories using proposals, anchors, points, or other empirical object priors.
  • Object detection: DETR introduced a query-based paradigm using a fixed set of learnable queries, which subsequently attracted many extensions.
  • Diffusion model: Diffusion models generate data by starting from random distributions and gradually recovering samples through denoising.
  • Diffusion model for perception tasks: Diffusion models have been widely successful in image generation, while their potential for discriminative perception tasks remains less explored.
  • Diffusion model for perception tasks: DiffusionDet applies diffusion to detection by perturbing proposal boxes during training and sampling them from a Gaussian distribution during evaluation.

3. Approach

DiffusionDet trains a detector to reverse corruption from ground-truth boxes to noise, then iteratively denoises random boxes into object predictions. Its image encoder runs once while the decoder progressively refines boxes, supporting flexible sampling and evaluation settings.

  • 3.1. Preliminaries: DiffusionDet models detection as recovering clean bounding boxes from noisy boxes with a neural network conditioned on the image.Training adds Gaussian noise to ground-truth boxes and trains the network to predict the original boxes.
  • 3.2. Architecture: The image encoder extracts features once, allowing the detection decoder to refine box predictions without repeatedly processing the raw image.The decoder conditions on deep image features and uses proposal boxes to crop RoI features.
  • 3.3. Training: During training, ground-truth boxes are padded, corrupted with Gaussian noise, and used to train set-based classification and box-regression predictions.The training procedure constructs the forward diffusion process and learns its reverse using set prediction loss.
  • 3.4. Inference: At inference, DiffusionDet starts from Gaussian random boxes and progressively refines them through denoising steps, with DDIM estimating boxes for the next step.Predicted boxes or updated boxes from the previous step are passed into the decoder at each iteration.
  • 3.4. Inference: Box renewal replaces low-score predictions with new Gaussian random boxes so inference better matches the training corruption distribution.Remaining desired predictions are concatenated with newly sampled random boxes.
  • 3.4. Inference: DiffusionDet can use arbitrary numbers of random boxes and inference iterations, which need not match the numbers used during training.This flexibility follows from the random-box design and contrasts with methods requiring the same processed-box count during training and evaluation.

4. Experiments

Experiments show that DiffusionDet supports flexible inference settings, competitive detection performance, and gains from additional boxes and iterative refinement. It also exposes limitations in current benchmark performance and diffusion sampling speed.

  • Main Properties: DiffusionDet decouples training and inference box counts, allowing one trained model to vary evaluation boxes and iteration steps without retraining.This flexibility supports different speed–accuracy trade-offs across scenarios.
  • Main Properties: 38.4 AP is achieved by DETR with cloned queries at 4000 queries, down from 38.8 AP at 300 queries.Concatenating randomly initialized queries performs substantially worse, reaching 26.4 AP at 4000 queries, 12.4 below the peak.
  • Main Properties: DiffusionDet improves consistently with more iterative steps for 100, 300, and 500 random boxes, with larger refinement gains for fewer boxes.For 100 random boxes, AP increases from 41.9 with one step as iterations increase.
  • Benchmarking on Detection Datasets: 2.1 AP is gained on LVIS through iterative evaluation, compared with a 0.8 AP gain on COCO.Performance rises from 29.4 to 31.5 AP on LVIS and from 45.8 to 46.6 AP on COCO.
  • Ablation Study: DiffusionDet gains 0.7 AP on COCO and 3.1 AP on CrowdHuman when iteration steps increase, whereas DETR and Sparse R-CNN do not gain from additional iterations.Sparse R-CNN can degrade even when expanded to 12 stages.

5. Conclusion

DiffusionDet formulates object detection as a denoising diffusion process from noisy boxes to object boxes. Its noise-to-box pipeline supports flexible evaluation through dynamic box counts and iterative evaluation without retraining.

  • DiffusionDet views object detection as a denoising diffusion process from noisy boxes to object boxes.
  • The pipeline supports a dynamic number of boxes and iterative evaluation.
  • The same network parameters can support flexible evaluation without re-training the model.

A. Formulation of Diffusion Model

The diffusion formulation defines a forward process that progressively adds Gaussian noise to data, then models reverse denoising transitions with a neural network. The noise schedule uses β_t and ᾱ_t, while direct sampling permits arbitrary timesteps.

  • The forward Markovian process generates z_1 through z_T by gradually adding Gaussian noise to z_0.
  • The noise schedule is controlled by β_t, with α_t defined as 1 − β_t and ᾱ_t formed from the cumulative product of α_s.
  • The model can directly sample z_t at an arbitrary timestep without repeatedly applying the forward process.
  • The posterior q(z_{t−1}|z_t, z_0) is Gaussian with mean μ̃ and variance β̃_t I.
  • The reverse distribution p_θ(z_{t−1}|z_t) is approximated by a neural network predicting a mean and diagonal covariance matrix.
  • Diffusion models can generate z_0 by sampling from q(z_T) and running reverse steps until the data state.

B. Additional Experiments

This section adds experiments for more detailed analysis of the proposed method. It supplements the main experimental evaluation with additional evidence.

  • The section provides additional experiments beyond the main evaluation.
  • These experiments are intended to support more detailed analysis.
  • The passage introduces the section’s purpose without reporting a specific experimental result.

B.1. Dynamic Number of Boxes

DiffusionDet’s dynamic-box property is evaluated against prior detectors under varying evaluation box counts. Unlike those methods, DiffusionDet is designed to tolerate mismatches between training and evaluation settings.

  • The comparison evaluates DiffusionDet against Deformable DETR and Sparse R-CNN for dynamic box behavior.
  • Deformable DETR and Sparse R-CNN are adapted for evaluation settings involving different box counts through cloning or concatenating boxes and queries.
  • 9.6 AP is reported for Deformable DETR at N_eval = 4000, far below its 44.5 AP peak.
  • Sparse R-CNN declines more slowly than Deformable DETR but performs unsatisfactorily when N_eval differs from N_train.
  • The findings support DiffusionDet’s distinctive dynamic number of boxes property.

B.2. Iterative Evaluation

DiffusionDet supports iterative evaluation by progressively refining detections across steps. Unlike prior query-based methods, it can improve performance as refinement steps increase.

  • All compared models use six cascading decoder stages, with outputs from six previous stages fed into the next six stages.
  • Multiple-step evaluation either uses only the last step or ensembles outputs from multiple steps with NMS removing duplicates.
  • DiffusionDet gains performance with more refinement steps, whereas previous query-based methods still experience performance drops.Without ensembling, DiffusionDet and DETR show negligible degradation; ensembling generally mitigates degradation except for DETR.
  • Figure 6 compares iterative evaluation for DETR, Deformable DETR, and Sparse R-CNN against DiffusionDet.

C. Experimental Settings

The experiments specify reproduction and benchmark settings for DETR, COCO, and LVIS, with detailed configurations provided in accompanying tables.

  • The experimental settings are detailed in Sections 4.2 and 4.3.
  • DETR is reproduced with 300 object queries because its official repository provides a 100-query model.The reproduced model uses the official code and a Detectron2-based configuration.
  • Table 9 summarizes the DETR reproduction setting.
  • Table 10 specifies the COCO setting, while Table 11 specifies the LVIS setting.

C.2. Benchmark on COCO and LVIS

DiffusionDet is benchmarked on the COCO and LVIS datasets, using the configurations listed in Tables 10 and 11.

  • DiffusionDet is benchmarked on COCO and LVIS with training configurations provided in Tables 10 and 11, respectively.

D. Training Loss

DiffusionDet uses set prediction loss with matching costs that combine classification, L1 box, and generalized IoU components. Optimal transport assigns multiple predictions to each ground truth for training.

  • Set prediction loss is applied to DiffusionDet’s Ntrain predictions using pairwise matching costs based on class and box predictions.
  • The matching cost combines focal classification loss, L1 loss, and generalized IoU loss with weights λcls, λL1, and λgiou.
  • The loss weights are λcls = 2.0, λL1 = 5.0, and λgiou = 2.0.
  • Optimal transport assigns each ground truth’s top-k lowest-cost predictions as positives and the remaining predictions as negatives.
  • The total training loss combines classification, L1, and generalized IoU losses with the same component weights as the matching cost.The losses are computed only on matched prediction-ground-truth pairs.
Loading 2211.09788v2…