PaperlayerView exact revision on arXiv

Source-linked AI summary

DiffusionInst: Diffusion Model for Instance Segmentation

Zhangxuan Gu, Haoxing Chen, Zhuoer Xu, Jun Lan, Changhua Meng, Weiqiang Wang

arXiv:2212.02773v3cs.CV

TL;DR

Instance segmentation requires pixel-level masks, and diffusion models' potential for visual understanding remains underexplored. DiffusionInst represents instances with filters and treats segmentation as noise-to-filter denoising; it reports competitive single-model results on COCO and LVIS. The authors identify limited gains from multi-step denoising and slower training and inference as limitations.

  • Problem

    Diffusion models have succeeded in image generation, but their potential for visual understanding remains underexplored, including instance segmentation.

    Source: §1 · p. 1

  • Method

    DiffusionInst represents instance masks with instance-aware filters and reconstructs global masks through a noise-to-filter denoising process.

    Source: §1 · p. 2

  • Results

    DiffusionInst achieves comparable single-model results on COCO and LVIS; on COCO validation, its ResNet-50 model reaches 37.3% AP.

    Source: §1 · p. 2, §5 · p. 7

  • Takeaways & Limitations

    The authors propose DiffusionInst as a baseline to inspire more efficient diffusion frameworks for discriminative tasks.

    Source: §5 · p. 7

    Four-step denoising improves AP by less than 1%, and training takes longer and inference runs slower than SOLO and Mask RCNN.

    Source: §3.4 · p. 5

Abstract

from arXiv · show

Diffusion frameworks have achieved comparable performance with previous state-of-the-art image generation models. Researchers are curious about its variants in discriminative tasks because of its powerful noise-to-image denoising pipeline. This paper proposes DiffusionInst, a novel framework that represents instances as instance-aware filters and formulates instance segmentation as a noise-to-filter denoising process. The model is trained to reverse the noisy groundtruth without any inductive bias from RPN. During inference, it takes a randomly generated filter as input and outputs mask in one-step or multi-step denoising. Extensive experimental results on COCO and LVIS show that DiffusionInst achieves competitive performance compared to existing instance segmentation models with various backbones, such as ResNet and Swin Transformers. We hope our work could serve as a strong baseline, which could inspire designing more efficient diffusion frameworks for challenging discriminative tasks. Our code is available in https://github.com/chenhaoxing/DiffusionInst.

1 Introduction

DiffusionInst formulates instance segmentation as noise-to-filter denoising, using instance-aware filters and a common mask feature to reconstruct global masks. It reports competitive results on COCO and LVIS, including strong COCO validation performance with ResNet-50 and Swin backbones.

1 Introduction

  • Instance segmentation predicts binary masks, while two-stage methods crop RoI features and single-stage methods use dense heads requiring NMS at inference.The paper presents these as the two standard approach groups.Source: §1 · p. 1
  • Query-based methods use random queries and one decoder pass, while diffusion models can progressively denoise over multiple inference steps.The paper treats query-based approaches as a special case of diffusion models.Source: §1 · p. 1
  • DiffusionInst represents masks with instance-aware filters and a common mask feature, formulating segmentation as noise-to-filter denoising.The framework adds noisy filters alongside bounding boxes and uses a mask branch for multi-scale global mask reconstruction.Source: §1 · p. 2
  • Randomly selected diffusion times expose filters to different distribution noises, which the authors say increases learning difficulty and contributes to robustness and performance.The paper describes T denoising steps as potentially corresponding to T different distribution noises.Source: §1 · p. 2
  • 37.3% AP with ResNet-50 and 47.8% AP with Swin Transformers on COCO validation; the latter exceeds QueryInst at 44.6% AP.The paper also reports similar conclusions on COCO test-dev and LVIS.Source: §1 · p. 2

2 Related Works

Related work spans instance-segmentation approaches, diffusion models, and their emerging use in visual understanding. DiffusionInst is presented as the first instance-segmentation framework to formulate the task as a generative denoising process.

2.1 Instance Segmentation

  • DiffusionInst formulates instance segmentation as a generative denoising process, unlike prior top-down, bottom-up, direct, and query-based approaches.The paper describes it as the first such formulation, with competitive performance.Source: §2.1 · p. 2

2.2 Diffusion Model

  • Diffusion models are parameterized Markov chains that reconstruct data samples from random distributions through gradual denoising.The paper notes their use across vision, language, robust learning, and temporal data modeling.Source: §2.2 · p. 2

2.3 Diffusion Model for Visual Understanding.

  • Prior diffusion work addressed image synthesis, panoptic masks, and object detection; DiffusionInst applies diffusion to instance segmentation.The authors describe their work as the first use of a diffusion model for instance segmentation.Source: §2.3 · p. 2

3 Methodology

DiffusionInst casts instance segmentation as denoising instance-aware filters, using multi-scale mask features to reconstruct masks. Its method covers diffusion preliminaries, filter-based mask representation, training and inference, and remaining limitations.

3.1 Preliminaries

  • Diffusion models add scheduled Gaussian noise to data during a forward process and learn to reverse it, while DiffusionDet applies this setup to padded sets of ground-truth boxes.Inference begins from random noise and iteratively applies the reverse chain; DiffusionDet uses DDIM to refine boxes between sampling steps.Source: §3.1 · pp. 2–3

3.2 Mask Representation

  • Instance masks are generated by applying an instance-specific filter to fused multi-scale mask features, supporting global mask prediction rather than cropped RoI features.The mask head uses three 1 × 1 convolutional layers whose weights are assigned by the filter; the paper motivates global masks by the need for larger receptive fields to capture instance edges.Source: §3.2 · pp. 3–4
  • DiffusionInst achieves competitive COCO instance-segmentation performance, especially with large backbones such as Swin-B and Swin-L.The table caption reports AP and inference FPS, with models trained only on COCO training data.Source: Table 1 · p. 4

3.3 DiffusionInst

  • The architecture extracts image features once, then iteratively refines noisy boxes and associated filters before reconstructing masks from the denoised filters and mask features.The mask branch fuses FPN scales into Fmask; inference starts from Gaussian-sampled boxes and progressively refines predictions.Source: §3.3 · pp. 4–5
  • Training perturbs ground-truth boxes and generates noisy filters from noisy-box features, then reverses the process using detection and Dice mask losses.The loss balances the two terms with λ = 5 and includes mask-loss supervision at multiple decoder stages.Source: §3.3 · p. 4

3.4 Discussions

  • DiffusionInst’s noise-to-filter training still depends on ground-truth boxes because obtaining ground-truth filters is difficult.The authors identify direct training without box-based objectives as future work.Source: §3.4 · p. 5
  • Four-step denoising improves AP by less than 1%, and discriminative diffusion needs accurate instance contexts whose backbone features and receptive-field designs remain open questions.The authors propose exploring sampling strategies beyond DDIM for more effective multi-step denoising.Source: §3.4 · p. 5
  • DiffusionInst takes more training epochs to reach satisfactory performance than SOLO and Mask RCNN, and its inference is slower than theirs.The authors identify faster training and more efficient denoising as unresolved design goals.Source: §3.4 · p. 5

4 Experiments

Experiments evaluate DiffusionInst on COCO and LVIS across dataset settings, backbones, and architecture variants. Results show strong benchmark performance alongside slower convergence and inference, limited multi-step gains, and missed small or occluded instances.

4.1 Datasets

  • Experiments use COCO train2017, val2017, and test-dev, plus LVIS v1.0, a 1,203-category long-tail benchmark with rare, common, and frequent groups.LVIS has more than two million high-quality instance-mask annotations; the groups are reported as APr, APc, and APf.Source: §4.1 · p. 5

4.2 Implement Details

  • The implementation uses ResNet and Swin backbones with FPN, and trains a ResNet-50 model for about 26 hours on eight A100 GPUs.Swin backbones are pretrained on ImageNet-22k; the stated augmentations are horizontal flips, scale jitter, and random crops, without MixUp or Mosaic.Source: §4.2 · p. 5

4.3 Comparison with State-of-the-art

  • Stronger backbones bring larger gains; RPN removal slows convergence, multi-step denoising usually adds less than 0.3% AP, and inference is slower.DiffusionInst improves by about 9% AP when changing from ResNet-50 to Swin-Base; RPN-free training usually needs a 3x or 5x schedule. On a single A100, it runs at 15 FPS.Source: §4.3 · p. 6

4.5 Visualizations

  • DiffusionInst performs better on large instances but sometimes misses small ones; visualizations also show missed small-region persons and misclassified occluded persons and books.The visualization uses a ResNet-50 model with one-step denoising on COCO validation.Source: §4.3 · p. 6, §4.5 · p. 7, Figure 3 · p. 7

4.3 Comparison with State-of-the-art

  • 48.3% AP with Swin-Large on COCO test-dev; on LVIS, DiffusionInst is best with Swin-Base and widens its advantage over Mask R-CNN.On COCO test-dev, ResNet-50 reaches 37.1% AP, below all five baselines; Swin-Large outperforms SOLQ.Source: Table 2 · p. 5, §4.3 · p. 6, Table 3 · p. 6

4.4 Ablation Studies

  • With ResNet-50, a 5x schedule, and one-step denoising, λ=5 with multi-stage mask-loss supervision adds 2.2% AP; eight mask channels and three mask-head convolutions each reach 37.3% AP.The study selects 500 predefined filters as a balance of performance and model complexity.Source: §4.4 · pp. 6–7, Table 4 · p. 6

5 Conclusion

DiffusionInst frames instance segmentation as a noise-to-filter process and achieves comparable single-model results on COCO and LVIS. The authors identify limits in small-instance performance, multi-step denoising gains, training time, and inference speed.

5 Conclusion

  • DiffusionInst frames instance segmentation as a noise-to-filter process and reports comparable single-model results with a Swin-Large backbone on COCO and LVIS.The authors describe it as a possible solution and hope it will serve as a baseline for more efficient frameworks and rethinking learning targets.Source: §5 · p. 7
  • The authors identify reliance on bounding boxes, slightly poor small-instance performance, unsatisfactory gains from multi-step denoising, longer training, and slower inference as limitations.Source: §5 · p. 7
  • The paper presents diffusion as a possible solution for instance segmentation and proposes the work as a baseline for future framework design.Source: §5 · p. 7

Related papers

Loading 2212.02773v3…