PaperlayerView exact revision on arXiv

Source-linked AI summary

Diffuse2Seg: Diffusion Models Can Segment Anything Without Supervision

Christoph Hümmer, Joachim Sicking, Fabian Hüger, Hanno Gottschalk

arXiv:2609.06491v1cs.CV

TL;DR

Dense labels for open-world entity segmentation are costly to collect. Diffuse2Seg generates multi-granular instance masks from pretrained diffusion self-attention without retraining, and its labels outperform prior unsupervised generators across five domains. Models trained on those labels also provide an initialization for semi-supervised learning, though small objects remain difficult when latent resolution cannot represent them.

  • Problem

    Densely annotating open-world entities across things, stuff, and parts is costly and time-consuming, limiting the scalability of this training approach.

    Source: §1 · p. 1

  • Method

    Diffuse2Seg propagates a grid of point prompts through pretrained diffusion self-attention to generate class-agnostic, multi-granular instance labels without retraining.

    Source: §1 · p. 2

  • Results

    +4.3–7.1 p.p. in AR1000 over UnSAM for generated labels across five domains.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Diffuse2Seg models can serve as an effective initialization for semi-supervised learning and can surpass a fully supervised reference with significantly fewer labeled images.

    Source: §5 · p. 14

    Pseudo-labels struggle with particularly small objects that cannot be recognized at the latent resolution.

    Source: §5 · p. 14

Abstract

from arXiv · show

Open-world entity segmentation aims to predict masks for arbitrary objects across domains and at multiple granularities, from parts to whole objects. In this setting, SAM sets a strong standard: trained on SA-1B, comprising 11M images and over 1B carefully annotated masks, it achieves remarkable zero-shot performance. Collecting such labels is expensive and time-consuming, however, which limits how far this recipe can scale. Text-to-image diffusion models offer a way around this. Their intermediate features transfer well across perception tasks, and since object structure emerges as the model denoises a noise sample into an image conditioned on a text prompt, that structure is already encoded in these representations. They can therefore be exploited for open-world entity segmentation without retraining or supervision. Building on this observation, we introduce Diffuse2Seg, which repurposes generative diffusion models for automatic mask generation by propagating a grid of point prompts through their self-attention representations in an edge-preserving manner. Diffuse2Seg produces multi-granular instance masks and outperforms prior state-of-the-art label generators by 4.3-7.1 p.p. in AR_1000 across five domains. Training an instance segmentation model on these generated masks advances detector-free open-world segmentation by 7.4 and 7.7 p.p. on "things" and "stuff+things" datasets and surpasses the detector-based UnSAM on "stuff+things" by 2.1 p.p. in AR_1000. Finally, we show that a model trained on Diffuse2Seg labels provides a strong initialization for semi-supervised learning, outperforming its fully supervised counterpart with already 5k labeled images.

1 Introduction

Diffuse2Seg uses pretrained diffusion self-attention to generate multi-granular instance labels without supervision, addressing the cost of densely annotating open-world entities. The authors report gains in label generation, trained segmentation, and semi-supervised fine-tuning.

1 Introduction

  • Dense open-world entity labels span things, stuff, and parts, but annotating them is costly and time-consuming.The introduction identifies this as a scalability challenge for open-world segmentation.Source: §1 · p. 1
  • Diffuse2Seg propagates a grid of point prompts through pretrained diffusion self-attention to produce class-agnostic, multi-granular instance labels without retraining.The method turns the resulting candidates into masks at the original image resolution.Source: §1 · p. 2
  • +4.3–7.1 p.p. in AR1000 over UnSAM across five domains for generated labels.Source: §1 · p. 3
  • +7.7/+7.4/+10.0 p.p. AR over SOHES on stuff+things/things/parts datasets, using half the training image budget and a smaller backbone.These models are trained solely on Diffuse2Seg pseudo-labels.Source: §1 · p. 3
  • +2.1 p.p. AR over UnSAM on stuff+things datasets in a detector-augmented setting.Source: §1 · p. 3
  • At 100 fine-tuning images, the semi-supervised model recovers 93.5% of a fully supervised reference and surpasses it by 1.4 p.p. at 10k images.Source: §1 · p. 3

2 Related Work

Related work draws on transferable features from self-supervised and generative vision models, while prior unsupervised segmentation methods differ in their granularity and use of prompts or detectors. Diffuse2Seg is positioned among methods that use internal diffusion features for class-agnostic segmentation.

Vision Foundation Models

  • SSL and generative vision models provide transferable features; Diffuse2Seg uses diffusion self-attention for generalization and spatial grouping.The paper mainly uses SD2, a UNet-based model; it notes that stronger downsampling in diffusion transformers limits small-instance discovery.Source: §2 · p. 3

Unsupervised Instance Segmentation

  • SSL-based instance methods use feature affinities as graph structures, while CutLER and CuVLer generate labels through multi-object graph cuts or ensembles of pretrained DINO models.Several approaches operate at fixed granularities; HASSOD and SOHES use hierarchical merging to produce labels at multiple granularities.Source: §2 · pp. 3–4
  • UnSAM combines CutLER coarse-object detection with HASSOD clustering of finer parts, addressing high-resolution inputs and part-to-object tradeoffs.The paper describes UnSAM as the state of the art in unsupervised instance segmentation.Source: §2 · p. 4

Training-Free Segmentation with Generative Models

  • Diffusion segmentation methods use either image-text cross-attention for open-vocabulary masks or internal features for class-agnostic masks; Diffuse2Seg focuses on the latter and generates automatic, multi-granularity masks.Prior examples include DiffSeg [Tian et al., 2024], DiffCut [Couairon et al., 2024], TRACE [Jo et al., 2026], and M2N2 [Karmann & Urfalioglu, 2025].Source: §2 · p. 4

3 Diffuse2Seg: Diffusion-Based Instance Segmentation Without Supervision

Diffuse2Seg adapts pretrained diffusion-model representations to generate unsupervised, multi-granular instance labels, then trains a segmentation model on them. Its pipeline extracts self-attention, propagates point prompts into object maps, merges those maps into masks, and refines labels through two-round training.

3.1 Preliminaries

  • The method mirrors SAM’s grid-based automatic mask generation and two-round self-training in an unsupervised setting, using diffusion-model self-attention to produce the masks.SAM’s SA-1B dataset includes objects and object parts, supporting multiple label granularities.Source: §3.1 · pp. 4–5

3.2 Overview of Diffuse2Seg

  • Diffuse2Seg extracts diffusion-model self-attention, propagates a grid of point prompts into soft object maps, merges them into instance pseudo-labels, and trains a segmentation network on those labels.The second training round uses prediction-enhanced labels to fill missing segments and correct model errors.Source: §3.2 · p. 6

3.3 Step 1: Self-Attention Extraction via Single-Step Denoising

  • Single-step denoising extracts self-attention matrices from the SD2 UNet, aggregates their heads and layers into an affinity matrix, and supplies that matrix to label generation.The method uses self-attention from the first two highest-resolution UNet layers; layer weights are hyperparameters.Source: §3.3 · p. 6, §3.1 · p. 6

3.4 Step 2: Prompt-Based Segment Generation

  • Equidistant one-hot prompts are propagated over the affinity graph with edge-preserving p-Laplacian smoothing, producing a soft object map for each prompt.Propagation stops when the squared update norm falls below τprop; low-affinity neighbors limit smoothing across object edges.Source: §3.4 · pp. 6–7, Figure 3 · p. 7
  • Prompt maps are clustered by symmetric KL dissimilarity at multiple thresholds, then upsampled into connected-component masks; area-descending NMS removes redundant candidates.Smaller clustering thresholds yield more clusters suited to small objects, while larger thresholds favor whole or coarse objects; at most 1,000 masks are retained per image.Source: §3.4 · pp. 7–8

3.5 Step 3: Two-Round Model Training on Generated Segments

  • A lightweight, class-agnostic Mask2Former model is trained in two rounds, with the second round enriching generated labels using whole-image and crop-based predictions.Round 1 uses copy-and-paste augmentation to increase small-segment frequency; crop refinement retains children contained within parent predictions and does not depend on CutLER.Source: §3.5 · p. 8

4 Experiments

Experiments evaluate training-free pseudo-label generation across domains, train zero-shot instance-segmentation models on those labels, test semi-supervised fine-tuning, and ablate design choices. Diffuse2Seg improves on prior methods across these evaluations, with crop-refined labels and label-efficient fine-tuning extending its performance.

4.2 Evaluation of Generated Pseudo-Labels

  • On SA-1B, Diffuse2Seg’s mAR exceeds DiffSeg by 4.7 p.p. and UnSAM by 4.3 p.p., despite using a low-resolution representation.Table 1 compares class-agnostic AR by object size with 1,000 proposals per image.Source: §4.2 · p. 10, Table 1 · p. 10
  • Diffuse2Seg consistently achieves the highest recall among training-free methods across domains, including 13.6 AR1000 on PACO, 2.9 p.p. above trained CutLER.On ADE20K and EntitySeg, it gains 6.5 and 4.5 p.p. of mAR over UnSAM, respectively.Source: §4.2 · p. 11, Table 2 · p. 11

4.3 Evaluation of Zero-Shot Instance Segmentation

  • Diffuse2Seg gains +7.4 p.p. AR on things (37.2 vs. 29.8), +7.7 p.p. on stuff+things (40.3 vs. 32.6), and +10.0 p.p. on parts (27.1 vs. 17.1).These results require only half the training images and use a much smaller image encoder: ResNet50 for Diffuse2Seg versus ViT-B for SOHES.Source: §4.3 · p. 11
  • With CutLER proposals added, Diffuse2Seg consistently surpasses UnSAM on stuff+things datasets, while its gains over UnSAM across SA-1B budgets are 3.4, 2.6, and 2.1 p.p. mAR.On things and parts datasets, it performs on par with UnSAM.Source: §4.3 · pp. 11–12, Table 3 · p. 12

4.4 Semi-Supervised Diffuse2Seg-Semi

  • At 10,000 fine-tuning images, Diffuse2Seg-Semi exceeds the fully supervised 100k reference by 1.4 p.p. in average recall across domains.At 5,000 images it matches that reference; at 100 images it recovers 93.5% of its mAR, versus 90.7% for UnSAM.Source: Figure 4 · p. 12, §4.4 · p. 13

4.5 Ablation Study

  • Choosing p = 1.6 prioritizes mAP, while lower p values improve recall but can introduce noisy masks; timestep t = 150 maximizes mAP while retaining strong mAR.Earlier timesteps focus more on fine texture, whereas later timesteps capture coarser semantics.Source: §4.5 · p. 13, Figure 5 · p. 13
  • Six hierarchy levels raise mAR from 16.3 to 18.1 versus three levels, while lowering τIoU from 0.9 to 0.5 reduces mAR by 3.1 p.p.The study retains six levels and τIoU = 0.9 for experiments.Source: Table 4 · p. 13, §4.5 · p. 14
  • Crop-refined masks improve SA-1B and PACO by 3.7 and 1.3 p.p., while CutLER mixing adds 3.6 p.p. on COCO and 2.5 p.p. AR1000 on EntitySeg.In the second self-training round, crop-based labels improve stuff+things, things, and parts by 2.2, 2.0, and 1.9 p.p., respectively; global-only self-training raises stuff+things mAR from 40.3 to 40.7.Source: §4.5 · p. 14, Table 5 · p. 14

5 Conclusion

Diffuse2Seg extracts dense, multi-granular instance pseudo-labels from pretrained diffusion models without retraining or supervision. Its pseudo-labels support strong zero-shot models and semi-supervised initialization, though small objects and some large regions remain challenging.

5 Conclusion

  • Diffuse2Seg extracts dense multi-granular instance pseudo-labels from pretrained diffusion models without retraining or supervision.Experiments report general pseudo-label generation and strong zero-shot performance from models trained on those labels.Source: §5 · p. 14
  • Pseudo-labels struggle with particularly small objects below latent resolution and can oversegment large regions such as sky.The authors suggest coarse-to-fine approaches and objectness ranking as practical solutions.Source: §5 · p. 14

A.1 Pseudo-Label Generation

Diffuse2Seg propagates grid prompts through diffusion-model features to form soft object maps, then merges and filters them into multi-granular instance-mask proposals. Its generator settings are selected on a held-out SA-1B split and kept fixed across datasets and training runs.

A.1 Pseudo-Label Generation

  • Diffuse2Seg propagates equidistant one-hot grid prompts through the SD UNet to produce soft object maps.The method uses a vectorized iterative Gauss-Jacobi solver and combines features from selected layers.Source: §A.1 · pp. 18–19
  • The map-merging stage consolidates redundant or differently granular maps into complementary masks by normalizing maps and computing pairwise prompt distances.It thresholds maps, extracts connected components above a minimum area, then sorts proposals by area and applies IoU-based NMS with a proposal cap.Source: §A.1 · p. 19
  • The final proposal filtering keeps connected components above the area threshold, orders them by descending area, and removes proposals whose IoU with an accepted mask exceeds the threshold.Selection stops at the maximum proposal count.Source: §A.1 · p. 19
  • Figure 6 shows shared optima for some parameters but precision–recall trade-offs for others, while larger grid spacing or resolution can increase computational cost.For the selected s = 6 grid, recall retains 96.6% of that of s = 5; p = 1.6 is chosen to prioritize precision over p = 1.2.Source: Figure 6 · p. 20, §A.1 · p. 20
  • Generator hyperparameters are tuned once on a held-out 1k SA-1B split and frozen across datasets and downstream runs.The configuration uses 1120 × 1120 resolution, τatt = 0.55, grid spacing s = 6, and six log-spaced clustering thresholds.Source: §A.1 · p. 20

A.2 CascadePSP Pseudo-Label Post-Processing

Diffuse2Seg follows UnSAM’s CascadePSP mask-refinement procedure, with filtering criteria and a lower maximum crop size intended to accelerate refinement.

A.2 CascadePSP Pseudo-Label Post-Processing

  • CascadePSP refinement discards masks with area fraction above 0.9 or refinement overlap IoU below 0.5.The procedure follows UnSAM.Source: §A.2 · p. 20
  • Mask crops are grouped into spatial buckets, zero-padded to squares, and processed with FP16 autocast.Source: §A.2 · p. 20
  • The refinement crop side length is capped at 480 rather than UnSAM’s 900 to accelerate processing.The paper attributes the smaller cap to generating more large masks that need less refinement.Source: §A.2 · p. 20

A.3 Training Hyperparameters

The paper trains Mask2Former models under multiple data budgets, then enriches Round-1 labels for Round 2 and evaluates unsupervised checkpoints as semi-supervised initializations.

A.3 Training Hyperparameters

  • Mask2Former with a ResNet-50 backbone is trained with 1%, 2%, and 4% data budgets using shared optimizer and architecture settings.Round 1 initializes the encoder from DINO ResNet-50 and randomly initializes the decoder; Round 2 starts from the matching Round-1 checkpoint.Source: §A.3 · p. 21
  • Round 1 trains the detector-free model on Diffuse2Seg labels only, for approximately eight epochs per budget.Both Round-1 configurations use hierarchical copy-paste augmentation with pcp = 0.5.Source: §A.3 · p. 21
  • Round 2 re-infers predictions on full images and padded parent crops, replacing original masks when their IoU with new masks exceeds 0.5.Round 2 fine-tunes only with 0.1M crop-refined labels, independent of the Round-1 image budget.Source: §A.3 · p. 21
  • Semi-supervised fine-tuning freezes the ResNet-50 backbone and trains only the Mask2Former decoder across budgets from 100 to 10k labeled images.The comparison includes Diffuse2Seg and UnSAM initializations and a 100k fully supervised baseline trained from scratch.Source: §A.3 · p. 21

A.4 Semi-Supervised Results

On ADE20K, Figure 7 compares UnSAM and Diffuse2Seg fine-tuned with the same 10k-label budget. The Diffuse2Seg predictions show greater mask coverage, sharper boundaries, and finer structures.

A.4 Semi-Supervised Results

  • With the same 10k-label budget on ADE20K, Figure 7 compares predictions from UnSAM and Diffuse2Seg checkpoints fine-tuned on labeled images.Source: Figure 7 · p. 22
  • The Diffuse2Seg predictions show higher mask coverage and crisper object boundaries than the UnSAM predictions.Source: Figure 7 · p. 22
  • Diffuse2Seg predictions better segment fine structures, including furniture legs, windows, and facade details.Source: Figure 7 · p. 22
Loading 2609.06491v1…