PaperlayerView exact revision on arXiv

Source-linked AI summary

Complementing DINO Features with Image Structure for Part Discovery

Samyak Rawlekar, Nikhil Chowdary Paleti, Amey Gupta, Narendra Ahuja

arXiv:2610.32293v1cs.CVcs.AIcs.LG

TL;DR

Patch-based DINO methods discover semantic parts but produce imprecise boundaries, in part because patches can mix pixels from multiple parts. Part-DINO combines low-level image boundaries with DINO features to group geometrically coherent regions. It outperforms training-free baselines across four benchmarks and remains competitive with trained methods, with limitations tied to specified part granularity and image contrast.

  • Problem

    Patch-based part-discovery methods using DINO features remain spatially imprecise, with assignments bleeding across neighboring part contours.

    Source: §1 · p. 1

  • Method

    Part-DINO uses low-level image-segmentation boundaries to define regions, pools frozen DINO features within them, and groups the regions semantically without training.

    Source: §1 · p. 2

  • Results

    Part-DINO outperforms the patch-based training-free baseline on NME and ARI across all four benchmarks and on NMI for three of four, while remaining competitive with trained methods.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    In Part-DINO, image-derived regions provide boundaries while DINO features are used to group regions into semantic parts.

    Source: §1 · p. 2, §6 · p. 9

    Part-DINO requires the number of parts to be specified in advance, and its contrast-derived boundaries can under-segment low-contrast scenes or introduce spurious edges from shadows and illumination gradients.

    Source: §6 · p. 10

Abstract

from arXiv · show

Self-supervised Vision Transformers (ViTs) like DINO encode rich semantics, enabling training-free segmentation of objects from the background and discovery of their parts via feature clustering. This paper focuses on part discovery: partitioning a foreground object into semantic parts. Existing approaches produce imprecise part boundaries. We trace this to the rigid ViT patch grid used in training DINO, where patches straddling boundaries contain pixels from multiple parts. While DINO's invariance-based training could absorb this pixel-level mixing into clean features, experiments with over 26K foreground patches show it does not: boundary-overlapping patches produce features with higher entropy and lower assignment confidence, blurring predicted part boundaries. Rather than make the massive effort of retraining DINO over geometrically coherent regions, an easier approach is to take boundaries directly from a boundary detector, which supplies the geometric information DINO's features lack. Motivated by this, we introduce Part-DINO, a training-free framework that combines boundaries from low-level image segmentation with DINO's semantic features. We use the boundaries to partition the image into regions, pool DINO features within each region, and group them to discover parts. Without any training, Part-DINO produces structurally coherent part assignments, outperforming training-free baselines across four standard benchmarks (CUB, CelebA, PartImageNet-OOD, PartImageNet-Seg) and remaining competitive with weakly-supervised and unsupervised methods.

1 Introduction

DINO features support semantic part discovery, but patch-based methods produce spatially imprecise assignments that bleed across part contours. Part-DINO combines image-derived boundaries with DINO features and, without training, outperforms training-free baselines across four benchmarks while remaining competitive with trained methods.

1 Introduction

  • Boundary-overlapping patches mix pixels from multiple semantic regions, and across 26K foreground patches this mixing corresponds to higher feature entropy and lower assignment confidence.DINO-similarity edges blur at part contours, whereas image-contrast edges align with them.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • Part-DINO uses low-level image-segmentation boundaries to partition images into regions, pools DINO features within each region, and assigns regions to their nearest part prototypes.The framework is training-free and uses geometry for region boundaries and DINO features for semantic grouping.Source: §1 · p. 2
  • Part-DINO outperforms the patch-based training-free baseline on NME and ARI across all four benchmarks and on NMI for three of four, while remaining competitive with trained models.The slight NMI drop on PartImageNet-OOD reflects a metric trade-off in which ARI rewards spatial coherence over fine fragmentation.Source: §1 · p. 2
  • The paper traces spatial imprecision in patch-based part assignments to the ViT’s fixed patch grid, whose geometry-agnostic patches can span multiple parts.DINO’s invariance-based training does not guarantee that pixel mixing in boundary patches is resolved in learned features.Source: §1 · pp. 1–2

2 Related Work

Prior part-discovery methods use weak supervision, unsupervised training, or frozen DINO patch features; the trained approaches require computationally expensive, dataset-specific optimization. Part-DINO follows the training-free approach but uses geometric regions as its units.

Part Discovery

Region Generation

  • SLIC superpixels trade boundary adherence for compactness and can cross irregular part edges, while Ramp Transform and Watershed define regions from contrast gradients.The cited methods are SLIC and Ramp Transform and Watershed (Akbas and Ahuja, 2020, 2014).Source: §2 · p. 3

3 Part-DINO

Part-DINO combines image-derived regions, which preserve object geometry, with DINO features, which support semantic grouping. It builds category-level part prototypes from reference images and assigns each test-image region to its nearest prototype without additional training.

3 Part-DINO

  • Across 26,459 foreground patches, boundary overlap is associated with higher feature entropy and lower assignment margin; 21.9% have spatial purity below 1.Feature quality degrades continuously as overlap increases.Source: §3 · p. 3

3.1 Region Generation

  • Part-DINO extracts object regions from image-contrast edges, then merges regions smaller than smin into their most similar neighbor.Watershed places boundaries along ridges in the edge map; merging removes boundaries but does not add them.Source: §3.1 · pp. 4–5

3.2 Global Prototype Generation

  • Each region’s feature is the ℓ2-normalized average of upsampled DINO pixel features, pooling semantic information within its structural boundary.The regions are intended to avoid the boundary-information loss of fixed-grid patches.Source: §3.2 · p. 5
  • K-Means over region features from a reference subset creates category-specific global prototypes, each representing a part across instances.For datasets with multiple categories, the prototype dictionary is generated separately for each class.Source: §3.2 · p. 5

3.3 Region Assignment

  • At inference, each test-image region is assigned to its nearest semantic prototype, and every pixel in that region receives the same part label.For multiple categories, a CLS-token comparison selects the category-specific dictionary before part assignment.Source: §3.3 · pp. 5–6

4 Experiments

Part-DINO is evaluated on CUB, CelebA, and the PartImageNet OOD and Segmentation splits using NMI, ARI, and NME. It improves on the training-free baseline across several settings and is competitive with trained methods, with outcomes varying by dataset and metric.

Evaluation setup

  • Evaluation covers four benchmarks with frozen DINOv2 ViT-B/14 features and uses NMI, ARI, and NME to assess clustering alignment and centroid alignment.Input resolution is 448 × 448 for CUB and 224 × 224 for the other datasets; foreground masks come from SAM3.Source: §4 · p. 7

Results

  • On CUB at K=16, Part-DINO raises NMI from 56.92 to 65.45 and ARI from 35.10 to 46.00 over the patch-based training-free baseline.It matches or exceeds several weakly supervised and unsupervised methods at K=16; on PartImageNetO it has higher ARI but slightly lower NMI than the patch baseline.Source: §4 · p. 7
  • On CelebA, Part-DINO beats GANSeg at K=4 on ARI, 32.20 versus 28.06, while trained methods remain stronger.The patch-based baseline has ARI below 0.1 across all K.Source: §4 · p. 7
  • On PartImageNetS, Part-DINO improves over the training-free baseline on NMI and ARI for K ∈{8, 25, 50}, maintains ARI above 66, and beats MPAE at K=8.At K=8, NMI/ARI are 37.21/66.10 for Part-DINO versus 33.51/65.05 for MPAE.Source: Table 5 · p. 8, §4 · p. 8

5 Analysis

The analyses compare atomic units, region-generation methods, granularity, and edge maps, and report a small computational overhead. They find that geometric regions and structural edges improve part discovery, with finer granularity helping until performance saturates.

Unit (Patches vs Regions)

  • Combining region-based assignment with region-based prototypes gives the largest improvement across metrics; using regions only for assignment yields larger gains than using them only for prototypes.At CUB K=4, the combined setting improves NMI from 32.83 to 53.93 and ARI from 13.96 to 23.01.Source: §5 · p. 8

Region Generation

  • At 500 regions, RAMP + Watershed improves ARI by 7.36 over SLIC and also beats SLIC at 500 regions with only 142 regions.RAMP + Watershed places boundaries along intensity gradients, while SLIC’s compactness constraint trades boundary adherence for spatial uniformity.Source: §5 · p. 8

Region Granularity

  • Reducing dmin from 15 to 2 increases the average region count from 22 to 500 and improves all metrics, including ARI from 25.21 to 46.00.Performance saturates around 300 regions; coarse regions can span multiple parts, while finer regions are more likely to remain within one part.Source: §5 · pp. 8–9, Figure 4 · p. 9

Structural vs. Semantic Edge Map

  • At K=16, structural edges improve NMI by 9.06 and ARI by 7.05 over DINO semantic edges; the advantage widens at finer granularities.Structural edges are derived from RAMP, while DINO semantic edges are blurred at object boundaries.Source: Table 6 · p. 9, §5 · p. 9

Computational Efficiency

  • Part-DINO’s region-construction pipeline adds 72 ms per image over a shared DINOv2 backbone and SAM3 masking, alongside gains of 8.6 NMI and 10.9 ARI at dmin=2.The added pipeline comprises RAMP, watershed, merging, and pooling.Source: §5 · p. 9

6 Discussion

Part-DINO addresses boundary-patch mixing by using low-level image segmentation to define regions and DINO features to group them. It outperforms the patch-based training-free baseline on NME and ARI across four benchmarks, but depends on specified part granularity and image contrast.

Discussion

  • Part-DINO outperforms the patch-based training-free baseline on NME and ARI across all four benchmarks and granularities, and matches or exceeds several training-based methods.Its motivation is the finding that boundary-overlapping patches propagate pixel-space mixing into feature space.Source: §6 · pp. 9–10

Limitations and future work

  • Part-DINO requires K in advance, and low-contrast or uniform scenes can weaken boundaries and cause under-segmentation, while illumination gradients or shadows can create spurious edges.The method discovers unlabeled part clusters; assigning semantic labels with vision-language models is proposed as future work.Source: §6 · p. 10

Broader Impact

  • The training-free approach reduces the computational energy footprint of dataset-specific part discovery, while enhanced unsupervised localization could be misused for unauthorized biometric surveillance.Source: §6 · p. 10

NeurIPS Paper Checklist

The NeurIPS checklist records the paper’s reported reproducibility, experimental-setting, limitations, societal-impact, and asset disclosures. The appendix previews additional efficiency, boundary-mixing, qualitative-comparison, and region-generation analyses.

NeurIPS Paper Checklist

  • The authors report a limitations section, discussion of positive and negative societal impacts, and proper credit for existing assets.They also say no new assets are introduced and the work involves neither crowdsourcing nor human subjects.Source: NeurIPS Paper Checklist · pp. 12, 15, 16, 17
  • The authors report that implementation details provide reproducibility information and that the paper specifies experimental settings and compute resources.The checklist also says the code details are provided to faithfully reproduce the main experiments.Source: NeurIPS Paper Checklist · pp. 14–15
  • The checklist answers No on statistical-significance reporting, citing prior-work experimentation and identical results from centroid use.The checklist’s answer is No; its justification says the experimentation follows prior work and the centroid results are the same.Source: NeurIPS Paper Checklist · p. 14

Appendix

  • The appendix previews extended computational-efficiency, boundary-mixing, structural-versus-semantic-boundary, and region-generation analyses.It also says region generation remains effective without the DINO-based micro-region merge.Source: Appendix · p. 19

A Computational Efficiency: Extended Analysis

Part-DINO adds region construction to a shared DINO and SAM3 pipeline, with a reported 72 ms per-image overhead and improved NMI and ARI at dmin = 2.

Computational Efficiency

  • The efficiency measurements use the CUB test split at 448×448 on one NVIDIA A40 GPU with 48 CPU cores and no batching.The patch baseline shares the same backbone and SAM3 segmenter call; only feature extraction and assignment differ.Source: A Computational Efficiency: Extended… · p. 19
  • RAMP edge estimation and watershed seeding take about 57 ms independently of dmin, while merging takes 7.7 ms at dmin = 2 and 5.2 ms at dmin = 15.Pooling takes about 1 ms regardless of region granularity; the reported merge times correspond to means of 538 and 23 regions, respectively.Source: A Computational Efficiency: Extended… · p. 19

B Boundary Analysis Details

The boundary analysis relates distance from a part contour to patch purity, feature entropy, and assignment margin, finding that boundary-adjacent patches are less reliable.

Boundary Analysis Details

  • The analysis uses 26,459 PartImageNet foreground patches and measures distance to a boundary against purity, entropy, and top-2 assignment margin.Ground-truth part centroids are built from training patches with purity π > 0.95; purity is the fraction of foreground pixels assigned to the majority part.Source: B Boundary Analysis Details · p. 19
  • ρ(d, π) = +0.71: spatial purity increases with distance from the nearest part boundary.Patches on or adjacent to contours pool pixels from distinct parts.Source: B Boundary Analysis Details · p. 19
  • ρ(d, H) = −0.38 and ρ(d, m) = +0.36: near boundaries, feature entropy rises and assignment margin narrows.Despite dataset-wide mean patch purity of 0.952, these metrics degrade sharply at d ≈0.Source: B Boundary Analysis Details · p. 19

C Extended Qualitative Analysis: Semantic vs. Structural Boundaries

The qualitative analysis contrasts semantic edges derived from DINO similarity with structural boundaries derived from image contrast and the RAMP transform.

Extended Qualitative Analysis: Semantic vs. Structural Boundaries

  • Structural boundaries from the RAMP transform reduce semantic bleeding and produce sharper parts than DINO-similarity edges across CelebA and other complex object categories.The comparison is shown for CelebA in Figure 6 and other categories in Figures 8 and 9.Source: C Extended Qualitative Analysis:… · pp. 19–20

D Extended Qualitative Analysis on Region Generation: SLIC vs. Watershed

The region-generation ablation contrasts rigid SLIC boundaries with Watershed regions that follow structural-gradient valleys around physical contours.

D Extended Qualitative Analysis on Region Generation: SLIC vs. Watershed

E DINO-Free Region Generation

Disabling DINO-based micro-region merging leaves Part-DINO ahead of the patch-based baseline on CUB, supporting the role of DINO-independent boundaries in its gains.

E DINO-Free Region Generation

  • DINO participates only in merging tiny watershed regions, which removes boundaries but never places them; setting smin=0 makes region generation DINO-independent.Source: E DINO-Free Region Generation · p. 20
  • 18.5 extra regions per image with smin=0 changed NMI and NME by 0.03 and reduced ARI by 1.91; versus the patch-based baseline, NMI was +8.09, ARI +8.82, and NME 0.51 lower.Both Part-DINO rows come from the same run; smin=0 uses no DINO features for region generation.Source: E DINO-Free Region Generation · p. 20, Table 7 · p. 20
  • Table 8 reports timing in milliseconds, average region counts, and clustering performance metrics.Source: Table 8 · p. 20
  • Across CelebA, CUB, and PartImageNet OOD and Seg examples, structural boundaries eliminate semantic bleeding and improve part-boundary precision and geometric alignment.Source: Figure 6 · p. 21, Figure 8 · p. 22, Figure 9 · p. 23
  • Watershed follows structural-gradient valleys around physical contours, whereas SLIC compactness can force grid-like boundaries across irregular anatomy.Source: Figure 10 · p. 24

Related papers

Loading 2610.32293v1…