PaperlayerView exact revision on arXiv

Source-linked AI summary

UnSAMv2: Self-Supervised Learning Enables Segment Anything at Any Granularity

Junwei Yu, Trevor Darrell, XuDong Wang

arXiv:2511.13714v1cs.CVcs.AIcs.LG

TL;DR

SAM’s supervised training ties objectness to human-labeled masks and limits prompts to discrete hypotheses, while scenes contain nested part–whole structures. UnSAMv2 discovers hierarchical masks and continuous scales from unlabeled images, then uses granularity control to condition SAM-2. It improves reported measures and achieves state-of-the-art results across interactive, whole-image, and video segmentation benchmarks.

  • Problem

    SAM’s supervised masks tie objectness to human labels and offer at most three discrete hypotheses per prompt, limiting continuous control and hierarchical part–whole reasoning.

    Source: §3.1 · p. 4, §3.2 · p. 4

  • Method

    UnSAMv2 discovers hierarchical masks and continuous granularity scales from unlabeled data, then conditions SAM-2 on a point prompt and granularity scalar.

    Source: §1 · p. 2

  • Results

    UnSAMv2 achieves state-of-the-art results across interactive, whole-image, and video segmentation; NoC90 improves from 5.69 to 4.75 using only unlabeled data.

    Source: §1 · pp. 2–3, §6 · p. 9

  • Takeaways & Limitations

    With 6K unlabeled images, UnSAMv2 learns to traverse part–whole hierarchies and control segmentation with a single scalar.

    Source: §6 · p. 9

    SAM’s supervised training produces up to three discrete masks per prompt, limiting continuous granularity control in interactive segmentation.

    Source: §3.2 · p. 4

Abstract

from arXiv · show

The Segment Anything Model (SAM) family has become a widely adopted vision foundation model, but its ability to control segmentation granularity remains limited. Users often need to refine results manually - by adding more prompts or selecting from pre-generated masks - to achieve the desired level of detail. This process can be ambiguous, as the same prompt may correspond to several plausible masks, and collecting dense annotations across all granularities is prohibitively expensive, making supervised solutions infeasible. To address this limitation, we introduce UnSAMv2, which enables segment anything at any granularity without human annotations. UnSAMv2 extends the divide-and-conquer strategy of UnSAM by discovering abundant mask-granularity pairs and introducing a novel granularity control embedding that enables precise, continuous control over segmentation scale. Remarkably, with only $6$K unlabeled images and $0.02\%$ additional parameters, UnSAMv2 substantially enhances SAM-2, achieving segment anything at any granularity across interactive, whole-image, and video segmentation tasks. Evaluated on over $11$ benchmarks, UnSAMv2 improves $\text{NoC}_{90}$ (5.69 $\rightarrow$ 4.75), 1-IoU (58.0 $\rightarrow$ 73.1), and $\text{AR}_{1000}$ (49.6 $\rightarrow$ 68.3), showing that small amounts of unlabeled data with a granularity-aware self-supervised learning method can unlock the potential of vision foundation models.

1. Introduction

UnSAMv2 frames objectness as something users can define with a point prompt and continuous granularity, addressing SAM’s limited, discrete mask choices and missing part–whole correspondences. It discovers hierarchical masks from unlabeled images and adds granularity control to SAM-2, improving results across segmentation tasks.

1. Introduction

  • Users can define objectness with a single point prompt and continuous granularity score, moving from fine parts toward larger, more coherent entities.This replaces three discrete mask hypotheses with a continuous axis of user-defined granularity.Source: §1 · p. 2
  • SAM’s supervised training ties objectness to human annotation and limits each prompt to three discrete mask hypotheses, despite scenes’ nested part–whole structures.SA-1B lacks explicit correspondences between instance- and part-level masks, making it difficult for supervised models to learn continuously varying granularity.Source: §1 · p. 2
  • Across interactive, whole-image, and video segmentation benchmarks, UNSAMV2 surpasses SAM-2 and prior promptable methods, improving NoC90 from 5.69 to 4.75, 1-IoU from 58.0 to 73.1, and AR1000 from 49.6 to 68.3.These results use only unlabeled data; training used 6,000 unlabeled images.Source: §1 · pp. 2–3
  • UnSAMv2 extends UnSAM’s divide-and-conquer approach by extracting instance masks, recursively merging similar pixels into finer parts, and assigning each mask a continuous part–whole scale.The resulting hierarchical pseudo-labels encode relative scale.Source: §1 · p. 2
  • A lightweight granularity encoder and mask token condition SAM-2 on a point prompt and scalar g, enabling prediction at the requested scale.Training the lightweight decoder for four hours on 2 A100 GPUs uses 6,000 unlabeled images and adds 0.02% parameters.Source: §1 · p. 2

2. Related Work

Prior promptable segmentation methods offer limited choices or discrete granularity controls, while unsupervised work has developed strategies for discovering masks and hierarchies. UnSAMv2 builds on hierarchical mask discovery by assigning each pseudo-mask an explicit granularity scale.

2. Related Work

  • SAM and Semantic-SAM [35], constrain point prompts to a fixed number of masks, requiring manual selection or additional prompts; Figure 3 contrasts this with continuous control.The caption describes up to three SAM-2 masks per point and a continuous granularity variable in UNSAMV2.Source: §2 · p. 3, Figure 3 · p. 3
  • GARField [23] and SAMPart3D [54] [55] use absolute scale conditioning for 3D decomposition, whereas GraCo [58] adds discrete granularity inputs to SimpleClick [29].Source: §2 · p. 3
  • Unsupervised segmentation includes CutLER’s MaskCut strategy [44], VideoCutLER’s cut–synthesize–learn pipeline [46], CutS3D [38], and SOHES’s bottom-up merging [4].UnSAM [47] introduced a divide-and-conquer approach for hierarchical pseudo-labels.Source: §2 · p. 3
  • UnSAMv2 builds on UnSAM’s divide-and-conquer pipeline [47] and assigns an explicit granularity scale to each pseudo-mask for segmentation at arbitrary detail.Source: §2 · p. 3

3. Method

UnSAMv2 learns segmentation granularity from unlabeled image hierarchies and exposes it as a continuous scalar control. It adds granularity-aware decoding to SAM and offers a lightly supervised variant that incorporates annotated masks.

3.1. Preliminaries

  • SAM-2 is trained on human-labeled masks and uses three fixed mask tokens, limiting each prompt to three discrete hypotheses.Its image, prompt, and mask decoder provide the promptable segmentation framework that UnSAMv2 extends.Source: §3.1 · p. 4

3.2. Limitations of SAM’s Supervised Paradigm

  • Human-labeled supervision encourages a flat object representation, making intermediate detail and nested part–whole relationships difficult to capture.With ambiguous prompts, SAM requires users to select among discrete masks rather than traverse scales continuously.Source: §3.2 · p. 4

3.3. Granularity-Aware Divide-and-Conquer

  • The pipeline filters N-Cut masks, links instance and part masks by overlap, refines instances through patch merging, then assigns granularity scores by relative area.Patch merging and mask combination expand the hierarchy while maintaining mask quality.Source: §3.3 · pp. 4–5
  • Most discovered masks have granularity scores below 0.4, yet the model learns stable semantics across scales.The resulting hierarchy is rich and left-tailed, with many fine-grained structures.Source: §3.3 · p. 5

3.4. UNSAMV2 Architecture

  • A Fourier embedding and MLP map the scalar granularity input into decoder features, while a learnable mask token attends to prompt and image features.The added encoding and token modules introduce less than 0.02% extra parameters.Source: §3.4 · p. 5

3.5. Distinctions with prior methods.

  • Unlike discrete or absolute-size approaches, UNSAMv2 defines granularity continuously and relatively within the instance–part hierarchy.The authors argue that this better reflects how parts and wholes are interpreted in relation to one another.Source: §3.5 · p. 5

3.6. UNSAMV2+: Light-Supervised Variant

  • UNSAMv2+ adds SA-1B ground-truth masks to the divide stage, then uses the same hierarchy and granularity-assignment steps.The hybrid variant is intended to produce cleaner masks while retaining the scalability and flexibility of unsupervised granularity learning.Source: §3.6 · p. 5

4. Experiments

UNSAMv2 is evaluated for interactive, whole-image, and video segmentation after training on 6,000 unlabeled images. It outperforms SAM-2 on interactive benchmarks and reports gains over SAM and UnSAM in whole-image segmentation.

4.1. Model Training Settings

  • Training uses 6,000 unlabeled images and 8 A100 GPU hours, with the vision encoder frozen and decoder components fine-tuned.The setup uses SAM-2-small, five epochs, and LoRA for the two-way transformer block in the mask decoder.Source: §4.1 · p. 5

4.2. Evaluation Datasets and Metrics

  • Interactive evaluation reports NoC80, NoC90, and one-click IoU; whole-image evaluation uses AR1000 because class-limited AP does not capture open-world performance.The interactive comparisons exclude datasets that are in-distribution with a model’s training data.Source: §4.2 · p. 6

4.3. Experimental Results

  • With 6,000 images and combined supervised and unsupervised labels, UNSAMv2+ achieves state-of-the-art performance across interactive segmentation metrics and multiple datasets.The reported comparison reproduces results for SimpleClick [29] and GraCo [58] using their official code and checkpoints.Source: Table 2 · p. 7, §4.3 · p. 7
  • On average, UNSAMv2 surpasses SAM-2 by 16.5% in NoC90; on whole-image segmentation, it exceeds SAM and UnSAM [47] by 37.7% and 29.8% in AR1000, respectively.For whole-image evaluation, UNSAMv2 aggregates masks at granularity levels from 0.1 to 1.0 in increments of 0.1 and filters low-confidence masks.Source: §4.3 · p. 7, Table 3 · p. 7

5. Ablations

The ablations examine pseudo-label sources, training efficiency, mask-token design, and training settings, alongside examples of granularity control across image and video tasks.

Ablations

  • In whole-image examples, low granularity reveals fine parts while higher values recover whole objects, including in densely packed scenes.The visualization presents controllable, scalable whole-image segmentation.Source: Figure 8 · p. 8
  • For video, a point and granularity value on the first frame are propagated to later frames, where masks remain consistent over time despite image-only training.The figure describes this as evidence of temporal coherence and transferability.Source: Figure 9 · p. 8
  • 1,000 images provide a solid grasp of granularity; the authors attribute sample efficiency to relative-scale hierarchies and updating only 0.1% of parameters.Freezing the remaining pretrained parameters preserves the model’s segmentation ability.Source: §5 · p. 8
  • Directly finetuning SAM-2’s original mask tokens yields limited gains, while a new granularity-aware token enables efficient granularity learning.The new token and its MLP decode masks using granularity scales, with self-attention to the encoded granularity and cross-attention to image embeddings.Source: Figure 10 · p. 8, §5 · pp. 8–9
  • SA-1B ground-truth-only training performs worse than training with divide-and-conquer pseudo-labels, which are pivotal for granularity learning.The comparison indicates that the unsupervised pseudo-labels contribute to learning granularity beyond human-labeled annotations alone.Source: Table 5 · p. 8, §5 · p. 9
  • A single granularity value can isolate a target with one prompt, while multi-point prompts remain available for finer control.The comparison contrasts this with SAM-2’s frequent need for multiple clicks.Source: Figure 11 · p. 9
  • LoRA lets UNSAMV2 learn granularity efficiently while retaining SAM-2’s segmentation capability.The study also finds that three correction points per mask are used for efficiency, moderate masks per iteration support learning and stability, and a moderate Fourier dimension distinguishes granularity levels smoothly.Source: §5 · p. 9

6. Conclusion

UNSAMV2 uses continuous granularity scales derived from unlabeled data to control segmentation across the part–whole hierarchy. Trained on 6K unlabeled images, it achieves state-of-the-art results across interactive, whole-image, and video segmentation.

6. Conclusion

  • UNSAMV2 derives continuous granularity scales from unlabeled data and uses a single scalar to control segmentation across part–whole hierarchies.The conclusion frames this as transforming segmentation from discrete prediction into continuous, controllable reasoning.Source: §6 · p. 9

A1. Evaluation Datasets

The evaluation uses interactive and whole-image benchmarks spanning instance-, part-, and semantic-level segmentation, including zero-shot evaluations on several datasets.

Interactive Segmentation

  • Interactive evaluation covers GrabCut, Berkeley, DAVIS, SA-1B, SBD, PascalPart, and PartImageNet, including instance- and part-level annotations.The stated evaluation subsets include 345 DAVIS frames and 1,000 SA-1B images not used for training.Source: A1. Evaluation Datasets · p. 13

Whole-Image Segmentation

  • Whole-image segmentation evaluates UNSAMV2 across COCO, SA-1B, LVIS, EntitySeg, and ADE20K, spanning object and instance masks, open-world class-agnostic entities, and semantic categories across five datasets.COCO Val2017 (5,000 images), EntitySeg validation (1,314 low-resolution images), and ADE20K test (2,000 images) are evaluated zero-shot; SA-1B uses 1,000 images excluded from UNSAMV2 training, while LVIS uses 5,000 validation images.Source: A1. Evaluation Datasets · p. 13
  • COCO Val2017 is evaluated on 5,000 images using AR1000, while evaluation also uses 5,000 LVIS validation images and 1,314 low-resolution EntitySeg validation images.ADE20K covers 365 scenes and provides 150 semantic categories for its test split.Source: A1. Evaluation Datasets · p. 13

A2. Training Details.

The pipeline generates mask–granularity pseudo-label pairs from 6,000 SA-1B images, then finetunes selected SAM-2 components while freezing its image encoder.

Pseudo mask-granularity data generation

  • The divide-and-conquer pipeline generates mask–granularity pairs on 6,000 SA-1B images, averaging 112 pseudo-labels per image to represent instance–part relationships.The pipeline uses DINOv3 features and multiple cosine-similarity thresholds; unlike UnSAM, it intentionally generates fewer pseudo-labels per image.Source: A2. Training Details · p. 13

UNSAMV2 Training

  • Training finetunes SAM-2-small for five epochs, freezing the Hiera image encoder and training the granularity modules and mask-decoder transformer block.The granularity scalar is Fourier-encoded at dimension 128 and passed through a three-layer MLP; training uses focal and dice losses at a 20:1 ratio, batch size 4, and LoRA rank 8.Source: A2. Training Details · p. 13

A3. More Visualizations

The appendix presents qualitative visualizations of UnSAMv2 for interactive image, whole-image, and video segmentation.

A3. More Visualizations

  • Figure A1 compares interactive segmentation on SA-1B, with GraCo [58] above and UnSAMv2 below.Source: Figure A1 · p. 14
  • Figure A2 shows whole-image segmentation on SA-1B in rows ordered as raw images, SAM-2, then UnSAMv2.Source: Figure A2 · p. 15
  • Figure A3 presents UnSAMv2 interactive video segmentation on YouTubeVIS at various granularity levels.Source: Figure A3 · p. 16
Loading 2511.13714v1…