PaperlayerView exact revision on arXiv

Source-linked AI summary

Seeing as Humans Do: Learning from Motion to Segment Anything Without Supervision

Weijian Jian, Xiaoyue Zhang, Bin Xiao, Chunyu Xie, Yixiao He, Yutao Liu, Dawei Leng, Yuhui Yin

arXiv:2609.39785v1cs.CV

TL;DR

SAM depends on massive manual annotations, while existing unsupervised methods can overfit to moving objects and lack multi-granularity understanding. MoSA learns a transferable objectness prior from unlabeled videos through motion pseudo-labels, contrastive grouping, and transfer to prompt-guided segmentation. Across seven benchmarks, it outperforms existing unsupervised methods and achieves performance comparable to SAM without manual annotations.

  • Problem

    SAM relies on costly manual annotations, while prior unsupervised video methods can overfit to moving entities and lack multi-granularity understanding.

    Source: §1 · pp. 1–3

  • Method

    MoSA generates multi-granularity motion pseudo-labels, trains an appearance-driven Perceptual Grouping Model with contrastive learning, and transfers its learned prior to prompt-guided segmentation.

    Source: §1 · p. 3

  • Results

    Across seven benchmarks, MoSA consistently outperforms existing unsupervised methods and achieves zero-shot segmentation performance comparable to SAM without manual annotations.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The findings support learning general object perception from large-scale unlabeled motion as a feasible and scalable path toward general-purpose segmentation.

    Source: §1 · p. 3, §6 · p. 15

    The paper identifies motion-based prior methods as prone to overfitting to moving objects and lacking multi-granularity understanding.

    Source: §1 · p. 3

Abstract

from arXiv · show

The Segment Anything Model (SAM) relies heavily on massive manual annotations, creating a fundamental bottleneck for model scaling. While unsupervised methods attempt to learn object concepts from motion, they typically overfit to moving entities, lacking both multi-granularity understanding and the ability to generalize to static objects. To overcome this, we introduce Motion-Grounded Segment Anything (MoSA), a highly scalable unsupervised framework that learns a transferable objectness prior from unlabeled videos. MoSA operates in three progressive stages: (1) automatically generating multi-granularity motion pseudo-labels from large-scale video data; (2) training a Perceptual Grouping Model (PGM) via contrastive learning to internalize a generalized, appearance-driven concept of objects; and (3) transferring this learned prior into a prompt-guided architecture for segment-anything-style inference on images. Extensive zero-shot evaluations across seven challenging benchmarks (e.g., COCO and ADE20K) demonstrate that MoSA significantly outperforms existing unsupervised methods. Notably, despite using zero manual annotations, MoSA achieves segmentation performance comparable to the fully supervised SAM. Our findings reveal that harnessing large-scale unlabeled motion is a feasible and highly scalable alternative to annotation-driven segment-anything pipelines.

1 Introduction

SAM’s reliance on costly manual annotations limits scalable segmentation, while prior unsupervised approaches struggle with static objects and multiple object granularities. MoSA learns objectness from unlabeled videos and reports results comparable to SAM across seven benchmarks.

1 Introduction

  • SAM’s reliance on the manually annotated SA-1B dataset creates a bottleneck for model scaling, motivating annotation-free unsupervised segmentation.Source: §1 · pp. 1–2
  • Image-based methods such as CutLER and UnSAM use appearance cues, including self-supervised features, but overlook motion-based grouping.Source: §1 · p. 2
  • Video-based work ranges from binary moving-foreground masks to methods such as OCLR, RCF, and DyStaB that pursue instance- or scene-level understanding.Source: §1 · p. 2
  • Existing video-based methods can overfit to moving objects and lack multi-granularity understanding, limiting their generalization to static or weakly moving entities.The introduction gives roof tiles as an example of a static object.Source: §1 · p. 3
  • Across seven benchmarks, MoSA consistently outperforms existing unsupervised methods and achieves zero-shot segmentation performance comparable to SAM without manual annotations.The framework learns from unlabeled videos.Source: §1 · p. 3
  • MoSA generates multi-granularity motion pseudo-labels, trains an appearance-driven Perceptual Grouping Model, and transfers the learned prior to prompt-guided segmentation.The PGM uses contrastive learning.Source: §1 · p. 3

2 Related Work

Related work spans appearance-based image segmentation and motion-grounded video approaches. The paper identifies limits in both, including insufficient world-grounded object priors and motion bias in methods focused on instance masks.

Unsupervised Image Segmentation

  • Image-based approaches use pretrained vision-transformer features and clustering or pseudo-label pipelines; examples include LOST, TokenCut, CutLER, and UnSAM.Source: Unsupervised Image Segmentation · pp. 3–4
  • These image-based methods commonly rely on DINO features and hierarchical clustering, but may lack the world-grounded prior humans develop through long-term observation.Source: Unsupervised Image Segmentation · p. 4

Motion-to-Objectness Bootstrapping

  • Motion-to-objectness methods include DyStaB, RCF, slot attention, and DIOD, yet remain focused on instance masks and can fail on static or rarely moving objects.The paper identifies multi-granularity object understanding and transferable object priors as open challenges for this line of work.Source: Motion-to-Objectness Bootstrapping · p. 4

3 Motion Pseudo-Label Generation

MoSA builds multi-granularity motion pseudo-labels from synthetic and real-world videos, then filters them for use in later training. Its pipeline combines rigid-instance and moving-foreground masks to cover different motion patterns.

3 Motion Pseudo-Label Generation

  • MGMS applies synthetic-data-trained modules to large-scale real-world videos and assesses candidate-mask quality before producing pseudo-labels for subsequent training.The pipeline generates masks at multiple granularities.Source: §3 · pp. 4–5

3.1 Synthetic Video Generation

  • Synthetic videos use pixel-accurate masks, multiple occluding objects, camera motion, and smooth trajectories to train motion-segmentation modules.Their unambiguous motion cues support training despite limited semantic realism.Source: §3.1 · p. 5

3.2 Multi-Granularity Motion Segmentation

3.3 Mask Quality Assessment and Filtering

  • Mask filtering combines prediction confidence and boundary sharpness, then applies a threshold and NMS to retain high-fidelity, non-duplicate pseudo-labels.Maskness averages predictions above a confidence threshold; boundary sharpness measures strong gradients near the mask contour.Source: §3.3 · pp. 6–7

3.4 Pseudo-Label Dataset Construction

  • Around 21 million high-quality motion pseudo-labels were generated from approximately 10,000 hours of raw video.The dataset spans human activities, animal behaviors, and driving footage.Source: §3.4 · p. 7

4 Learning from Motion to Segment Anything

MoSA uses sparse motion-derived masks to train an appearance-based notion of objectness that can operate on static images without motion at inference. It then transfers PGM-generated masks to train whole-image and promptable segmentation models.

4 Learning from Motion to Segment Anything

  • PGM learns objectness from sparse motion masks and aims to discover unlabeled objects beyond the provided supervision.The intended inference setting is static images, without motion cues.Source: §4 · p. 7

4.1 Perceptual Grouping Model Architecture

  • PGM uses a ViT backbone with parallel projection heads assigned to pseudo-masks by bounding-box size, encouraging specialization across object scales.Copy-Paste augmentation is used to improve robustness to occlusions and complex scenes.Source: §4.1 · pp. 7–8

4.2 Perceptual Grouping Contrastive Learning (PGCL)

  • PGCL contrasts patch features so within-mask patches group together and patches outside the mask separate, addressing sparse labels and fixed-slot limitations.Its binary labels mark whether an anchor and another patch belong to the same mask; the loss is averaged across heads and sampled anchors.Source: §4.2 · p. 8, Figure 3 · p. 9

4.3 Adaptation to Segment Anything

  • PGM-generated masks on high-resolution static images train whole-image Mask2Former and click-prompt Semantic-SAM models for distinct segmentation tasks.The promptable model is trained with simulated positive clicks sampled from mask foregrounds.Source: §4.3 · p. 9, Training the Segment Anything Models · pp. 9–10

5 Experiments

Experiments assess PGM masks and MoSA’s whole-image and point-promptable segmentation, then ablate the learning objective, motion-label modules, prediction heads, and video-data scale.

5.1 Training Setups

5.2 Evaluation Datasets and Metrics

  • Evaluation covers seven whole-image benchmarks using AR and COCO Val2017 for point prompting using MaxIoU and OracleIoU.AR is used because dataset labels cover only subsets of objects while MoSA generates masks across heterogeneous levels and open-world categories.Source: §5.2 · pp. 10–11

5.3 Evaluation Results

  • PGM masks outperform UnSAM’s CascadePSP-refined pseudo-labels without refinement, and qualitative comparisons show whole-object and hand-drawn-image generalization.PGM segments an entire snail where DINOv3 activates on only part, and identifies a child figure in an out-of-distribution hand-drawn image.Source: Perceptual Grouping Masks · pp. 11–12, Table 1 · p. 11, Figure 4 · p. 11
  • 42.1% average AR: MoSA outperforms previous unsupervised methods across seven datasets and exceeds supervised SAM on PartImageNet and PACO by 24.4% and 10.3% in AR, respectively.It uses a ResNet-50 backbone and 1% of SA-1B training data; Table 2 reports whole-image results.Source: Whole-Image Segmentation · p. 12, Table 2 · p. 12
  • 41.6% MaxIoU and 63.4% OracleIoU on COCO: MoSA beats UnSAM by +1.3% and +3.9%, respectively, while supervised SAM scores 52.1% MaxIoU and 68.2% OracleIoU.MoSA requires 3.4× fewer parameters and 100× less SA-1B data than SAM.Source: Point-Based Promptable Segmentation · pp. 12–13, Table 3 · p. 13

5.4 Ablation Study

  • 28.3 AR: PGCL improves over Slot Attention’s 8.2 AR, while BCE+Dice fails when sparse motion labels suppress unlabeled static regions as background.The comparison uses identical backbone architectures and training data.Source: PGM Training · p. 13, Table 4 · p. 13
  • 36.6% AR1000: combining ROSM and MFSM performs best, with ROSM stronger on small objects and MFSM on large objects.ROSM alone reaches 20.6% ARs; MFSM reaches 43.1% ARl and 13.7% ARs.Source: PGM Training · p. 14, Table 5 · p. 14
  • 36.6% AR is reached with four heads versus 27.9% with one, while five heads score 36.5%; scaling video data from 100h to 1,000h to 10,000h raises AR from 16.8% to 28.3% to 36.6%.The reported data-scale trend is non-saturating over the tested range.Source: Number of Prediction Heads · p. 14, Impact of Pre-training Video Data Scale · p. 14

6 Conclusion

MoSA learns generalized segmentation from motion through a three-stage unsupervised framework. Its synthetic videos combine moving foreground objects with dynamic backgrounds and programmatically generated masks.

6 Conclusion

  • MoSA is a three-stage unsupervised framework for learning generalized segmentation from motion.The conclusion frames visual motion as the learning signal for perceptual grouping.Source: §6 · p. 14

A Synthetic Video Generation Details

  • Synthetic videos combine a dynamic background with multiple moving foreground objects, and their masks are generated programmatically without manual annotation.The synthesis process is described as unsupervised.Source: A Synthetic Video Generation Details · p. 19

A.1 Background

  • Backgrounds simulate camera motion through smooth B-spline trajectories and random perspective changes, with fast- and slow-camera scenarios paired with different foreground speeds.Fast camera motion pairs with slower foregrounds; slow camera motion pairs with faster foregrounds.Source: Motion Generation · p. 19, Motion Profile · p. 19

A.2 Foreground Objects

A.3 Synthetic Data Visualizations

The synthetic-data visualizations show how generated motion relates to segmentation targets. MGMS is the data foundation for producing motion pseudo-labels from large-scale unannotated videos.

A.3 Synthetic Data Visualizations

  • Synthetic examples prioritize diverse, complex motion captured by optical flow, rather than realistic textures or precise foreground shapes.The downstream model relies solely on optical flow as input.Source: §A.3 · p. 20
  • Fig. A1 shows sequential frames, central-frame optical flow, individual moving-object masks, and the complete motion-foreground mask.Each row is one example, designed to include substantial occlusion and complex object interactions.Source: §A.3 · p. 20, Figure A1 · p. 21

B Motion Pseudo-Labels Generation Details

  • MGMS is designed to automatically generate high-quality motion pseudo-labels from large-scale unannotated videos.The section describes its input video datasets, preprocessing, specialized modules, and outputs.Source: B Motion Pseudo-Labels Generation… · p. 21

B.1 Dataset Description

MoSA draws training videos from public datasets without using their human-provided annotations. It filters for motion, applies two segmentation modules, and post-processes their masks for quality and uniqueness.

B.1 Dataset Description

  • Training data are derived from Kinetics-700 (about 650,000 clips across 700 action classes), BDD100K (100,000 driving videos), and a random 5% sample of YouTube-8M.Source: §B.1 · pp. 21–22

B.2 Data Preprocessing

  • Videos are filtered for significant optical-flow motion, then seven frames from a central 1.5-second segment yield bidirectional flow inputs for ROSM and MFSM.Flow is computed with GMFlow; the 24 channels consist of six forward and six backward flows, each with two channels.Source: §B.2 · p. 22

B.3 Module Outputs

  • ROSM can segment rigid moving objects, parallax-visible static foreground objects, and object parts, while MFSM extracts broader moving foreground elements.Examples of MFSM outputs include pedestrians, animals, and human-object interactions.Source: §B.3 · p. 22

B.4 Data Postprocessing

  • Mask post-processing combines mask confidence and boundary sharpness, applies NMS, and removes image-label pairs with insufficient image sharpness.Masks below the quality threshold are discarded before blur filtering based on the variance of the masked image region’s Laplacian.Source: §B.4 · p. 23

C Model Implementation Details

MoSA generates motion-derived masks with MGMS, trains PGM to learn grouping from those masks, and distills that knowledge into automatic and promptable segmentation models. Its mask-filtering ablation reports higher average AR when maskness and boundary sharpness are combined.

C.1 MGMS Pipeline

C.2 Perceptual Grouping Model (PGM)

  • PGM learns grouping from motion-derived pseudo-masks using contrastive learning on about 10,000,000 video frames, with four heads assigned masks by bounding-box scale.Motion cues come from 10,000 hours of video; frames average 2.1 pseudo-masks each.Source: PGM Training Details · p. 25
  • PGM inference combines a global image pass with overlapping multi-scale tiles, then applies NMS and CRF refinement to the masks.The CRF uses original-image color information to sharpen boundaries.Source: PGM Inference Strategy · p. 26

C.3 Adaptation to Segment Anything

D.1 Mask Quality Filtering

  • 28.8 average AR with combined maskness and boundary sharpness, versus 24.1 with maskness alone and 22.8 with boundary sharpness alone.Table A3 reports AR across seven evaluation benchmarks and their average.Source: §D.1 · p. 27, Table A3 · p. 27

D.2 Video-Based Alternatives

MGMS is designed to extract multi-granularity motion masks for a transferable object prior, and its pseudo-labels support robust PGM scaling as video pre-training data grows. On DAVIS2017, it performs competitively with VideoCutLER and OCLR.

D.2 Video-Based Alternatives

  • 55.9 J &F on DAVIS2017 places MGMS close to VideoCutLER (57.3) and OCLR (55.1) in zero-shot video segmentation.MGMS and OCLR use only optical flow, while VideoCutLER uses RGB appearance cues.Source: §D.2 · p. 27, Table A4 · p. 28
  • MGMS targets multi-granularity motion masks for learning a transferable object prior, rather than only instance-level video segmentation.OCLR and VideoCutLER mainly target instance-level video segmentation.Source: §D.2 · p. 27
  • Motion-driven pseudo-labels yield consistent PGM improvements as video pre-training data grows, unlike the limited scaling trend with VideoCutLER pseudo-labels.The VideoCutLER trend is attributed to reliance on DINO-based appearance features and likely inherited static-appearance biases.Source: §D.2 · pp. 27–28, Figure A2 · p. 28

E Additional Qualitative Results

Additional qualitative results illustrate MoSA's whole-image and promptable image segmentation capabilities.

E Additional Qualitative Results

  • Fig. A5 presents additional qualitative results for whole-image segmentation, comparing MoSA with SAM and UnSAM.UnSAM's refinement stage uses the supervised CascadePSP refiner, so it is not strictly unsupervised.Source: E Additional Qualitative Results · p. 28
  • Fig. A6 presents comparative examples of promptable image segmentation for MoSA, SAM, and UnSAM.The figure describes point prompts and multi-granular masks from MoSA.Source: E Additional Qualitative Results · p. 28
  • Together, the figures provide qualitative examples of MoSA's segment-anything capabilities in whole-image and promptable image settings.Source: E Additional Qualitative Results · p. 28

F Limitations

Evaluation is limited to image datasets, while low-light optical-flow degradation can weaken supervision for fine structures; the paper identifies video evaluation and flow robustness as future work.

F Limitations

  • All evaluation experiments use image datasets; extending MoSA to video and temporal segmentation is left for future research.Source: F Limitations · p. 28
  • Low-light optical-flow degradation can weaken pseudo-label supervision for fine structures, and MoSA can struggle with subtle body parts such as dark-region legs.The paper identifies robustness to low-light videos and other flow-degraded conditions as future work.Source: F Limitations · pp. 28–29
  • SAM can still segment dark-region legs in the cited example, which the paper attributes to dense manual supervision.Source: F Limitations · p. 29
  • ROSM examples include rigid objects, articulated parts, smaller moving components, and static objects segmented using camera-motion parallax cues.Source: Figure A3 · p. 30
  • MFSM examples show extraction of moving foreground elements; the additional whole-image and promptable comparisons include UnSAM, whose refiner uses supervised CascadePSP.Source: Figure A4 · p. 31, Figure A5 · p. 32, Figure A6 · p. 33

Related papers

Loading 2609.39785v1…