Source-linked AI summary
Segment Everything Everywhere All at Once
Xueyan Zou, Jianwei Yang, Hao Zhang, Feng Li, Linjie Li, Jianfeng Wang, Lijuan Wang, Jianfeng Gao, Yong Jae Lee
TL;DR
Segmentation research lacks a universal interface that can handle diverse prompts and tasks in one model. SEEM introduces a multimodal decoder with visual and text prompting in a joint visual-semantic space, plus interactive memory. The single model achieves competitive performance across interactive, generic, referring, and video object segmentation, while generalizing to novel prompt combinations.
Problem
Existing segmentation models lack a universal interface for accommodating varied human prompts and different segmentation tasks.
Method
SEEM uses a multimodal encoder-decoder with visual, text, and memory prompts interacting through a joint visual-semantic space.
Results
SEEM achieves competitive performance across generic, referring, interactive, and video object segmentation, including a 10.5 cIoU improvement from adding a visual compositional prompt for referring segmentation.
Takeaways & Limitations
SEEM provides a universal image-segmentation interface that supports diverse prompt types and their compositions, including open-vocabulary and cross-image visual prompts.
Abstract
from arXiv · showhide
In this work, we present SEEM, a promptable and interactive model for segmenting everything everywhere all at once in an image, as shown in Fig.1. In SEEM, we propose a novel decoding mechanism that enables diverse prompting for all types of segmentation tasks, aiming at a universal segmentation interface that behaves like large language models (LLMs). More specifically, SEEM is designed with four desiderata: i) Versatility. We introduce a new visual prompt to unify different spatial queries including points, boxes, scribbles and masks, which can further generalize to a different referring image; ii) Compositionality. We learn a joint visual-semantic space between text and visual prompts, which facilitates the dynamic composition of two prompt types required for various segmentation tasks; iii) Interactivity. We further incorporate learnable memory prompts into the decoder to retain segmentation history through mask-guided cross-attention from decoder to image features; and iv) Semantic-awareness. We use a text encoder to encode text queries and mask labels into the same semantic space for open-vocabulary segmentation. We conduct a comprehensive empirical study to validate the effectiveness of SEEM across diverse segmentation tasks. Notably, our single SEEM model achieves competitive performance across interactive segmentation, generic segmentation, referring segmentation, and video object segmentation on 9 datasets with minimum 1/100 supervision. Furthermore, SEEM showcases a remarkable capacity for generalization to novel prompts or their combinations, rendering it a readily universal image segmentation interface.
1 Introduction
SEEM addresses the lack of a universal segmentation interface by unifying diverse human prompts and segmentation tasks in one promptable model. Its prompting scheme and decoder target flexible, interactive, compositional, and semantically aware segmentation.
- A universal segmentation interface is needed to accommodate varied human prompts and different segmentation tasks within one promptable model.
- SEEM combines a Transformer encoder-decoder with an extra text encoder to process multimodal inputs through a multimodality-in-multimodality-out interface.
- Its prompting scheme encodes user intents into a joint visual-semantic space, supporting flexibility across tasks and generalization to unseen prompt combinations.
- SEEM is presented as a universal interactive interface with versatility, compositionality, interactivity, and semantic awareness across multiple segmentation tasks.
- The decoder overview shows image, text, and human inputs becoming queries, features, and prompts before producing class and mask embeddings.Its machine loop memorizes history masks, while its human loop supplies corrections for the next round.
2 Related Work
Prior work developed specialized or unified models for interactive, generic, and multi-task segmentation, but these approaches differ in prompt types, semantic capability, and generalization strategy. SEEM is situated among efforts to make segmentation models more flexible across tasks and distributions.
- Interactive segmentation: Interactive segmentation uses user inputs such as clicks, boxes, polygons, and scribbles to segment objects.
- Interactive segmentation: SAM offers strong zero-shot interactive segmentation but produces semantically unlabelled masks and supports fewer prompt types than SEEM.
- Generic segmentation: Generic segmentation includes instance, semantic, and panoptic subtasks that operate at different semantic levels.
- Unified vision models: Unified vision models target multiple vision tasks or data distributions through multi-task training or zero-shot generalization strategies.
3 Method
SEEM uses a prompt-aware encoder-decoder to predict masks and semantic concepts from visual, textual, and memory prompts. Its design supports diverse spatial inputs, prompt composition, iterative refinement, and semantic labeling.
- Model Design: SEEM-Decoder predicts masks and semantic concepts from image features using object, text, and visual queries that interact with text, visual, and memory prompts.The decoder is initialized from image features and produces mask and class outputs through query-prompt interactions.
- Compositional: Learnable queries are duplicated across generic, referring, and interactive segmentation, enabling zero-shot composition of prompt types at inference.Visual and textual prompts can be concatenated even though that combination was not used during training.
- Versatile: Visual prompts unify points, boxes, scribbles, polygons, and referred regions by sampling feature vectors from user-specified image regions.The sampler uses features from either the target image or a referred image and interpolates at most 512 vectors per region.
- Interactive: Memory prompts retain previous-mask information through mask-guided cross-attention, restricting image-feature interaction to regions specified by the prior mask.Updated memory prompts then interact with other prompts to convey segmentation history to the current round.
- Semantic-aware: Visual query outputs receive semantic labels through alignment with textual features, despite interactive segmentation being trained without semantic labels.The calculated logits are aligned through the joint visual-semantic space.
- Model Pipeline and Loss Functions: SEEM is trained with a linear combination of panoptic, referring, and interactive segmentation losses, including cross-entropy, binary cross entropy, and dice terms.The supplied loss weights assign separate coefficients to the panoptic, referring, and interactive objectives.
4 Experiments
SEEM is evaluated as one model across generic, referring, interactive, and zero-shot video object segmentation, including varied prompt types and compositional interactions. Experiments report competitive or improved performance, broad prompt generalization, and gains from selected training choices.
- Main Results: SEEM is evaluated across generic, referring, interactive, and zero-shot video object segmentation using shared parameters and diverse prompt types.The experiments include panoptic, instance, semantic, referring, interactive, and video object segmentation settings.
- Generic segmentation: Around 10 points on panoptic segmentation metrics separate SEEM from strong generalist baselines including Painter and SegGPT.The comparison concerns generic segmentation performance against UViM, Pix2Seqv2, Painter, and SegGPT.
- Referring segmentation: 10.5 cIoU points improve referring segmentation when a visual compositional prompt is added.The passage reports this gain relative to referring segmentation without the added visual compositional prompt.
- Interactive segmentation: SEEM achieves comparable performance with specialized interactive models and better performance than SAM despite SAM using 100× more segmentation data.It also supports text, points, scribbles, boxes, and images within the same interface.
- Interactive segmentation: SEEM achieves the best 1-IoU performance over three datasets with extremely limited clicks across point, stroke, scribble, and box prompts.1-IoU is defined as the mean IoU of all images with a single click.
- Video object segmentation: Zero-shot video object segmentation reaches close performance to a fully supervised DAVIS17 method without observing DAVIS/VOS data, while interactive DAVIS16 performance is comparable with supervised baselines after one first-frame click.The model transfers visual prompts from another image or the first video frame without modification.
- Ablations: Increasing interactive training iterations from 1 to 5 improves NoC0.9 from 5.41 to 4.59, so the main results use iteration 3 to limit computation.The first N-1 iterations are performed without gradient, and computation cost increases with more clicks.
- Qualitative Results: SEEM simultaneously produces masks and semantic labels from clicks or scribbles, labels unseen categories with a candidate vocabulary, and generalizes visual references across domains and changing video appearances.Reported examples include cartoons, games, plush toys, grassland scenes, blur, and intensive deformation.
5 Conclusion
SEEM unifies open-vocabulary and interactive image segmentation through diverse visual, textual, and referring-region prompts. Experiments report competitive benchmark performance and robust generalization to varied user intents.
- SEEM segments all semantics and pixels while supporting multiple prompt compositions.Its interface accepts visual, textual, and referring-region prompts.
- The prompt encoder maps diverse visual prompts into a joint visual-semantic space for flexible composition.The conclusion lists clicks, boxes, polygons, scribbles, text, and referring regions from another image.
- Extensive experiments show competitive performance on open-vocabulary and interactive segmentation benchmarks.Further studies report robust generalization to diverse user intents.