Source-linked AI summary
OMG-Seg: Is One Model Good Enough For All Segmentation?
Xiangtai Li, Haobo Yuan, Wei Li, Henghui Ding, Size Wu, Wenwei Zhang, Yining Li, Kai Chen, Chen Change Loy
TL;DR
Most segmentation models address individual tasks, and existing unified approaches still face limits in task coverage and performance. OMG-Seg uses a shared query representation and decoder with joint training across image and video datasets; it reports comparable performance across four settings using one model.
Problem
Most deep segmentation models focus on individual tasks, and existing unified approaches often lack video or interactive segmentation or trail task-specific models.
Source: §1 · p. 2Method
OMG-Seg uses a shared encoder-decoder with unified task queries and jointly trains segmentation tasks on combined image and video datasets.
Source: §3 · p. 3, §3.3 · p. 5Results
OMG-Seg achieves comparable results across image, video, open-vocabulary, and interactive segmentation settings on eight datasets using one shared model.
Source: §1 · p. 2Takeaways & Limitations
The framework supports competitive performance across ten segmentation tasks while reducing parameter size and specialized model-design needs.
Source: §5 · p. 8The stated scope is primarily pure visual, 2D segmentation, excluding medical and referring segmentation.
Source: §1 · p. 2
Abstract
from arXiv · showhide
In this work, we address various segmentation tasks, each traditionally tackled by distinct or partially unified models. We propose OMG-Seg, One Model that is Good enough to efficiently and effectively handle all the segmentation tasks, including image semantic, instance, and panoptic segmentation, as well as their video counterparts, open vocabulary settings, prompt-driven, interactive segmentation like SAM, and video object segmentation. To our knowledge, this is the first model to handle all these tasks in one model and achieve satisfactory performance. We show that OMG-Seg, a transformer-based encoder-decoder architecture with task-specific queries and outputs, can support over ten distinct segmentation tasks and yet significantly reduce computational and parameter overhead across various tasks and datasets. We rigorously evaluate the inter-task influences and correlations during co-training. Code and models are available at https://github.com/lxtGH/OMG-Seg.
1. Introduction
Most segmentation models target individual tasks, while unified approaches have struggled to cover video and interactive segmentation or match task-specific performance. OMG-Seg uses shared query and decoder representations and achieves comparable results across image, video, open-vocabulary, and interactive settings.
1. Introduction
- Recent progress has expanded segmentation from image semantic, instance, and panoptic tasks to video and prompt-driven interactive tasks.Open-vocabulary segmentation extends recognition to novel objects using text descriptions.Source: §1 · p. 2
- Most deep segmentation models focus on one task, motivating a generalizable model for a broader range of segmentation tasks.The paper argues that training on large, varied data corpora can enhance adaptability and effectiveness across tasks.Source: §1 · p. 2
- Prior unified models often lack video or interactive segmentation, while more general vision frameworks lag behind task-specific segmentation models.The passage cites transformer-based approaches integrating open-vocabulary and multi-dataset segmentation, but notes limits in broader task coverage.Source: §1 · p. 2
- The paper scopes its claim primarily to pure visual, 2D segmentation, excluding medical and referring segmentation.It says those tasks could be integrated with appropriate input adaptations.Source: §1 · p. 2
- A shared encoder-decoder processes unified queries representing mask labels, image or tube masks, unique IDs, and visual prompts.Co-training on combined image and video datasets supports up to ten segmentation tasks across datasets.Source: §1 · p. 2
- OMG-Seg achieves comparable results across image, video, open-vocabulary, and interactive segmentation settings on eight datasets with one shared model.The authors describe this as the first single model to cover all four settings.Source: §1 · p. 2
2. Related Work
Prior work has unified subsets of image, video, and prompt-driven segmentation, but comprehensive joint training across these domains remains unexamined in the cited literature. The section also situates open-vocabulary segmentation and visual foundation models within related research.
Universal Image/Video Segmentation
- Table 1 compares representative models by segmentation setting; prior methods unify subsets, but the cited work had not thoroughly investigated joint training across image, video, and prompt-driven data.OneFormer unifies three image tasks, UNINEXT focuses on instance-level tasks, and TarVIS combines video tasks using target prompts.Source: §2 · p. 2, Table 1 · p. 3
Visual Foundation Models and Open Vocabulary Segmentation
- Visual foundation models include vision-only, vision-language, and visual-prompting approaches; SAM demonstrates generalizable segmentation, while such models typically fall short on video segmentation.Open-vocabulary methods use vision-language representations to recognize novel classes from text embeddings.Source: §2 · p. 3
Unified Modeling
- Bridging the performance gap between unified segmentation models and purpose-built models remains an open problem.The cited generalist methods primarily unify tasks across domains.Source: §2 · p. 3
3. Methodology
OMG-Seg aims to unify diverse segmentation tasks in one versatile model, using a shared query-based framework and decoder. It trains across image and video datasets jointly and adapts decoding to task-specific inputs.
3.1. Unified Task Representation
- Image, video, interactive, and open-vocabulary segmentation outputs are represented as object queries carrying masks, labels, IDs, or prompts in a shared framework.Video queries can represent tracked tube masks; visual prompts are encoded as object queries, while open-vocabulary classification uses CLIP text embeddings.Source: §3 · p. 3, §3.1 · pp. 4–5
- CLIP text embeddings classify masks across datasets, avoiding fixed label spaces and taxonomy conflicts.The model measures distance between visual query features and class embeddings rather than using a learned classifier.Source: §3.1 · p. 5
3.2. OMG-Seg Architecture
- A frozen CLIP visual encoder, pixel decoder, and shared multi-task mask decoder process semantic and location queries.Semantic queries represent image masks or tracked tubes; location queries encode visual prompts, which bypass self-attention and are conditioned on prompts and image content.Source: Figure 2 · p. 4, §3.2 · p. 5
3.3. Training and Inference
- Joint co-training matches predicted queries to image and video entity masks, then supervises classification and mask predictions with classification, cross-entropy, and Dice losses.Entities include things, stuff, and class-agnostic masks; instances sharing an ID form tube masks.Source: §3.3 · p. 5
- Inference uses query matching for video instance IDs and mask matching for video object segmentation; combining prompts with video or open-vocabulary settings supports additional applications.Interactive inference takes box or point prompts to produce binary masks.Source: §3.3 · p. 6
4. Experiments
Experiments compare OMG-Seg across image, video, open-vocabulary, and SAM-like settings, then examine multi-dataset co-training and design choices. Results show broad competitive performance and parameter reduction, alongside task-specific trade-offs.
4.1. Main Results
- OMG-Seg performs on par with leading image and video methods, beats a vision generalist, but trails some COCO specialists while covering more segmentation settings.The authors report near-state-of-the-art VIPSeg results and favorable open-vocabulary comparisons, including outperforming ODISE in some scenarios.Source: §4.1 · p. 6, Table 2 · p. 7
- Joint co-training generally improved three VPS and VIS datasets and reduced model parameters from 1326M to 221M, but significantly diminished ADE-20k performance.The study used five datasets for balanced co-training over 12 epochs.Source: §4.1 · p. 7, Table 3 · p. 7
- With a pre-trained backbone, image segmentation improved while video segmentation efficacy declined slightly, which the authors attribute to unbalanced optimization objectives.They suggest increasing the volume of video training examples could potentially address this issue.Source: §4.1 · p. 7
- Qualitative examples show segmentation and tracking across tasks, including open-vocabulary video instance segmentation that identifies a lizard class absent from training.The examples use a ConvNeXt-Large model across five tasks.Source: §4.1 · p. 7
4.2. Ablation Study and Analysis
- COCO, VIPSeg, and YouTube-VIS together yielded the preferred performance balance; adding VIPSeg alone slightly reduced COCO performance but improved both video datasets.COCO-only training also produced zero-shot results on other datasets, attributed to frozen CLIP visual features.Source: Table 4 · p. 8, §4.2 · p. 8
- A separate video decoder slightly decreased performance; shared decoding better aligned optimization, particularly benefiting video datasets with short clips.Image datasets were trained using pseudo-video samples.Source: Table 5 · p. 8, §4.2 · p. 8
- Adapters helped with fewer training epochs but not extended training, while larger CLIP models and longer training improved all three datasets and converged by 24 epochs.The authors chose not to add adapters, retaining a simpler framework.Source: Table 6 · p. 8, §4.2 · p. 8, Table 7 · p. 8
5. Conclusion
OMG-Seg combines image, video, open-vocabulary, and interactive segmentation in one co-trained framework. It supports more than ten tasks with competitive performance while reducing parameter size and specialized engineering.
5. Conclusion
- OMG-Seg uses a unified query representation and shared decoder in a joint co-training framework for image, video, open-vocabulary, and interactive segmentation.Source: §5 · p. 8
- A single model performs across more than ten segmentation tasks with competitive performance compared with task-specific models.Source: §5 · p. 8
- The framework reduces parameter size and the need for specialized engineering across segmentation applications.Source: §5 · p. 8
6. Appendix
The appendix adds method details, experiment results, and demonstrations spanning image, video, open-vocabulary, and interactive segmentation.
A. More Method Details
The appendix compares OMG-Seg with partial or task-specific systems and details its training and inference choices. It describes shared capabilities across image and video tasks, open-vocabulary segmentation, and interactive prompting.
A. More Method Details
- Compared with specific models, OMG-Seg uses extreme parameter sharing and performs tasks those models cannot.Source: A. More Method Details · p. 9
- Joint training on image and video datasets supports open-vocabulary and interactive segmentation alongside image segmentation, without video-specific tuning.Source: A. More Method Details · p. 9
- Compared with partial unified models, OMG-Seg adds capabilities including video segmentation beyond Semantic-SAM [46], interactive and panoptic segmentation beyond UNINEXT, and video and interactive segmentation beyond FreeSeg.Source: A. More Method Details · p. 9
- Training balances dataset sampling to the COCO dataset size, uses Mask2Former augmentation, and keeps class-name text embeddings fixed for training and inference.Source: A. More Method Details · p. 9
- Video inference uses query matching without an extra tracking-query embedding; open-vocabulary inference fuses frozen CLIP visual scope with predicted scope.Source: A. More Method Details · p. 9
- The appendix includes ResNet50- and ViT-backbone results, and its self-attention ablation table describes masks that filter query correlations.Source: Table 8 · p. 9, Table 9 · p. 9, Table 10 · p. 9
- Interactive evaluation mainly uses point prompts, while box prompts can also be used; frozen CLIP features support open-vocabulary interactive masks.Source: A. More Method Details · p. 9
B. More Experiment Results
Additional experiments report backbone and interactive-attention findings. Frozen CLIP-ViT performs worse than the explored alternatives, while allowing query interaction in interactive mode produces a performance drop.
B. More Experiment Results
- A jointly co-trained ResNet50 model trained for 24 epochs achieves considerable results with less parameter cost than Mask2Former trained for 50 epochs.Source: B. More Experiment Results · p. 9
- Frozen CLIP-ViT leads to inferior results because fixed 224-position embeddings and bilinear upsampling hurt the original representation; learned architecture still leaves a gap with convolution-based CLIP.Source: B. More Experiment Results · p. 9
- Fine-tuning without frozen CLIP removes the open-vocabulary ability.Source: B. More Experiment Results · p. 9
- Interactive-mode query masking prevents queries from interacting during cross-attention; without it, performance drops significantly on COCO-SAM and COCO-PS.The authors attribute the drop to global information adding noise when local features are sufficient for interactive segmentation.Source: B. More Experiment Results · p. 10
C. More Visualization Example
Figure 4 adds open-vocabulary panoptic and interactive examples, while the discussion identifies a model-capacity limitation and future scaling directions.
C. More Visualization Example
- ADE-20k examples show good zero-shot quality for open-vocabulary panoptic segmentation.These examples appear in Figure 4’s top row.Source: C. More Visualization Example · p. 10
- ImageNet-1k interactive segmentation uses class labels from a simple CLIP score, yielding open-vocabulary interactive segmentation.Source: C. More Visualization Example · p. 10
- The model’s capacity is a limitation: keeping its architecture frozen to preserve open-vocabulary ability leads to inferior results for one specific dataset or task.Source: C. More Visualization Example · p. 10
- Future work proposes more dataset co-training with the learned backbone to improve performance, and more text-image pairs or classification data to scale the model.The authors also describe improved performance on close sets alongside open-vocabulary segmentation ability.Source: C. More Visualization Example · p. 10
- A text path is proposed as future work to support language-driven segmentation, including referring image or video segmentation and joint reasoning with large language models.Source: C. More Visualization Example · p. 10