Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks

Tianhe Ren; Shilong Liu; Ailing Zeng; Jing Lin; Kunchang Li; He Cao; Jiayu Chen; Xinyu Huang; Yukang Chen; Feng Yan; Zhaoyang Zeng; Hao Zhang; Feng Li; Jie Yang; Hongyang Li; Qing Jiang; Lei Zhang

Source-linked AI summary

Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks

Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kunchang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, Zhaoyang Zeng, Hao Zhang, Feng Li, Jie Yang, Hongyang Li, Qing Jiang, Lei Zhang

arXiv:2401.14159v1Jan 25, 2024cs.CV

TL;DR

Grounded SAM targets the challenge of precise open-set segmentation under limited high-quality segmentation data. It combines Grounding DINO’s text-conditioned open-set detection with SAM’s promptable segmentation and connects additional expert models. The resulting pipeline supports diverse visual tasks and reports 48.7 mean AP on the SegInW zero-shot benchmark with Grounding DINO-Base and SAM-Huge.

Problem
High-quality segmentation data is limited, making precise open-set segmentation in the wild challenging despite more tractable open-set detection.
Source: §3.2 · p. 3
Method
Grounded SAM first uses Grounding DINO to generate text-conditioned boxes and then applies SAM for promptable segmentation, enabling expert-model assembly.
Source: §1 · p. 2, §3.2 · p. 3
Results
48.7 mean AP is achieved on the SegInW zero-shot benchmark with Grounding DINO-Base and SAM-Huge.
Source: §4 · p. 7
Takeaways & Limitations
Assembled expert models extend Grounded SAM across automatic annotation, controllable image editing, and promptable human motion analysis.
Source: §1 · p. 2, §3.4 · p. 5, §3.5 · p. 5
Unified vision models remain limited to a restricted number of tasks and often underperform task-specific models.
Source: §2.2 · p. 3

Abstract

from arXiv · show

We introduce Grounded SAM, which uses Grounding DINO as an open-set object detector to combine with the segment anything model (SAM). This integration enables the detection and segmentation of any regions based on arbitrary text inputs and opens a door to connecting various vision models. As shown in Fig.1, a wide range of vision tasks can be achieved by using the versatile Grounded SAM pipeline. For example, an automatic annotation pipeline based solely on input images can be realized by incorporating models such as BLIP and Recognize Anything. Additionally, incorporating Stable-Diffusion allows for controllable image editing, while the integration of OSX facilitates promptable 3D human motion analysis. Grounded SAM also shows superior performance on open-vocabulary benchmarks, achieving 48.7 mean AP on SegInW (Segmentation in the wild) zero-shot benchmark with the combination of Grounding DINO-Base and SAM-Huge models.

1. Introduction

Grounded SAM addresses the lack of robust open-set segmentation by combining open-set detection with promptable segmentation. Its assembled expert-model pipeline extends to automatic annotation and precise image editing.

Open-world applications require robust and versatile visual perception models for interpreting and interacting with complex environments.Source: §1 · p. 2
Existing unified-model approaches have limited data scope, particularly for complex tasks such as open-set segmentation.Source: §1 · p. 2
Grounded SAM divides open-set segmentation into open-set detection and promptable segmentation using Grounding DINO and SAM.Source: §1 · p. 2
The pipeline facilitates efficient fusion of different expert models for more intricate open-world tasks.Source: §1 · p. 2
Integrating Recognize Anything or BLIP enables automatic image annotation, while Stable Diffusion supports highly precise image editing.Source: §1 · p. 2

2. Related Work

Related work includes broad computer-vision systems and unified vision models spanning diverse modalities and tasks. However, unified models remain limited in task coverage and often underperform task-specific models, while model-assembling alternatives avoid LLM control.

Computer vision research spans recognition, detection, segmentation, tracking, generation, editing, human-centric understanding, and motion generation.Source: §2.1 · p. 2

Unified Models

Unified vision models seek either to merge vision modalities into one representation or to support different modality outputs.Source: §2.2 · p. 2
Existing unified vision models handle only a limited number of tasks and often fall short of task-specific models in performance.Source: §2.2 · p. 3

Foundation-model assembling decouples complex tasks and enables step-by-step visual reasoning without an LLM controller or training.Source: §2.3 · p. 3

3. Grounded SAM Playground

Grounded SAM combines open-set detection with promptable segmentation, then connects this foundation to expert models for automatic annotation, image editing, and promptable human motion analysis.

More Extensions for Grounded SAM

Grounded SAM serves as a foundation for fusing expert models from different domains into more comprehensive visual-task pipelines.The chapter presents model combinations spanning annotation, image editing, and human motion analysis.Source: §3 · p. 3, §3.6 · p. 6

Grounded-SAM: Open-Vocabulary Det & Seg

Grounded SAM combines Grounding DINO’s open-set detection with SAM’s segmentation to address text-guided open-set segmentation.Grounding DINO generates boxes from text prompts, which SAM uses as box prompts for precise masks.Source: §3.2 · pp. 3–4, §3.1 · p. 3
Grounded SAM detects and segments user-specified objects, including long-tail categories such as Zale Horrida and Gazania Linearis.The demonstrations include examples sampled from the V3Det dataset.Source: Figure 2 · p. 4, §3.2 · p. 4

RAM-Grounded-SAM: Automatic Dense Image Annotation

RAM-Grounded-SAM combines RAM tags with Grounded SAM to automatically generate precise boxes and masks for image instances without textual input.Image captions or tags can be fed into Grounded SAM to support dense automatic labeling.Source: §3.1 · p. 3, §3.3 · p. 4, Figure 3 · p. 5

Grounded-SAM-OSX: Promptable Human Motion Analysis

Grounded-SAM-OSX uses text-prompted Grounded SAM boxes to select a specific person before OSX estimates that person’s whole-body mesh.This changes human motion analysis from instance-agnostic detection to promptable, instance-specific analysis.Source: §3.1 · p. 3, §3.5 · p. 5, Figure 5 · p. 6

Grounded-SAM-SD: Highly Controllable Image Editing

Grounded-SAM-SD combines Grounded SAM’s open-set ability with inpainting for highly controllable image editing.The broader pipeline supports precise region selection through interactive inputs and grounding with text prompts.Source: §3.4 · p. 5, Figure 4 · p. 6

4. Effectiveness of Grounded SAM

Grounded SAM is evaluated on the SGinW zero-shot benchmark, where Grounding DINO and SAM-Huge combinations improve performance over earlier unified open-set segmentation models.

SGinW comprises 25 zero-shot in-the-wild datasets used to evaluate Grounded SAM.Source: §4 · p. 7
Grounding DINO Base and Large Model combined with SAM-Huge significantly improve SGinW zero-shot performance over UNINEXT and OpenSeeD.Source: §4 · p. 7

5. Conclusion and Prospects

Grounded SAM’s model-assembly approach expands task coverage by combining expert models, while decomposing tasks into observable subtasks. The authors also propose annotation, language-driven execution, and cross-modal dataset-generation prospects.

Conclusion and Prospects

With n expert models, model combinations can support up to 2n −1 tasks instead of n tasks.The approach decomposes complex tasks into subtasks handled by available expert models.Source: §5 · p. 7
Decomposing tasks into subtasks makes the assembly pipeline more explainable because intermediate outputs expose the reasoning process.Source: §5 · p. 7
Combining expert models can reduce annotation costs, while human annotators filter or fine-tune inaccurate predictions before continual model training.Source: §5 · p. 7
Assembled models with varied input and output modalities could be invoked by large language models through language prompts to execute computer-vision tasks.Source: §5 · p. 7

6. Contributions and Acknowledgments

The acknowledgments list project contributors and their roles across Grounded SAM pipelines, demonstrations, tracking, motion analysis, editing, and model optimization.

Contributions and Acknowledgments

Tianhe Ren and Shilong Liu contributed to the Grounded SAM pipeline and its online or model-related development.Source: §6 · pp. 7–8
Ailing Zeng and Jin Ling led the Grounded-SAM-OSX pipeline and support or code optimization.Source: §6 · p. 8
He Cao led Grounded-SAM-SD and Interactive SAM Editing, while Kunchang Li led the BLIP-Grounded-SAM pipeline.Source: §6 · p. 8
Xinyu Huang led the RAM-Grounded-SAM demonstration, Jiayu Chen led the Grounded SAM ModelScope demo, and contributors are listed in randomized order within roles.Source: §6 · pp. 7–8
Feng Yan led Grounded SAM with VISAM tracking, while Yukang Chen led 3D-Box via Segment Anything.Source: §6 · p. 8