Source-linked AI summary
Segment and Track Anything
Yangming Cheng, Liulei Li, Yuanyou Xu, Xiaodi Li, Zongxin Yang, Wenguan Wang, Yi Yang
TL;DR
Video segmentation needs temporal coherence, semantic understanding, and interaction modes suited to diverse object-selection requirements. SAM-Track combines SAM, DeAOT, and Grounding-DINO to provide multimodal and automatic tracking, with reported effectiveness on DAVIS benchmarks and diverse applications.
Problem
Applying image-based SAM directly to video produces suboptimal results because it lacks temporal coherence and sufficient semantic information for language-guided tasks.
Method
SAM-Track combines SAM-based keyframe segmentation, DeAOT tracking, and Grounding-DINO to support interactive multimodal selection and automatic tracking of new objects.
Results
SAM-Track shows promising performance in DAVIS benchmarks and application-based experiments validating its effectiveness, efficiency, and real-world promise.
Takeaways & Limitations
SAM-Track provides two tracking modes and flexible interactions for requirements across fields including medical imaging, autonomous driving, and augmented reality.
Takeaways & Limitations
Directly applying image-based SAM to video remains limited by its lack of temporal coherence and semantic labels.
Abstract
from arXiv · showhide
This report presents a framework called Segment And Track Anything (SAMTrack) that allows users to precisely and effectively segment and track any object in a video. Additionally, SAM-Track employs multimodal interaction methods that enable users to select multiple objects in videos for tracking, corresponding to their specific requirements. These interaction methods comprise click, stroke, and text, each possessing unique benefits and capable of being employed in combination. As a result, SAM-Track can be used across an array of fields, ranging from drone technology, autonomous driving, medical imaging, augmented reality, to biological analysis. SAM-Track amalgamates Segment Anything Model (SAM), an interactive key-frame segmentation model, with our proposed AOT-based tracking model (DeAOT), which secured 1st place in four tracks of the VOT 2022 challenge, to facilitate object tracking in video. In addition, SAM-Track incorporates Grounding-DINO, which enables the framework to support text-based interaction. We have demonstrated the remarkable capabilities of SAM-Track on DAVIS-2016 Val (92.0%), DAVIS-2017 Test (79.2%)and its practicability in diverse applications. The project page is available at: https://github.com/z-x-yang/Segment-and-Track-Anything.
1 Introduction
Video segmentation must accommodate diverse interaction needs across applications, but image-based SAM does not preserve temporal coherence or provide sufficient semantic understanding for video. SAM-Track addresses these limitations with unified multimodal and automatic tracking modes, supported by experiments and application studies.
- Video segmentation supports applications including drones, autonomous driving, medical imaging, augmented reality, and biological analysis, creating demand for multiple interaction modes.
- SAM provides flexible point, box, and text prompts and strong zero-shot image-segmentation performance.
- SAM-Track extends SAM into a unified video framework that uses keyframe segmentation as reference for DeAOT tracking.
- SAM-Track offers interactive click, drawing, and text selection alongside automatic tracking of new objects appearing during videos.
- Extensive DAVIS benchmarks and application-based experiments validate SAM-Track’s effectiveness, efficiency, and promise for real-world scenarios.
2 Preliminaries
SAM-Track combines SAM, DeAOT, and Grounding-DINO within a video-tracking pipeline. DeAOT supplies multi-object tracking, while Grounding-DINO enables language-based object detection and selection.
- DeAOT associates multiple targets in a shared embedding space, enabling multi-object tracking at the speed of tracking a single object.
- DeAOT uses hierarchical gated propagation to preserve object-agnostic visual information during deep propagation.
- DeAOT achieved first place in four tracks of the VOT2022 challenge.
- The SAM-Track pipeline uses interactive tracking for first-frame annotations and automatic tracking every nth frame thereafter.
- Grounding-DINO detects target objects from text categories or detailed descriptions and returns an external rectangle for each target.
3 Methodology
SAM-Track is a unified video segmentation framework combining interactive, automatic, and fusion tracking modes. It uses SAM-based annotations with DeAOT tracking, Grounding-DINO for language-guided selection, and methods for incorporating newly appearing objects.
- Interactive tracking mode: Grounding-DINO detects target objects from text descriptions, SAM converts their boxes into masks, and DeAOT tracks the resulting objects through the video.This integration provides multimodal interaction for selecting objects in the reference frame.
- Automatic tracking mode: Automatic tracking uses Segment Everything or Object of Interest Segmentation to obtain annotations for new objects at every nth frame before DeAOT tracks them.The Object of Interest Segmentation method uses Grounding-DINO and SAM, while Segment Everything obtains masks for every object in a key reference frame.
- Automatic tracking mode: CMR defines an object as new when its size ratio between the SAM annotation and new-object mask exceeds threshold t, reducing interference with existing object identities.The new-object mask is obtained from the DeAOT tracking background and SAM annotation result.
- Framework and tracking modes: SAM-Track combines interactive and automatic tracking, allowing first-frame selections and tracking of new objects appearing later.Fusion tracking combines both modes selectively to support diverse application requirements.
- Evaluation setup: Table 1 evaluates SAM-Track on single-object DAVIS-2016-Val and multi-object DAVIS-2017-Test using the R50-DeAOT-L tracking model.The supplied table caption identifies the benchmark types and tracking model but does not provide the numerical rows.
4 Experiments
Experiments evaluate SAM-Track on DAVIS benchmarks and across multiple application domains. The reported qualitative results show simultaneous multi-object tracking in complex scenarios and use interactive, automatic, and fusion modes.
- Benchmark evaluation: SAM-Track is evaluated on DAVIS-2016 Val and DAVIS-2017 Test, with reference-frame annotations obtained through mouse clicking.The experiments use two popular video object segmentation benchmarks to assess effectiveness and efficiency.
- Cross-domain applications: Across different domains, the application examples use interactive tracking, automatic tracking, and fusion tracking modes.The paper presents these modes as representative qualitative results across diverse fields.
5 Applications
SAM-Track demonstrates practical video object segmentation across diverse application fields, using interactive and automatic tracking modes to accommodate different object-selection and tracking requirements.
- SAM-Track provides two tracking modes and varied interactions for applications with different requirements.Figure 3 reports representative results across diverse fields using different tracking modes and interaction methods.
- Sports Analysis: For sports analysis, a preset “soccer players” text prompt and clicking can support continuous tracking of players and the football field.
- Medical Field: In medical applications, clicking enables zero-shot tracking of rare cells and organs without specialized tracker training.
- Smart City: SAM-Track’s automatic mode addresses smart-city videos where new vehicles continually appear.
- Autonomous Driving: SAM-Track can track numerous objects, including pedestrians, pets, and other precise objects, supporting autonomous-driving applications.
6 Conclusion
SAM-Track is presented as a unified video segmentation model that combines multimodal interaction with fast multi-object tracking, including objects appearing during a video. Its efficiency and versatility support applications with different requirements and its use as a baseline for real-world VOS applications.
- SAM-Track supports multimodal interactions and can track new objects of interest that appear midway through a video.
- The efficient DeAOT tracking mode enables fast inference while tracking multiple objects.
- SAM-Track’s efficiency and versatility make it applicable across fields with different requirements.
- The authors propose SAM-Track as a reliable baseline intended to accelerate real-world applications of video object segmentation models.