Source-linked AI summary
Track Anything: Segment Anything Meets Videos
Jinyu Yang, Mingqi Gao, Zhe Li, Shang Gao, Fangjing Wang, Feng Zheng
TL;DR
Video tracking and segmentation remain difficult to initialize and maintain consistently without costly annotation, while SAM’s image strength does not transfer directly to videos. TAM combines interactive SAM prompting with XMem temporal propagation, refinement, and correction, achieving reported one-pass results with minimal clicks. The method performs well on DAVIS benchmarks and complex qualitative cases, while long-video failures indicate limits in SAM refinement and the underlying video models.
Problem
Video tracking and segmentation require costly annotation or precise initialization, and SAM performs poorly on consistent video segmentation because of limited temporal correspondence.
Method
TAM interactively initializes targets with SAM, tracks them with XMem, refines masks with SAM, and adds human correction when needed.
Results
TAM obtains J&F scores of 88.4 on DAVIS-2016-val and 73.1 on DAVIS-2017-test-dev with click initialization and one-pass evaluation.
Takeaways & Limitations
TAM supports interactive tracking and segmentation of arbitrary video objects with very little human participation and one-pass inference.
Takeaways & Limitations
Failed cases arise in long videos, where SAM refinement is less effective than expected and current VOS models can produce mask shrinkage or insufficient refinement.
Abstract
from arXiv · showhide
Recently, the Segment Anything Model (SAM) gains lots of attention rapidly due to its impressive segmentation performance on images. Regarding its strong ability on image segmentation and high interactivity with different prompts, we found that it performs poorly on consistent segmentation in videos. Therefore, in this report, we propose Track Anything Model (TAM), which achieves high-performance interactive tracking and segmentation in videos. To be detailed, given a video sequence, only with very little human participation, i.e., several clicks, people can track anything they are interested in, and get satisfactory results in one-pass inference. Without additional training, such an interactive design performs impressively on video object tracking and segmentation. All resources are available on {https://github.com/gaomingqi/Track-Anything}. We hope this work can facilitate related research.
1 Introduction
The paper targets labor-intensive video tracking and segmentation, where existing methods require large annotated datasets or precise mask initialization. It proposes interactive TAM, combining SAM-based prompting with video tracking to achieve one-pass perception using minimal human participation.
- Motivation: Existing video trackers and segmenters depend on large manually annotated datasets and bounding-box or mask initialization.Semi-supervised VOS may specifically require object-mask ground truth for initialization.
- Motivation: SAM provides strong image segmentation and flexible interactive prompts, but direct video use suffers from deficient temporal correspondence.Its image capabilities motivate adapting SAM for interactive video tracking.
- Motivation: Complex video scenarios such as scale variation, deformation, motion blur, camera motion, and similar objects motivate interaction-based tracking and segmentation.The paper frames interactive perception as a response to these challenges.
- Proposal: TAM combines SAM and XMem in an interactive process that initializes targets with clicks and produces tracking and segmentation in one-pass inference.XMem supplies temporal video predictions while SAM supports interactive target definition.
- Contributions: The paper promotes SAM to video by integrating it into temporal correspondence construction rather than applying it independently to each frame.This is presented as a core contribution of the approach.
- Contributions: TAM uses very small amounts of human participation to support efficient annotation and interactive tracking and segmentation.The contribution is framed as addressing difficult video object perception with limited user effort.
2 Track Anything Task
The Track Anything task defines flexible object tracking in arbitrary videos through user-directed object selection. Its scope includes varied video lengths, object changes, and multiple tracking and video-segmentation settings.
- Task Definition: Track Anything enables users to flexibly select, add, or remove target objects in arbitrary videos.The task is designed around user interests rather than fixed targets.
- Task Definition: The task allows arbitrary video lengths and types instead of restricting evaluation to trimmed or natural videos.This broadens the task setting beyond conventional video assumptions.
- Task Scope: The framework covers single- and multiple-object tracking, short- and long-term tracking, and several VOS variants.Listed applications include unsupervised, semi-supervised, referring, interactive, and long-term VOS.
3 Methodology
The methodology combines SAM’s promptable image segmentation with XMem’s temporal video propagation, adding refinement and human correction to address degradation and difficult cases. The supplied passages describe the pipeline and its reported failure setting.
- Preliminaries: SAM offers promptable image segmentation, while XMem tracks an initialized object and generates masks across subsequent video frames.SAM supports weak prompts; XMem uses the initial mask for semi-supervised VOS.
- Preliminaries: Existing interactive VOS methods commonly require multiple refinement rounds, motivating a more efficient one-pass interactive process.The paper contrasts its design with iterative interaction.
- Pipeline: TAM organizes tracking into four steps: SAM initialization, XMem tracking, SAM refinement, and human correction.The pipeline is presented as a single interactive Track-Anything process.
- Tracking: XMem performs semi-supervised VOS after initialization and can trigger refinement when mask quality is unsatisfactory.The method saves XMem predictions and intermediate probes and affinities for the refinement stage.
- Refinement: SAM refines XMem masks by converting probes and affinities into point prompts and using the predicted mask as an additional mask input.This stage addresses increasingly coarse masks during VOS inference.
- Correction: Human correction is added for extremely challenging scenarios, especially long videos where accurate object distinction remains difficult.The paper describes small human efforts as producing a qualitative performance improvement.
4 Experiments
TAM achieves strong one-pass video tracking and segmentation after click initialization, including difficult scene changes and motion. Its failures remain concentrated in long-term memory and complex object structures.
- Quantitative Results: TAM obtains J&F scores of 88.4 on DAVIS-2016-val and 73.1 on DAVIS-2017-test-dev with click initialization and one-pass evaluation.The evaluation uses the validation set of DAVIS-2016 and the test-development set of DAVIS-2017.
- Qualitative Results: Qualitative results show TAM handling multi-object separation, target deformation, scale change, and camera motion with click initialization and one-round inference.
- Failed Cases: Failed cases commonly arise when short-video VOS models lose long-term memory, causing mask shrinkage or insufficient refinement in long videos.SAM refinement and additional human interaction may help, but excessive interaction reduces efficiency.
- Failed Cases: Complex object structures, such as bicycle wheels with many cavities, make fine-grained initialization from clicks difficult.
5 Applications
TAM supports interactive video annotation, long-term tracking, video editing, and visualized development across multiple video tasks. Demonstrations include precise multi-object tracking through shot changes and subsequent video inpainting.
- Applications: TAM’s interactive design supports diverse downstream tasks through flexible tracking and segmentation in videos.
- Efficient video annotation: TAM enables efficient video annotation by segmenting user-selected regions with click-based interaction.It supports annotation for video object tracking and video object segmentation.
- Long-term object tracking: TAM handles target disappearance and reappearance across shot changes in long videos, extending long-term tracking toward real-world applications.
- User-friendly video editing: TAM provides object masks that support removing or altering existing objects in videos, evaluated with E2FGVI for video inpainting.
- Visualized development toolkit for video tasks: The toolkit provides visualized interfaces for VOS, VOT, video inpainting, and other tasks, enabling instantaneous result visualization on real-world videos.Corresponding demos are available in Hugging Face.
- Applications demonstration: A Captain America: Civil War demonstration shows precise multiple-object tracking through many shot changes and usefulness for video inpainting.Representative raw frames, object masks, and inpainted results are shown.