PaperlayerView exact revision on arXiv

Source-linked AI summary

SAMURAI: Adapting Segment Anything Model for Zero-Shot Visual Tracking with Motion-Aware Memory

Cheng-Yen Yang, Hsiang-Wei Huang, Wenhao Chai, Zhongyu Jiang, Jenq-Neng Hwang

arXiv:2411.11922v2cs.CV

TL;DR

SAM 2’s tracking is challenged by motion, crowded scenes, and memories that can retain irrelevant features during occlusion. SAMURAI adds motion-based mask selection and motion-aware memory selection, and the authors report consistent improvements across VOT benchmarks without retraining or fine-tuning.

  • Problem

    SAM 2 can neglect motion cues in crowded or fast-moving scenes, while indiscriminate recent-frame memory selection can introduce irrelevant features during occlusion.

    Source: §1 · p. 1

  • Method

    SAMURAI adapts SAM 2 with motion modeling for mask selection and hybrid motion, object, and mask-affinity scoring to select memories.

    Source: §1 · p. 1

  • Results

    SAMURAI shows consistent improvement across multiple VOT benchmarks on all metrics, and on GOT-10k SAMURAI-B improves AO by 2.1% and OP0.5 by 2.9% over SAM2.1-B.

    Source: §5.2 · p. 6, §6 · p. 8

  • Takeaways & Limitations

    The authors report robust, real-time online tracking across multiple VOT benchmarks without retraining or fine-tuning.

    Source: §6 · p. 8

Abstract

from arXiv · show

The Segment Anything Model 2 (SAM 2) has demonstrated strong performance in object segmentation tasks but faces challenges in visual object tracking, particularly when managing crowded scenes with fast-moving or self-occluding objects. Furthermore, the fixed-window memory approach in the original model does not consider the quality of memories selected to condition the image features for the next frame, leading to error propagation in videos. This paper introduces SAMURAI, an enhanced adaptation of SAM 2 specifically designed for visual object tracking. By incorporating temporal motion cues with the proposed motion-aware memory selection mechanism, SAMURAI effectively predicts object motion and refines mask selection, achieving robust, accurate tracking without the need for retraining or fine-tuning. SAMURAI operates in real-time and demonstrates strong zero-shot performance across diverse benchmark datasets, showcasing its ability to generalize without fine-tuning. In evaluations, SAMURAI achieves significant improvements in success rate and precision over existing trackers, with a 7.1% AUC gain on LaSOT$_{\text{ext}}$ and a 3.5% AO gain on GOT-10k. Moreover, it achieves competitive results compared to fully supervised methods on LaSOT, underscoring its robustness in complex tracking scenarios and its potential for real-world applications in dynamic environments.

1. Introduction

SAM 2’s video-segmentation capability does not prevent tracking errors when motion, crowded scenes, and occlusions challenge mask selection and memory use. SAMURAI adapts it with motion modeling and motion-aware memory selection for zero-shot visual tracking.

Tracking challenges

  • SAM 2 can neglect motion cues and prioritize appearance similarity, causing tracking errors with rapid movement, complex interactions, or similar objects in crowded scenes.The introduction also identifies confusion in crowded scenes and ineffective memory use during occlusions as common failure patterns.Source: §1 · p. 1, Figure 1 · p. 2
  • Indiscriminately storing recent frames can introduce irrelevant features during occlusions and compromise tracking performance.The paper frames improved memory management as necessary for robust video object tracking.Source: §1 · p. 1

Proposed approach

  • SAMURAI combines motion modeling to refine mask selection with hybrid scoring of mask affinity, object, and motion to retain relevant memories.The paper reports zero-shot state-of-the-art performance across LaSOT, LaSOT_ext, GOT-10k, and other VOT benchmarks without additional training or fine-tuning.Source: §1 · pp. 1–2

Contributions

  • SAMURAI is reported to improve tracking accuracy for fast-moving and occluded objects and to generalize across diverse benchmarks without retraining or fine-tuning.The paper describes its performance as real-time and state of the art on LaSOT, LaSOT_ext, GOT-10k, and other VOT benchmarks.Source: §1 · pp. 1–2

2. Related Works

The related work reviews tracking methods that use similarity, memory, and attention, SAM’s development into video segmentation, and heuristic or learnable motion models. It positions SAMURAI as learning-based motion modeling with an enhanced heuristic scheme.

2.1. Visual Object Tracking (VOT)

  • Siamese- and transformer-based trackers learn embedding similarity, while memory-bank and attention approaches map current frames to historical information.The cited discussion notes that single-forward-pass trackers can drift toward distractors.Source: §2.1 · p. 2

2.2. Segment Anything Model (SAM)

  • SAM provides prompt-based segmentation, and SAM 2 extends it to video with memory mechanisms for objects across frames.Related work also describes SAM applications in video understanding, editing, and medical imaging, and notes SAM2Long’s tree-based memory approach for long videos.Source: §2.2 · p. 2

2.3. Motion Modeling

  • Motion models include heuristic Kalman filters and learnable trajectory-based approaches; SAMURAI is characterized as learning-based motion modeling with an enhanced heuristic scheme.The cited examples include Tracktor, MotionTrack, and MambaTrack; the discussion notes that Kalman filters can fail under intense or abrupt motion and involve multiple hyperparameters.Source: §2.3 · pp. 2–3

3. Revisiting Segment Anything Model 2

SAM 2 processes video with image and mask components plus memory attention and encoding. It selects among predicted masks using affinity, while its fixed-window memory can retain low-confidence outputs that propagate errors in long tracking sequences.

SAM 2 components

  • SAM 2 comprises an image encoder, mask decoder with prompt encoder, memory attention layer, and memory encoder.The paper introduces these components as the baseline context for SAMURAI.Source: §3 · p. 3

Prompt Encoder

  • For tracking, SAM 2 uses the initial target bounding box as point prompts and then uses the previous frame’s predicted mask as the prompt.The prompt encoder supports sparse prompts such as points and boxes, and dense prompts such as masks.Source: §3 · p. 3

Mask Decoder

  • The mask decoder produces candidate masks with affinity scores and a frame-level object score; the original implementation selects the mask with the highest affinity.The mask affinity score represents overall mask confidence, while the object score indicates whether a mask should exist.Source: §3 · p. 3

Mask selection

  • Because affinity alone is not robust in crowded scenes with self-occluding similar objects, SAMURAI adds a motion score to help select the prediction.The passage identifies motion modeling as an additional aid to mask selection.Source: §3 · p. 3

Memory Encoder and Memory Bank

  • SAM 2 appends each frame’s memory embedding to a FIFO bank of recent memories, but fixed-window storage can retain incorrect or low-confidence objects and propagate errors in long sequences.SAMURAI’s motion-aware selection replaces the original bank composition so better memory features condition image features.Source: §3 · p. 3

4. Method

SAMURAI addresses SAM 2’s tracking errors and long-sequence error propagation with Kalman-filter motion modeling and motion-aware memory selection. The additions integrate into SAM 2 without retraining or extra computational overhead.

4. Method

  • SAMURAI combines Kalman-filter motion modeling with hybrid memory selection to address SAM 2’s incorrect or low-confidence masks and resulting error propagation.The method is designed for complex video scenarios and requires neither fine-tuning nor additional training.Source: §4 · pp. 3–4
  • The modules refine mask selection without retraining or additional computational overhead, and are described as model-agnostic beyond SAM 2.The paper characterizes the resulting tracker as a real-time online VOT solution.Source: §4 · p. 4, §4.2 · p. 5

4.1. Motion Modeling

  • KF models box center, size, and velocities, then combines predicted-box IoU with affinity to select a mask; motion is used only after successful updates within τkf frames.Each candidate mask’s bounding box is derived from its non-zero pixels.Source: §4.1 · p. 4

4.2. Motion-Aware Memory Selection

  • Instead of relying on recent frames that may fail under occlusion or deformation, SAMURAI filters past frames by mask-affinity, object-occurrence, and motion thresholds, selecting seven memories for decoding.The selected bank is passed through memory attention and then to the mask decoder.Source: §4.2 · pp. 4–5

5. Experiments

SAMURAI is evaluated on diverse tracking benchmarks, where it improves on existing methods and remains competitive with supervised trackers. Ablations, attribute-wise analyses, runtime measurements, and qualitative comparisons examine its performance and behavior.

5.1. Benchmarks

  • LaSOT has 1,400 videos across 70 categories, while LaSOText adds 150 sequences focused on occlusions and small-object variation.The standard LaSOText protocol evaluates models trained on LaSOT directly on the extension.Source: §5.1 · p. 5

5.2. Quantitative Results

  • SAMURAI improves all three metrics over zero-shot and supervised methods on LaSOT/LaSOText, surpasses state of the art on LaSOText and GOT-10k, and matches or exceeds supervised AUC on three other benchmarks.The three additional benchmarks are TrackingNet, NFS, and OTB100.Source: §5.2 · p. 6, Figure 3 · p. 6, Table 2 · p. 6

5.3. Ablation Studies

  • Both proposed modules improve SAM 2, and combining them yields the best LaSOT AUC (74.23%) and Pnorm (82.60%); αmotion = 0.2 gives the best scores on both metrics.The motion-weight experiments examine the trade-off between motion score and mask affinity.Source: Table 3 · p. 6, Table 4 · p. 6, §5.3 · p. 7
  • SAMURAI improves on the SAM 2 baseline across all three metrics and nearly all attributes; on LaSOText, CM and FM gains are 16.5% and 9.9%, respectively.The reported attribute exception is illumination variation on LaSOText; occlusion-related attributes improve across model variants and datasets.Source: §5.3 · p. 7, Table 5 · p. 7, Table 6 · p. 7
  • Motion modeling and enhanced memory selection introduce minimal computational overhead, with runtime on one NVIDIA RTX 4090 remaining consistent with the baseline.Source: §5.3 · pp. 7–8

5.4. Qualitative Results

  • In qualitative comparisons, SAMURAI tracks more reliably amid similar-looking objects and short-term occlusions, without retraining or fine-tuning.The visualized baseline masks illustrate why a systematic way to decide which predictions to trust as memory is valuable.Source: §5.4 · p. 8, Figure 4 · p. 8

6. Conclusion

SAMURAI adds motion-based scoring for mask prediction and memory selection, improving tracking across SAM variants and VOT benchmarks without retraining or fine-tuning.

6. Conclusion

  • SAMURAI introduces motion-based scoring for mask prediction and memory selection to address self-occlusion and abrupt motion in crowded scenes.The framework supports real-time online inference.Source: §6 · p. 8
  • The proposed modules consistently improve all SAM model variations across multiple VOT benchmarks and all metrics.Source: §6 · p. 8
  • SAMURAI requires no retraining or fine-tuning while delivering robust performance and real-time online inference across multiple VOT benchmarks.Source: §6 · p. 8

Related papers

Loading 2411.11922v2…