PaperlayerView exact revision on arXiv

Source-linked AI summary

SAM2MOT: A Novel Paradigm of Multi-Object Tracking by Segmentation

Junjie Jiang, Zelin Wang, Manqi Zhao, Yin Li, DongSheng Jiang

arXiv:2504.04519v5cs.CV

TL;DR

MOT systems often depend on fragile detection–association pipelines or scarce labeled tracking data. SAM2MOT instead makes segmentation central, combining pretrained detection, segmentation, and tracking logic for zero-shot MOT, with state-of-the-art results across three benchmarks and a 2.1 HOTA, 4.5 IDF1 DanceTrack advantage.

  • Problem

    Detection–association MOT is vulnerable to ambiguous detections and occlusions, while alternative end-to-end methods depend on scarce, costly identity annotations.

    Source: Introduction · p. 1

  • Method

    SAM2MOT uses segmentation as the core tracking mechanism and integrates pretrained detector, segmentor, and tracking logic into a zero-shot system without scene-specific fine-tuning.

    Source: Introduction · p. 2

  • Results

    SAM2MOT achieves state-of-the-art results on DanceTrack, UAVDT, and BDD100K, reaching 75.8 HOTA and 83.9 IDF1 on DanceTrack.

    Source: Introduction · p. 2

  • Takeaways & Limitations

    The paper presents zero-shot segmentation-driven tracking as a path toward more generalizable MOT systems and lower-cost tracking-data construction.

    Source: Conclusion · p. 7

    SAM2MOT lacks holistic object modeling, so when only part of an object is visible its box may cover only the visible region.

    Source: Holistic Object Modeling · p. 7

Abstract

from arXiv · show

Inspired by Segment Anything 2, which generalizes segmentation from images to videos, we propose SAM2MOT--a novel segmentation-driven paradigm for multi-object tracking that breaks away from the conventional detection-association framework. In contrast to previous approaches that treat segmentation as auxiliary information, SAM2MOT places it at the heart of the tracking process, systematically tackling challenges like false positives and occlusions. Its effectiveness has been thoroughly validated on major MOT benchmarks. Furthermore, SAM2MOT integrates pre-trained detector, pre-trained segmentor with tracking logic into a zero-shot MOT system that requires no fine-tuning. This significantly reduces dependence on labeled data and paves the way for transitioning MOT research from task-specific solutions to general-purpose systems. Experiments on DanceTrack, UAVDT, and BDD100K show state-of-the-art results. Notably, SAM2MOT outperforms existing methods on DanceTrack by +2.1 HOTA and +4.5 IDF1, highlighting its effectiveness in MOT. Code is available at https://github.com/TripleJoy/SAM2MOT.

Introduction

SAM2MOT challenges detection–association MOT by making segmentation central to tracking and integrating pretrained components into a zero-shot system. Across DanceTrack, UAVDT, and BDD100K, it reports state-of-the-art results without fine-tuning.

Motivation

  • Conventional detection–association tracking is vulnerable to unstable detections in crowded or occluded scenes and depends heavily on detector quality.These weaknesses can make association error-prone or cause tracking to collapse when detections are ambiguous.Source: Introduction · p. 1
  • End-to-end alternatives address detection–association limitations but depend on scarce, costly, high-quality identity annotations, which can impair associations and generalization under limited supervision.The paper frames this data dependence as a motivation for rethinking MOT architecture.Source: Introduction · p. 1
  • The paper argues that segmentation’s pixel-level precision and spatial awareness have been underused because prior methods typically treat it as auxiliary information.Inspired by SAM2, it explores segmentation as a core tracking mechanism.Source: Introduction · pp. 1–2

Contribution

  • The method introduces a segmentation-driven MOT paradigm that places segmentation at the center of tracking rather than treating it as auxiliary information.It is presented as a first segmentation-centric framework that systematically addresses false positives and occlusions.Source: Introduction · pp. 1–2
  • SAM2MOT integrates a pretrained detector, pretrained segmentor, and tracking logic into a zero-shot system requiring no scene-specific fine-tuning.The framework is designed to reduce dependence on labeled tracking data.Source: Introduction · p. 2

Broader impact

  • The paper identifies SAM2MOT as the first systematic study of zero-shot tracking in MOT and connects its zero-shot capability to generating tracking pre-annotations.The authors state that this can reduce manual labeling costs and accelerate dataset creation.Source: Introduction · p. 2

Results

  • SAM2MOT achieves state-of-the-art results on DanceTrack, UAVDT, and BDD100K, including 75.8 HOTA and 83.9 IDF1 on DanceTrack.On DanceTrack, it surpasses the second-best method by 2.1 HOTA and 4.5 IDF1.Source: Introduction · p. 2

Related Works

Prior MOT work is organized around detection–association, query-based tracking, and segmentation-assisted systems, each retaining important dependencies or limitations. SAM2 provides the video-segmentation foundation that motivates a more central segmentation-based tracking design.

Conventional MOT Paradigms

  • Tracking-by-Detection methods improve association through motion, appearance, low-confidence detections, occlusion handling, and camera-motion compensation, but remain tightly coupled to detector quality.This coupling demands scene-specific tuning and hinders generalization.Source: Conventional MOT Paradigms · p. 2
  • Tracking-by-Query methods use parallel or sequential query updates, with memory and long-term feature mechanisms for temporal modeling, but rely on costly and scarce labeled tracking data.The passage distinguishes parallel methods such as VisTR and SeqFormer from sequential methods such as TransTrack, TrackFormer, and MOTR.Source: Conventional MOT Paradigms · p. 2

Segmentation in MOT

  • Earlier segmentation-based MOT methods often use masks as auxiliary association information or assemble separate components without deeply modeling segmentation as the core tracking mechanism.Examples include MASA, TrackAnything, and STAnything.Source: Segmentation in MOT · p. 2
  • The paper argues that prior segmentation systems often decompose MOT into independent single-object instances and fail to systematically address occlusion and false positives.This limitation makes direct comparison with conventional MOT paradigms difficult.Source: Segmentation in MOT · p. 2

SAM2 in Tracking

  • SAM2 extends segmentation from images to videos through a memory bank that stores appearance references and performs spatiotemporal matching for mask prediction.Its inference references the key frame and previous six frames, then updates the memory with new segmentations.Source: SAM2 in Tracking · p. 3
  • SAM2Long and SAMURAI address SAM2’s simplistic memory-frame selection using decision-tree memory and confidence-based memory pruning, respectively.These methods advance single-object tracking performance on relevant benchmarks.Source: SAM2 in Tracking · p. 3

Method

SAM2MOT builds a zero-shot MOT framework around segmentation, combining a pretrained detector, SAM2, and tracking logic to address occlusion, false positives, and prompt degradation. Its Cross-object Interaction and Trajectory Manager modules regulate memory, object initialization, reconstruction, and removal.

Overview

  • SAM2MOT integrates a pretrained detector, SAM2, and tracking logic for zero-shot multi-object tracking without scene-specific fine-tuning.The detector dynamically generates prompts, while SAM2 produces per-object masks rather than directly inheriting detector boxes.Source: Overview · p. 3

Cross-object Interaction

  • Cross-object Interaction detects occlusion-induced identity confusion through mask overlap, confidence comparison, and confidence variance over recent frames.When overlap exceeds 0.8, the module analyzes logits and their variance over N=10 frames to identify the corrupted track.Source: Cross-object Interaction · p. 4, Figure 3 · p. 4
  • The module prevents error propagation by excluding confused objects’ current-frame memories from memory-bank updates while retaining a low-confidence filtering mechanism.This combination preserves reliable historical features and supports re-identification after temporary occlusion failures.Source: Cross-object Interaction · p. 4, Figure 3 · p. 4

Trajectory Manager System

  • New-object initialization uses confidence filtering, Hungarian matching, and non-tracked-mask overlap to reject false positives and initialize sufficiently distinct candidates.Candidates are initialized when their overlap with unoccupied pixels exceeds r=0.7 after detector confidence filtering.Source: Trajectory Manager System · p. 4
  • SAM2MOT categorizes tracks as reliable, pending, suspicious, or lost using three logit thresholds, with pending tracks triggering appearance reconstruction and suspicious tracks left for future work.The reliable, pending, suspicious, and lost states correspond respectively to clear visibility, mild degradation, severe occlusion, and unavailable appearance.Source: Trajectory Manager System · p. 5
  • The Trajectory Manager System filters new prompts, reconstructs degraded keyframe information, and removes objects that remain lost beyond a frame-based tolerance.Quality reconstruction uses reliable detector boxes to update keyframe information only for pending objects whose tracking boxes match high-confidence detections.Source: Trajectory Manager System · pp. 4–5

Experiments

SAM2MOT is evaluated across diverse MOT benchmarks and consistently outperforms established detection–association baselines, with especially strong gains in identity association.

Benchmarks

  • The evaluation spans DanceTrack, UAVDT-MOT, and BDD100K-MOT, covering occlusion, varied motion, inter-object similarity, and indoor, aerial, and driving settings.HOTA, MOTA, and IDF1 are used generally; DanceTrack additionally reports DetA and AssA.Source: Benchmarks · p. 5

State-of-the-Art Comparison

  • HOTA +2.1, IDF1 +4.5, and AssA +6.3 over previous best methods on DanceTrack demonstrate SAM2MOT’s strongest reported association gains.With Co-DINO-L, it reaches HOTA = 75.5, MOTA = 89.2, and IDF1 = 83.4; Grounding-DINO-L yields 75.8, 88.5, and 83.9.Source: State-of-the-Art Comparison · p. 5
  • SAM2MOT achieves the best UAVDT-MOT results under IoU = 0.5, with MOTA = 55.6 and IDF1 = 74.4, rising to 66.1 and 79.4 at IoU = 0.4.The relaxed evaluation is presented as better reflecting performance given annotation noise.Source: State-of-the-Art Comparison · p. 5
  • On BDD100K-MOT, SAM2MOT obtains MOTA = 57.5 and IDF1 = 70.8 with eight classes, improving to 63.0 and 73.7 under three classes.The paper attributes lower eight-class performance to visually similar categories confusing the unfine-tuned detector.Source: State-of-the-Art Comparison · pp. 5–6

Ablation Study and Analysis

  • Using identical detector and unified settings, SAM2MOT consistently outperforms ByteTrack and OC-SORT across all benchmarks, especially on identity association.Its internally generated bounding boxes can recover true positives when the detector fails, reducing reliance on precise detections.Source: Ablation Study and Analysis · p. 6
  • Cross-object Interaction contributes most strongly to association accuracy, while Object Addition, Cross-object Interaction, and Quality Reconstruction form the evaluated core components.The component ablation is conducted on the DanceTrack test set.Source: Ablation Study and Analysis · p. 6

Limitation.

The paper identifies inference speed, incomplete object extent modeling, and insufficient long-term memory as principal limitations of SAM2MOT.

Inference Efficiency.

  • Inference speed is the architecture’s primary limitation, motivating acceleration through lighter detectors, faster segmentation, and parallel SAM2 inference.Despite limited efficiency, the paper reports strong zero-shot generalization and reduced manual labeling effort for dataset construction.Source: Inference Efficiency · pp. 6–7

Holistic Object Modeling.

  • When only part of an object is visible, SAM2MOT tracks it stably but predicts a box covering only the visible region rather than the full object extent.The paper contrasts this with detection–association methods, whose detectors infer full bounding boxes.Source: Holistic Object Modeling · p. 7

Long-term Memory.

  • Extended occlusions can cause tracking loss because SAM2 primarily encodes adjacent-frame motion and lacks stable long-term dependency modeling.The paper suggests adaptive memory-frame selection using appearance cues for implicit re-identification.Source: Long-term Memory · p. 7

Conclusion

SAM2MOT introduces a segmentation-driven MOT paradigm and a unified zero-shot framework, with experiments validating strong identity association across mainstream benchmarks.

  • SAM2MOT proposes segmentation-driven tracking and a unified framework with strong zero-shot generalization for data-scarce conditions.The paper reports superior identity association capability across multiple mainstream benchmarks.Source: Conclusion · p. 7
Loading 2504.04519v5…