PaperlayerView exact revision on arXiv

Source-linked AI summary

Segment Anything with Motion, Geometry, and Semantic Adaptation for Complex Nonlinear Visual Object Tracking

Deyi Zhu, Yuji Wang, Yong Liu, Yansong Tang, Bingyao Yu, Jiwen Lu, Jie Zhou

arXiv:2605.22538v1cs.CV

TL;DR

Complex nonlinear motion makes visual object tracking difficult, while SAM 2 lacks explicit tracking-oriented modeling of motion and cross-frame consistency. SAMOSA adapts SAM 2 with motion, geometry, and semantic cues, achieving state-of-the-art performance and strong generalization across general-purpose and anti-UAV benchmarks.

  • Problem

    Visual object tracking needs models that capture nonlinear target dynamics because constant-velocity assumptions struggle with frequent speed and direction changes.

  • Method

    SAMOSA adapts SAM 2 with a higher-order Markov motion predictor, error detection and recovery, and target-aware memory selection guided by tracking cues.

  • Results

    SAMOSA achieves state-of-the-art performance and strong generalization across general-purpose VOT and challenging anti-UAV benchmarks with controllable latency overhead.

  • Takeaways & Limitations

    As a lightweight, pluggable adapter, SAMOSA can be integrated into future generations of Segment Anything models.

Abstract

from arXiv · show

Traditional visual object tracking (VOT) methods typically rely on task-specific supervised training, limiting their generalization to unseen objects and challenging scenarios with distractors, occlusion, and nonlinear motion. Recent vision foundation models, exemplified by SAM 2, learn strong video understanding priors from large-scale pretraining and offer a promising foundation for building more robust and generalizable trackers. However, directly applying SAM 2 to VOT remains suboptimal, as it does not explicitly model target motion dynamics or enforce geometric and semantic consistency across frames, both of which are essential for reliable tracking. To address this issue, we propose SAMOSA, a new tracking framework that adapts SAM 2 to complex VOT scenarios by explicitly leveraging motion, geometry, and semantic cues. Specifically, we introduce a lightweight nonlinear motion predictor to model target dynamics and guide mask selection as well as memory filtering. We further exploit semantic cues to detect target shifts and recover from tracking failures, while geometric cues are incorporated as structural constraints to improve tracking stability. In this way, SAMOSA bridges the gap between the implicit video understanding prior of SAM 2 and explicit tracking-oriented modeling. Extensive experiments show that SAMOSA consistently outperforms state-of-the-art SAM 2--based approaches on general benchmarks, demonstrates stronger generalization than supervised VOT methods, and achieves substantial gains on anti-UAV datasets, which typify complex nonlinear motion scenarios. Our code is available at https://github.com/DurYi/SAMOSA.

I. INTRODUCTION

Existing SAM 2-based trackers are robust in general scenarios but struggle with complex nonlinear target motion because they lack explicit motion modeling and efficient geometric and semantic consistency. SAMOSA addresses this gap with motion prediction, cue-guided mask and memory management, and failure recovery, achieving stronger generalization and substantial gains on nonlinear-motion benchmarks.

  • Motivation: Existing SAM 2-based VOT methods perform strongly in general scenarios but struggle with complex motion because they lack explicit nonlinear dynamics and efficient geometric and semantic consistency.Nonlinear motion includes velocity variations, acceleration, direction changes, camera movements, shape variations, or temporary target disappearance.
  • Motivation: SAMOSA models complex tracking through complementary motion, geometry, and semantic cues that support prediction, distractor discrimination, and target-shift prevention.Motion describes temporal target evolution; geometry captures shape, area, and boundary structure; semantics help identify and prevent target-shift errors.
  • Method: SAMOSA introduces a higher-order Markov Motion Predictor and an Error Detection–Recovery Module to model nonlinear motion, guide mask selection, detect failures, and mitigate error propagation.The Motion Predictor is the only trainable component and uses annotated bounding-box trajectories without video frames.
  • Method: Its Target-Aware Memory Bank adaptively selects representative and reliable memory frames using confidence, occlusion, mask quality, target visibility, and motion information.The memory bank prioritizes reliable memories to support tracking stability and temporal association.
  • Results: SAMOSA achieves state-of-the-art performance across general VOT and challenging Anti-UAV benchmarks, with substantial improvements in nonlinear-motion scenarios.Evaluation covers LaSOText, OTB, TrackingNet, and Anti-UAV series benchmarks.

II. RELATED WORK … III. PRELIMINARY

The related work reviews conventional VOT and VOS-based tracking, emphasizing their limitations under occlusion, nonlinear motion, and domain shifts. The preliminary section introduces SAM 2’s memory-based architecture and explains why explicit motion, geometry, and semantic cues are needed for complex tracking.

  • A. Conventional Visual Object Tracking: Early VOT methods used correlation filters, followed by Siamese similarity learning, online discriminative learning, and transformer-based end-to-end architectures.Examples include SiamFC, SiamRPN++, DiMP, TransT, and OSTrack.
  • A. Conventional Visual Object Tracking: Existing trackers struggle with long-term occlusion, rapid appearance variation, complex nonlinear motion, and generalization to unseen targets and environments.Task-specific supervised training is identified as a key limitation for cross-domain generalization.
  • B. Video Object Segmentation for Visual Object Tracking: VOS-based tracking uses segmentation masks to follow non-rigid or irregular objects more robustly than bounding boxes under complex contours and structural variation.SAM 2 provides zero-shot segmentation and tracking capabilities but remains challenged by occlusions, distractors, and multiple similar objects.
  • B. Video Object Segmentation for Visual Object Tracking: Existing methods still struggle in nonlinear scenes, while adapting SAM 2 to such dynamics without substantial computational cost remains unresolved.SAMOSA is introduced as a lightweight enhancement of SAM 2 for complex nonlinear visual object tracking.
  • III. PRELIMINARY: SAM 2 extracts frame features with the pre-trained Hiera vision encoder and refines them through memory-attention over historical representations.The memory bank stores historical representations for subsequent tracking decisions.
  • III. PRELIMINARY: SAM 2 predicts mask affinity and quality with SIoU and target visibility with Sobj, then inserts decoded masks into a FIFO memory bank preserving spatial and semantic information.The two scores are produced by bidirectional-transformer and MLP components.
  • III. PRELIMINARY: Directly applying SAM 2 to complex nonlinear tracking is challenging because it lacks explicit historical-trajectory modeling and selects masks solely by SIoU.Robust localization requires integrated motion, geometry, and semantic cues for temporal consistency.

IV. METHOD · A. Motion Predictor (MP)

SAMOSA adapts SAM 2 with a nonlinear Motion Predictor that combines finite-history dynamics with geometric and motion cues for mask selection. The predictor is independently trained from annotated trajectories and plugged into SAM 2 inference, while the broader pipeline also includes recovery and memory modules.

  • IV. METHOD: SAMOSA integrates the Motion Predictor, Error Detection-Recovery Module, and Target-Aware Memory Bank into SAM 2 to handle nonlinear target dynamics.The Motion Predictor dominates stable conditions, whereas the recovery module overrides it under uncertainty.
  • A. Motion Predictor (MP): The conventional first-order predictor assumes constant velocity and straight-line motion, limiting its ability to capture varying speed and direction in VOT.Its state contains bounding-box position, size, and first-order derivatives.
  • A. Motion Predictor (MP): MP models nonlinear dynamics with a k-th order Markov framework conditioned on a sliding window of the past k states.This finite-history design balances modeling capacity and computational efficiency without requiring the entire sequence.
  • A. Motion Predictor (MP): MP is trained independently on annotated bounding-box trajectories using mean squared error and complete IoU losses.CIoU promotes overlap, center alignment, and aspect-ratio consistency; after training, MP operates as a plug-and-play inference module.
  • A. Motion Predictor (MP): During inference, a FIFO bank stores the most recent k outputs, allowing MP to predict the current bounding box from historical boxes and guide mask selection.The predicted box is compared against candidate boxes generated by the mask decoder.
  • A. Motion Predictor (MP): For each mask, geometric consistency measures aspect-ratio and area similarity, while the motion score uses IoU with the motion-predicted box to measure spatial alignment.The method evaluates N = 3 candidate boxes and combines these cues with SAM 2’s mask IoU score.
  • A. Motion Predictor (MP): Unlike SAM 2’s IoU-only selection, SAMOSA chooses the mask using a weighted combination of mask IoU, geometric score, and motion score.This encourages masks that both match the target and conform to physical motion patterns, improving tracking robustness.

B. Error Detection-Recovery Module (EDRM)

EDRM detects and recovers from tracking failures by comparing current geometric and semantic cues against a temporally aggregated Target Prototype. When errors are detected, it freezes the prototype and searches candidate masks for a high-confidence correction before resuming updates.

  • Error Detection-Recovery Module (EDRM): EDRM detects tracking failures caused by camera shake, occlusion, or distractors and mitigates error accumulation.It relies on the assumption that the target’s visual state remains relatively stable over short temporal intervals.
  • Target Prototype (TP): The Target Prototype combines geometry and semantics aggregated from the most recent T frames.Geometry comes from recent bounding boxes, while semantics come from mask-gated SAM 2 Hiera image embeddings.
  • Target Prototype (TP): The prototype updates throughout tracking but is temporarily frozen when EDRM detects a potential error, preventing contamination by erroneous outputs.This frozen state is maintained during recovery.
  • Error Detection and Recovery: EDRM compares current and prototype aspect ratio, area, and semantic similarity, entering recovery if any score falls below its predefined threshold.Similarity uses min(x, y)/max(x, y), while semantic similarity uses cosine similarity.
  • Error Detection and Recovery: During recovery, EDRM evaluates all N = 3 candidate masks and replaces MP’s choice when one exceeds every predefined recovery threshold.It then resumes Target Prototype updates and returns to error-detection mode.

C. Target-Aware Memory Bank (TAMB)

TAMB is a target-aware memory bank that selects representative memory frames using motion cues, mask quality, and target completeness. It combines threshold filtering with weighted top-k selection to retain reliable, temporally informative memories for tracking.

  • C. Target-Aware Memory Bank (TAMB): TAMB selects representative target memory frames using motion cues, mask quality, and target completeness.Its design addresses error propagation from low-quality masks and supports motion modeling by prioritizing high-quality memory.
  • C. Target-Aware Memory Bank (TAMB): SAM 2’s IoU and object scores measure mask quality and the likelihood of clear, unoccluded target visibility.These scores identify memory frames with reliable segmentation and target appearance.
  • C. Target-Aware Memory Bank (TAMB): The motion score from MP identifies stable target motion and filters frames that violate regular motion patterns.This score complements segmentation and visibility cues during memory selection.
  • C. Target-Aware Memory Bank (TAMB): TAMB always retains the most recent frame, then gathers 30 threshold-qualified candidates and selects the top 5 by weighted score, with 6 unprompted memory slots.The prompted frame is also retained to provide the initial tracking condition.

V. EXPERIMENTS · A. Experimental Settings

SAMOSA is evaluated on three general-purpose VOT benchmarks and four anti-UAV benchmarks spanning diverse difficulty levels and scenarios. The experiments compare against SAM 2-based and supervised VOT methods, using a largely frozen architecture with a trajectory-trained motion predictor.

  • A. Experimental Settings: SAMOSA is evaluated on three general-purpose VOT benchmarks and four anti-UAV tracking benchmarks.The benchmarks cover typical VOT settings and challenging anti-UAV conditions.
  • A. Experimental Settings: LaSOText, OTB, and TrackingNet contain 150, 100, and 511 videos, respectively.Their average video lengths are 2,393, 598, and 441 frames, respectively.
  • A. Experimental Settings: Anti-UAV300, Anti-UAV410, Anti-UAV600, and DUT Anti-UAV provide paired RGB-TIR, TIR, validation-set TIR, and RGB videos, respectively.They contain 91, 120, 50, and 20 videos, with average lengths of 938, 1,081, 1,179, and 1,240 frames, respectively.
  • A. Experimental Settings: Together, the general-purpose and anti-UAV benchmarks assess trackers across different difficulty levels and diverse scenarios.The paper describes this combination as a comprehensive assessment of model robustness.
  • A. Experimental Settings: The experiments compare SAMOSA with SAM 2-based methods and representative supervised VOT methods.The cited baselines include methods from references, [35]–,,,,,,, –.
  • A. Experimental Settings: Only the MP is trained, while TAMB, EDRM, and all SAM 2.1 backbone modules remain frozen.This setup is used to ensure a fair comparison.
  • A. Experimental Settings: The MP uses a 4-layer LSTM to predict target states from the past k = 5 frames.It is trained solely on trajectory annotations from LaSOT before evaluation on all other benchmarks.
  • A. Experimental Settings: Training the MP only on LaSOT trajectory annotations tests its generalization when integrated into evaluations on other benchmarks.TAMB and EDRM require no training during this process.

B. Main Results · C. Ablation Study · 1) Module-wise Ablation Study:

SAMOSA consistently outperforms prior SAM 2-based methods across general-purpose and anti-UAV benchmarks with minimal latency overhead, while generalizing better than supervised VOT methods. Ablations show that combining MP, EDRM, and TAMB performs best, with TAMB strongest on general benchmarks and MP on complex anti-UAV scenarios.

  • B. Main Results: SAMOSA consistently outperforms other SAM 2-based methods across LaSOText, OTB, and TrackingNet, particularly in AUC, Succ, and Pnorm.The method incurs only a marginal latency overhead.
  • B. Main Results: SAM2Long achieves comparable results to SAMOSA on OTB and TrackingNet, but incurs a 32× increase in ∆Latency.This highlights the computational efficiency of SAMOSA’s proposed modules.
  • B. Main Results: Supervised trackers perform comparably to SAMOSA on OTB and TrackingNet but fall behind on the larger-scale, more challenging LaSOText benchmark.The passage attributes this gap to distribution shift between training and evaluation data.
  • B. Main Results: Overall, SAMOSA achieves better cross-dataset generalization than supervised VOT methods and consistently outperforms prior SAM 2-based methods on general-purpose and anti-UAV benchmarks.Its advantages are especially remarkable in complex nonlinear tracking scenarios.
  • B. Main Results: SAMOSA shows more pronounced advantages on anti-UAV benchmarks than on general-purpose VOT benchmarks.SAMITE has slightly higher precision on Anti-UAV410 but lower Acc and AUC, while SAM2.1++ exceeds SAMOSA in precision on Anti-UAV600 with a 19× increase in ∆Latency.
  • 1) Module-wise Ablation Study:: The combination of MP, EDRM, and TAMB yields the best overall performance in the module-wise ablation.The study evaluates the individual contributions of MP, EDRM, and TAMB.
  • 1) Module-wise Ablation Study:: TAMB contributes most on general-purpose VOT benchmarks, whereas MP plays a dominant role on complex anti-UAV benchmarks.TAMB supports long-term tracking, frequent occlusion, and distractor-heavy scenes; MP models complex nonlinear motion.
  • 1) Module-wise Ablation Study:: EDRM adds limited gain when MP and TAMB are active but becomes more pronounced when either module is absent, mitigating errors from suboptimal mask selection or memory management.This reveals EDRM’s value as a complementary error-mitigation module.

2) Component-wise Ablation Study: · 3) Sensitivity Analysis of Parameters: · D. Discussion

Component-wise ablations show that combining motion, geometry, and semantic cues yields the strongest tracking behavior, while sensitivity experiments indicate robust performance across parameter variations. The discussion section frames these experiments as further analysis of the method and its variations.

  • 2) Component-wise Ablation Study:: The component-wise study evaluates the proposed motion, geometry, and semantic cues through the EDRM, MP, and TAMB modules.These ablations respectively investigate the contributions of the three cue types across the proposed modules.
  • 2) Component-wise Ablation Study:: Combining SIoU with motion and geometry cues achieves the best MP ablation performance, highlighting the value of comprehensive mask-selection cues.Either motion or geometry independently improves Acc and AUC over SIoU alone.
  • 2) Component-wise Ablation Study:: EDRM cue importance varies by task: semantics matter more on LaSOText, whereas geometry nearly dominates on Anti-UAV300.LaSOText targets are semantically rich and deformable, while Anti-UAV300 drone targets are small, semantically sparse, and geometrically stable.
  • 2) Component-wise Ablation Study:: All TAMB components contribute to memory-frame filtering and evaluation, with their combination producing the best results.The memory mechanism improves performance even using only SAM 2’s SIoU and Sobj, and motion cues further help complex scenarios.
  • 3) Sensitivity Analysis of Parameters:: Sensitivity analysis on Anti-UAV300 RGB finds that the method remains stable across different parameter configurations.The results indicate general robustness to parameter variations.
  • 3) Sensitivity Analysis of Parameters:: Moderate changes to MP weights α, β, and γ cause only marginal performance variation, indicating that precise tuning is not essential.The motion predictor’s prior generalizes across settings without significant performance degradation.
  • D. Discussion: The discussion presents additional experiments and further analysis of the method and its variations.This section contextualizes the component ablations and parameter sensitivity results as analyses of the proposed approach.

1) Different Variations of MP:

The motion predictor is most effective with an LSTM backbone, a five-frame context, and CIoU training, while remaining robust across training datasets. LSTM better captures abrupt nonlinear motion and shape changes than KF or EKF, whose predictions can be slow, overly sensitive, or inaccurate.

  • Backbone: KF and EKF underperform LSTM because KF assumes linear state transitions, while EKF remains sensitive to nonlinear dynamics despite tuned noise covariance.A lightweight MLP with 7.3K parameters is also evaluated, while larger or deformable targets require a more expressive model for complex motion and geometric patterns.
  • Backbone: LSTM-based MP handles abrupt motion-direction and target-shape changes more accurately than KF and EKF under nonlinear trajectories.KF adapts slowly to direction changes, EKF produces aggressive predictions, and both respond slowly to shape changes; LSTM trained on trajectory annotations performs better.
  • Context Length: A context length of 5 balances performance and latency, whereas longer histories marginally help LaSOText but degrade Anti-UAV300 performance.Rapidly changing drone motion benefits mainly from recent observations because longer contexts can introduce noise.
  • Training Set: MPs trained on LaSOT or TrackingNet achieve comparable results and consistently outperform prior methods, indicating limited dependence on the training dataset.This robustness persists even when training on the relatively simpler TrackingNet dataset.
  • Training Loss: CIoU achieves the best results among IoU, DIoU, and CIoU losses by jointly considering overlap, center distance, and aspect ratio.These factors improve bounding-box alignment.

2) Different SAM Backbones: … VI. CONCLUSION

SAMOSA consistently improves tracking across SAM backbone configurations and linear/nonlinear motion subsets, with particular robustness to distractors and occlusion. The framework combines motion prediction, geometric constraints, and semantic error recovery to deliver strong generalization with controllable latency overhead.

  • 2) Different SAM Backbones:: SAMOSA outperforms competing methods across SAM 3 and SAM 2.1 backbone configurations on LaSOText and Anti-UAV300.SAMITE is close under SAM 2.1-B+ but consistently underperforms under other configurations.
  • 2) Different SAM Backbones:: SAM 3 provides clear improvements over SAM 2.1, while SAMOSA further improves performance on both evaluated datasets.
  • 3) Performance in Nonlinear Scenes:: Motion nonlinearity is quantified from annotated trajectories using acceleration magnitude, acceleration angle deviation, and jerk.A frame is labeled nonlinear when any indicator exceeds its corresponding threshold.
  • 3) Performance in Nonlinear Scenes:: SAMOSA consistently outperforms baselines on both linear and nonlinear subsets, with particularly strong gains on nonlinear Anti-UAV300 RGB videos.The comparison is reported across the dataset splits in Table IX.
  • E. Qualitative Results: SAMOSA maintains correct target association against visually similar distractors by jointly using motion consistency and geometric constraints.Baseline trackers are easily confused by nearby objects with similar appearance.
  • E. Qualitative Results: During frequent crossing and occlusion of two skating shoes, S​​AMOSA provides more stable tracking than baselines that drift or produce incomplete results.
  • VI. CONCLUSION: SAMOSA addresses complex nonlinear VOT through a high-order Markov Motion Predictor and an Error Detection-Recovery mechanism using motion, geometry, and semantic cues.The framework targets nonlinear motion modeling and explicit error detection and recovery.
  • VI. CONCLUSION: With controllable latency overhead, SAMOSA achieves state-of-the-art performance and strong generalization across general-purpose VOT and challenging anti-UAV benchmarks.As a lightweight, pluggable adapter, it can integrate with future generations of Segment Anything models.
Loading 2605.22538v1…