PaperlayerView exact revision on arXiv

Source-linked AI summary

Exploring Decoupled Spatio-Temporal Consistency Learning and Self-Prompting Evolution for Self-Supervised Tracking

Yaozong Zheng, Bineng Zhong, Qihua Liang, Ning Li, Haiying Xia, Shuxiang Song, Rongrong Ji

arXiv:2507.21606v2cs.CV

TL;DR

Manual box annotations limit the scale and diversity of tracking data, motivating SSTrack++, a weak-to-strong self-supervised tracker with decoupled spatio-temporal consistency, self-prompting evolution, and instance contrastive learning. Across ten benchmarks, it achieves strong performance with limited annotations and narrows the gap with fully supervised tracking, while forward-tracking localization remains a reported limitation.

  • Problem

    Manual box annotations are costly and limit benchmark scalability, while existing self-supervised trackers struggle to learn accurate spatio-temporal relationships from unlabeled videos.

  • Method

    SSTrack++ combines weak-to-strong training, decoupled spatio-temporal consistency learning, self-prompting evolution, and multi-view instance contrastive learning.

  • Results

    Across ten tracking benchmarks, SSTrack++ achieves excellent performance with limited annotations and significantly narrows the gap between self-supervised and fully supervised methods.

  • Takeaways & Limitations

    SSTrack++ supports learning generalizable tracking representations and realistic target appearance and motion variations from low-annotation self-supervised data.

  • Takeaways & Limitations

    Backward tracking performance depends somewhat on forward-tracking localization accuracy, so improving forward tracking could further improve self-supervised tracking.

Abstract

from arXiv · show

The success of visual tracking has been largely driven by datasets with manual box annotations. However, these box annotations require tremendous human effort, limiting the scale and diversity of existing tracking datasets. In this work, we present a novel high-performance Self-Supervised Tracking model named SSTrack++, designed to eliminate the need of box annotations. Specifically, we design an effective weak-to-strong self-supervised training framework that aims to narrow the feature distribution gap between labeled and unlabeled frames. Building upon this framework, a decoupled spatio-temporal consistency training strategy is introduced to capture rich target information across timestamps by leveraging global spatial localization and local temporal association. Then, a self-prompting evolution module is designed to jointly mine the target appearance evolution at both the feature and decision levels, enabling more robust adaptation to complex and diverse unlabeled tracking scenarios. Furthermore, an instance contrastive loss is formulated to learn instance-level correspondences from a multi-view perspective, providing robust instance-level supervision without any additional annotations. This new design paradigm enables SSTrack++ to not only learn a generalizable tracking representation in a low-annotation self-supervised manner, but also makes it possible to simulate realistic appearance and motion variations of target instances in real-world scenarios. Extensive experiments on ten benchmark datasets demonstrate that SSTrack++ surpasses SOTA self-supervised tracking methods, achieving an improvement of more than 25.8%, 21.4%, and 15.8% in AUC (AO) score on the GOT10K, LaSOT, TrackingNet datasets, respectively, thereby significantly narrowing the performance gap with fully supervised trackers. Code is available at https://github.com/GXNU-ZhongLab/SSTrack.

1 Introduction

SSTrack++ addresses the limited scalability of box-annotated tracking datasets by learning from unlabeled videos with weak-to-strong self-supervision. Its decoupled consistency learning, self-prompting evolution, and instance contrastive learning target spatio-temporal representation and instance correspondence, with experiments showing strong benchmark performance.

  • Motivation: Manual box annotation limits the scalability and diversity of visual tracking benchmarks, while existing self-supervised methods struggle to capture accurate spatio-temporal evolution.These limitations motivate learning and associating tracked instances directly from unlabeled videos.
  • Framework: SSTrack++ uses a weak-to-strong self-supervised framework to narrow feature distribution differences between labeled and unlabeled frames.The framework is designed to exploit abundant unlabeled video data.
  • Core components: Decoupled spatio-temporal consistency learning combines global spatial localization with local temporal association to learn target information across timestamps.The strategy performs forward tracking for global object localization and backward tracking for local appearance and motion perception.
  • Core components: Self-prompting evolution jointly mines target information at feature and decision levels to improve adaptation to complex and diverse unlabeled tracking scenarios.It addresses continuously evolving target appearance and motion by dynamically updating target reference prompts.
  • Contributions and evaluation: An instance contrastive loss learns multi-view instance correspondences without labels, while SSTrack++ extends the conference version with deeper analysis, ablations, and visual results.Compared with the conference version, it improves LaSOT AUC by 2.1%.
  • Contributions and evaluation: Experiments on ten tracking benchmarks show excellent performance with limited annotations and a substantially narrower gap between self-supervised and fully supervised tracking.The evaluation covers datasets including LaSOT, TrackingNet, GOT10K, VOT, UAV, VastTrack, and OTB100.

2 Related Work

The related work frames large-scale annotation as a bottleneck for supervised tracking and reviews cycle-consistency and contrastive self-supervised approaches. SSTrack++ differs by using decoupled spatio-temporal modeling to reduce the impact of low-quality sample pairs.

  • Fully-Supervised Tracking: Fully supervised trackers achieve high performance using large labeled datasets, but constructing such datasets is time-consuming and costly.The resulting data limitations create distribution gaps with real-world scenarios.
  • Fully-Supervised Tracking: Large-scale, high-quality datasets are becoming a bottleneck for supervised tracking, motivating research into effective self-supervised alternatives.The stated goal is to alleviate reliance on expensive labeled data.
  • Self-Supervised Tracking: Existing self-supervised tracking methods are commonly based on cycle consistency or contrastive learning.Examples include self-SDCT and CycleSiam for cyclic consistency, alongside contrastive correspondence-learning approaches.
  • Self-Supervised Tracking: SSTrack++ introduces decoupled spatio-temporal modeling to avoid the impact of low-quality sample pairs during self-supervised training.The framework is presented as a new perspective on minimal-supervision visual tracking.

3 Methodology

SSTrack++ is a transformer-based weak-to-strong self-supervised tracker trained on unlabeled videos with an initial annotated frame. A dynamic spatio-temporal memory propagates temporal cues while the model localizes the instance frame by frame.

  • Framework overview: SSTrack++ extends SSTrack into a transformer-based self-supervised tracking framework with improved training architecture and efficient spatio-temporal memory use.The extension is reported to improve performance while maintaining the same computational efficiency.
  • Task formulation: The task uses an unlabeled RGB video and one initially annotated frame, then trains a tracker to localize the target in subsequent frames.The initial bounding box supplies the weak supervisory signal.
  • Task formulation: A transformer-based model localizes the instance frame by frame and uses dynamic spatio-temporal memory to propagate temporal cues through the video.The memory is the mechanism for carrying temporal information across the sequence.

3.3 Tracking Background

Fully supervised tracking embeds frame–bounding-box pairs into a tracking network to guide localization in search frames. SSTrack++ instead pursues high-performance tracking with minimal supervision through weak-to-strong training and spatio-temporal components.

  • Fully-Supervised Tracking: Fully supervised trackers embed paired frames and bounding boxes into a tracking network to guide localization.The paired inputs may consist of an initial pair or multiple frame–box pairs.
  • Fully-Supervised Tracking: For a search frame, the network predicts a bounding box using the supplied reference frame and bounding-box pairs as guidance.The notation defines the predicted search-frame box as B_s.
  • Self-Supervised Tracking: SSTrack++ replaces dependence on fully supervised labels with a weak-to-strong framework, decoupled spatio-temporal consistency learning, self-prompting evolution, and instance contrastive learning.These components are presented as the basis for high-performance self-supervised tracking.

3.4 Decoupled spatio-temporal Consistency Learning Strategy

SSTrack++ decouples tracking into global spatial localization and local temporal association, allowing target instances to be identified and linked across diverse video scenarios. The strategy is used during training, while inference retains only the local tracking component for efficiency.

  • Global-to-local tracking addresses the weakness of randomly cropped pairs, which can produce low-quality samples and underuse spatio-temporal context.
  • The decoupled strategy jointly achieves global spatial localization and local temporal association within one tracking framework.
  • Forward tracking globally searches an uncropped frame for the target’s spatial location using a shared ViT.
  • Backward tracking crops the current search frame and augments the initial frame to create multiple local views for temporal association.
  • During inference, SSTrack++ keeps only the local backward tracker to improve efficiency.

3.5 Self-Prompting Evolution Module

The self-prompting evolution module continually selects and updates target prompts from model predictions. It combines feature-level and decision-level evidence to retain informative appearance cues and adapt to changing scenes.

  • SPE dynamically updates target reference prompts because target appearance and motion evolve in long-term, highly dynamic environments.
  • The module maintains fine-grained local prompts and global tokens that integrate information across multiple timestamps.
  • Cross-attention produces search-token importance weights, while classification confidence and masking identify target-relevant regions and suppress background tokens.
  • SSTrack++ jointly evaluates feature-level and decision-level information to select information-rich target tokens for propagation.
  • The evolution process selects the top k −1 informative raw tokens and aggregates lower-quality tokens into a new global target prompt.
  • The strategy preserves beneficial features while progressively optimizing prompts with lower information content.

3.6 Instance Contrastive Learning

SSTrack++ uses instance-level contrastive learning to obtain robust correspondences from augmented views without box annotations. The loss separates representations of different instances while tolerating imperfect localization through filtering and feature-level supervision.

  • Frame-level similarity is insufficient for distinguishing instances in open and complex tracking scenarios, motivating instance contrastive learning.
  • Data augmentation creates multiple views that simulate appearance changes across timestamps and provide target correspondences without annotations.
  • Predicted-box masks isolate target regions from backgrounds before pooling produces instance representations.
  • Positive samples are augmented views of the same instance, whereas negative samples come from different instances and are compared using cosine similarity.
  • Invalid or extremely small predicted boxes are excluded to prevent degenerate tracking-instance representations.
  • Feature-level similarity and joint constraints from temporal consistency and prompt evolution make the representation learning more tolerant to localization noise.
  • The resulting objective pulls same-instance representations together and separates different-instance representations without extensive box annotations.

3.7 Weak-to-Strong Self-Supervised Training Framework

The weak-to-strong framework uses reliable weakly augmented predictions as supervision for strongly augmented frames, enabling self-supervised learning from unlabeled videos. Training combines supervised and unsupervised objectives while stopping gradients through the weak branch.

  • Earlier cycle-consistency methods leave forward tracking unsupervised, limiting their use of unlabeled video.
  • Weakly augmented inputs provide lower-noise, lower-variance predictions that supervise localization under strong augmentations.
  • Strong–weak pairs are formed through horizontal flipping and random jitter, then processed as temporal autoregressive sequences.
  • The weak sample branch is gradient-stopped, so its parameters are not updated during training.
  • The weak branch predicts a box, while the strong branch produces a corresponding prediction and temporal token representation.
  • The supervised loss uses annotated initial frames and combines classification, regression, and contrastive terms.
  • For unlabeled frames, the unsupervised loss enforces agreement between weak- and strongly augmented predictions.
  • Combining the training objectives allows the model to exploit unlabeled videos in a self-supervised manner for high-performance tracking.

3.8 Discussion on Fully Supervised Tracking

Self-supervised tracking should be valued beyond matching fully supervised methods: it reduces annotation dependence, supports broader unlabeled-data use, and may improve cross-domain generalization. The paper therefore frames it primarily as a foundational pre-training paradigm rather than a replacement for fully supervised tracking.

  • Self-supervised tracking should not be evaluated solely by whether it matches or surpasses fully supervised methods.
  • It can reduce annotation dependence by using abundant unlabeled videos to learn spatio-temporal representations.
  • It offers potential for cross-domain generalization because it does not rely on specific annotation distributions.The passage highlights nighttime UAV tracking, underwater environments, and camouflaged scenes as examples of differing data distributions.
  • The paper positions self-supervised tracking as a foundational pre-training paradigm rather than a substitute for fully supervised methods.

4 Experiments

Across diverse benchmarks, SSTrack++ consistently improves self-supervised tracking performance, narrows gaps with fully supervised trackers, and benefits from its decoupled consistency learning, self-prompting evolution, and instance contrastive loss.

  • Benchmark comparisons: 25.7% higher AO, 27.1% higher SR0.5, and 45.1% higher SR0.75 than TADS on GOT10K.The reported gain is attributed primarily to decoupled spatio-temporal consistency learning using labeled and unlabeled video data.
  • Benchmark comparisons: 66.9% AUC and 71.6% precision were achieved by SSTrack++-384 on LaSOT, reducing the AUC gap with supervised ODTrack to 7.3%.The results are associated with evolving target self-prompts and instance contrastive learning for appearance and motion information.
  • Benchmark comparisons: SSTrack++ outperformed TransT and SSTrack across multiple challenging LaSOT attributes, including rotation, fast motion, scale variation, and occlusion.The attribute analysis is presented as evidence of generalizability across challenging conditions.
  • Benchmark comparisons: 81.4% AUC, 87.1% PNorm, and 79.1% precision were achieved by SSTrack++-256 on TrackingNet.The method is reported to use dynamic temporal association of target instances in this short-term tracking benchmark.
  • Benchmark comparisons: SSTrack++ achieves best self-supervised mask-tracking results on VOT2020, with EAO scores of 50.3% for SSTrack-384 and 53.4% for SSTrack++-384.The results are reported for mask evaluations using Alpha-Refine to predict segmentation masks.
  • Ablation studies: The decoupled framework increased AO by 26.8%, adding spatio-temporal context produced a further 2.2% gain, and instance contrastive loss added 1.6%, 1.4%, and 1.6% in AO, SR0.5, and SR0.75.These ablations attribute complementary gains to consistency learning and instance-level representation discriminability.

5 Conclusion

SSTrack++ targets annotation-free self-supervised tracking through decoupled spatio-temporal consistency learning, self-prompting evolution, and instance-level contrastive supervision. Its reported limitation is that backward tracking depends partly on forward-tracking localization accuracy.

  • 5 Conclusion: SSTrack++ combines weak-to-strong training, decoupled spatio-temporal consistency learning, self-prompting evolution, and instance contrastive loss.The framework is designed for robust adaptation to complex and diverse unlabeled tracking scenarios while learning instance correspondences without labels.
  • 5 Conclusion: The self-prompting evolution module jointly mines target appearance evolution at feature and decision levels.This module is intended to improve adaptation across complex and diverse unlabeled tracking scenarios.
  • 5 Conclusion: The instance contrastive loss learns instance-level correspondences from multiple views without additional labels.It provides instance supervision within the self-supervised framework.
  • 5 Conclusion: Backward tracking performance depends somewhat on forward-tracking localization accuracy.The paper states that improving forward tracking could further enhance self-supervised tracking performance and narrow the gap with fully supervised methods.

6 Appendix

The appendix describes evaluation metrics and supporting ablations for SSTrack++. It reports that all three loss terms contribute to performance, while their weighting affects self-supervised results.

  • Evaluation Metrics: Tracking performance is evaluated with success score (AUC), normalized precision score (PNorm), and precision score (P).The evaluation follows each dataset's published protocol and corresponding toolkit.
  • Effect of different loss terms and weights: Table 12 compares different loss-weight ratios on the LaSOT benchmark.The ablation examines how weighting choices affect self-supervised performance.
  • Effect of different loss terms and weights: Reducing the weight of any individual loss term consistently causes a slight performance drop.This indicates that all three losses play indispensable roles during training.
  • Effect of different loss terms and weights: Table 13 confirms that each component contributes to overall performance to some extent.The comparison is conducted on the LaSOT benchmark.
Loading 2507.21606v2…