PaperlayerView exact revision on arXiv

Source-linked AI summary

MaST: Motion-aware Sparse Pipeline for Lightweight Object Tracking

Qingmao Wei, Fagui Liu, Dengke Zhang, Qingze He, Quan Tang

arXiv:2608.24365v1cs.CV

TL;DR

Transformer trackers are computationally expensive on edge devices, and existing token-pruning methods may use noisy early scores or revert to dense prediction heads. MaST combines motion-aware token sparsification with a natively sparse prediction head, achieving Pareto-optimal speed–accuracy trade-offs and higher measured edge-platform speeds. Its local search-region design remains bounded under sufficiently fast motion or prolonged occlusion.

  • Problem

    Dense Transformer token processing hinders real-time tracking on edge devices, while existing sparsification often prunes late because early scores are noisy and still uses dense prediction heads.

  • Method

    MaST combines a history-based motion prior for early search-region token reduction with a score-first, regress-once head operating directly on retained unstructured tokens.

  • Results

    MaST achieves Pareto-optimal speed–accuracy trade-offs on edge platforms and competitive or superior accuracy with substantially higher measured inference speeds across multiple benchmarks.

  • Takeaways & Limitations

    End-to-end sparsity from tokens to boxes is presented as a practical path to real-time tracking on edge platforms.

  • Takeaways & Limitations

    MaST remains limited by the local search region when inter-frame displacement exceeds its radius or occlusion is prolonged.

Abstract

from arXiv · show

Transformer-based object trackers are renowned for their strong performance, yet dense token processing often leads to prohibitive computational cost, limiting real-time deployment on edge devices. While recent works explore token pruning to reduce computation, they often stop short of an end-to-end sparse pipeline, as early-layer token scores can be noisy without a motion prior, and many trackers ultimately fall back to dense reshaping to feed the dense prediction head that partially negates the savings. We introduce Motion-aware Sparse Tracker (MaST), a sparse tracking framework that makes sparsity effective from tokens to boxes. First, MaST injects a lightweight motion prior to refine cross-attention-based importance scores, enabling earlier and more stable token reduction in the search region. Second, we introduce a natively sparse prediction head that operates directly on the retained unstructured tokens with a score-first, regress-once design, eliminating dense padding/reshaping and reducing redundant computation. Extensive experiments on multiple benchmarks demonstrate that MaST establishes new state of the art among lightweight trackers, where MaST-tiny attains 63.8 AUC on LaSOT and 80.1 SUC on TrackingNet, surpassing the prior best AsymTrack-S by +1.0 AUC and +2.2 SUC while running at 152 FPS on Jetson Nano, nearly twice as fast as AsymTrack-S at 88 FPS. Code is available at https://github.com/TsingWei/MaST.

1 Introduction

Visual tracking must estimate a target’s trajectory despite challenging visual changes, while Transformer trackers face costly dense processing that hinders edge deployment. MaST addresses this with motion-guided early token reduction and a natively sparse prediction head, achieving competitive accuracy at higher measured edge-device speeds.

  • Motivation: Transformer trackers incur quadratic attention cost over long token sequences, limiting deployment on resource-constrained platforms.Lightweight architectures reduce discriminative power, while adaptive computation leaves worst-case computational cost largely unchanged.
  • Limitations of Existing Sparsification: Early attention maps are diffuse and noisy, making conventional importance scores unreliable and causing pruning to be deferred to intermediate layers.Without spatial guidance, earlier pruning sharply reduces accuracy; with ground-truth guidance, accuracy remains nearly flat while GMACs decrease.
  • Limitations of Existing Sparsification: Dense prediction heads require sparse tokens to be padded and reshaped into full feature maps, wasting computation and potentially pruning the target-center token.This sparse-backbone/dense-head mismatch caps speedups and can degrade localization accuracy.
  • MaST: MaST injects a history-based motion prior into search-region token selection for early, task-driven reduction.Its design targets the temporal motion continuity that appearance-only criteria ignore.
  • MaST: MaST’s score-first, regress-once head operates directly on retained unstructured tokens, eliminating dense reshaping and redundant computation.This completes the sparse pipeline from token reduction through target decoding.
  • Results: MaST achieves competitive or superior accuracy with substantially higher measured inference speeds on Raspberry Pi and Jetson Nano across multiple benchmarks.The experiments cover LaSOT, TrackingNet, GOT-10k, and additional benchmarks.

2 Related Works

Related work improves lightweight tracking through compact models, compression, adaptive computation, temporal priors, and token sparsification. These approaches retain limitations involving representational capacity, accuracy, late or noisy pruning, and insufficiently explicit spatial localization.

  • Lightweight Visual Tracking: Compact tracker designs reduce parameter count but sacrifice representational capacity, while layer pruning and distillation can cause non-trivial accuracy degradation.These methods are presented as major approaches for narrowing the efficiency gap in Transformer tracking.
  • Lightweight Visual Tracking: Adaptive computation methods reduce average latency but leave worst-case computational cost largely unchanged.This limitation accompanies the broader efficiency challenges of one-stream Transformer trackers.
  • Temporal Priors in Visual Tracking: Temporal priors in modern trackers commonly use Hanning or cosine windows as post-processing penalties on predicted score maps.Other Transformer trackers encode historical predictions as tokens for learned temporal modeling.
  • Token Sparsification in Vision Transformers: Vision Transformer token sparsification enables pruning strategies unavailable to CNN-based trackers, including OSTrack’s cross-attention-based top-K search-token selection.Because early-layer attention scores are noisy, many methods prune progressively at intermediate layers.

3 Method

MaST combines early motion-aware token sparsification with a natively sparse prediction head, processing only retained search tokens through decoding without dense reshaping.

  • Framework overview: MaST processes template and search images as patch-token sequences through a Transformer encoder followed by sparsification and prediction.The framework divides images into non-overlapping patches, fuses features in sequence form, and further processes retained tokens with the prediction head.
  • Importance scoring: Cross-attention importance scores use the center template token to provide a lightweight measure of search-token relevance.The formulation simplifies aggregation to the interaction between each search token and the representative center template token.
  • Motion-aware sparsification: A motion window reweights attention scores using the previous predicted box, prioritizing tokens near the likely target region.The window is implemented as a 2D Gaussian whose spread depends on the previous box dimensions; γ = 0.5 provides reasonable performance.
  • Motion-aware sparsification: The combined scores retain only the top-K search tokens, after which subsequent Transformer blocks operate on the reduced token set.The motion prior helps prioritize likely target tokens even when early cross-attention scores are diffuse.
  • Sparse prediction head: The sparse head scores every retained token, selects the highest-scoring token, and regresses box parameters only once at that location.This score-first, regress-once path avoids dense feature computation while preserving local anchor-based box decoding and converting coordinates to search-image pixels.
  • Sparse prediction head: MaST preserves the dense head’s box parameterization while removing dense reshaping and redundant regressions, so head complexity scales with the retained-token count.The regression branch is evaluated only at the selected token, while scoring remains proportional to NK.

4 Experiment

MaST combines motion-guided early token sparsification with a natively sparse prediction head, achieving strong speed–accuracy trade-offs across benchmarks and edge platforms. Ablations show that motion guidance and sparse decoding preserve accuracy while reducing computation.

  • Benchmark comparisons: MaST-tiny achieves 63.8 AUC on LaSOT and 80.1 SUC on TrackingNet, leading the approximately 1G MACs groups.
  • Benchmark comparisons: MaST-tiny attains 66.6 AO on GOT-10k and MaST-small reaches 70.0 AO in the approximately 2G group.
  • Benchmark comparisons: MaST-tiny reaches 42.8 AUC on LaSOText at 22.6 FPS, over 3× faster than AsymTrack-B at 6.2 FPS.
  • Benchmark comparisons: MaST-tiny achieves 66.6 AUC on UAV123 at 22.6 FPS on Raspberry Pi 5, while MaST-nano reaches 30.1 FPS with 62.9 AUC.
  • Sparsification ablations: Fusing attention with motion nearly closes the accuracy gap to the dense baseline while preserving compute savings, outperforming attention-only and motion-only sparsification.
  • Prediction-head ablations: The sparse MLP head reaches 23.2 FPS with 63.8 AUC, matching the dense MLP counterpart while avoiding the reshaping bottleneck of convolutional heads.

5 Conclusion

MaST unifies motion-aware early token pruning with a natively sparse prediction head, enabling end-to-end sparsity and Pareto-optimal speed–accuracy trade-offs on edge platforms.

  • MaST combines motion-aware early token pruning with a natively sparse prediction head.The motion prior enables first-layer search-token reduction, while the score-first, regress-once head operates directly on retained tokens.
  • A remaining limitation is that high-resolution inputs still incur substantial attention computation before sparsification begins.The paper motivates input-adaptive pruning as future work.

A.1 Training Details.

MaST training uses paired template–search crops, a two-stage dense-to-sparse procedure, and gradual token-retention warmup across model variants.

  • Model Inputs: MaST takes template and search crops divided into non-overlapping 16 × 16 patches, with model-specific input sizes and token counts.MaST-nano and MaST-tiny use 128 × 128 templates and 256 × 256 search regions; MaST-small uses 192 × 192 and 384 × 384.
  • Augmentation: Training constructs template–search pairs from the same video sequence and applies random horizontal flipping and brightness jittering.The batch size is 128 image pairs.
  • Training Pipeline: Training proceeds in two stages: a dense base model is trained first, followed by sparse fine-tuning of three variants.Stage 1 runs for 300 epochs; Stage 2 fine-tunes MaST-tiny, MaST-small, and MaST-nano for 50 epochs each.
  • Training Pipeline: MaST-tiny preserves the base resolution, MaST-small interpolates position encodings for larger inputs, and MaST-nano removes the last four encoder layers.MaST-nano is fine-tuned with an eight-layer backbone at 128/256 resolution.
  • Sparsification Warmup: Token retention is linearly annealed from 100% to 30% during the first 10 sparse-fine-tuning epochs, then held constant.This warmup applies to all Stage 2 variants.
  • Motion Prior During Training: During training, the motion-window center is set to the search-image center because jittered crops approximately center the target.At test time, the window centers on the previous frame’s predicted bounding box.

A.2 Analysis of Motion Window.

The motion-window analysis compares window types, importance-score integration, and dynamic resizing, showing that combining spatial motion guidance with importance scores performs best.

  • Windowing Strategies: Combining importance scores with a Gaussian window achieves the best overall performance.The analysis also compares fixed and dynamically resized windowing approaches.
  • Windowing Strategies: The ablation evaluates Gaussian versus Cosine windows, cross-attention importance scores, and dynamic resizing based on the previous bounding box.Dynamic resizing adapts to the previous frame’s width, height, and aspect ratio.
  • Windowing Strategies: Motion-only sparsification outperforms cross-attention-only selection, while their combination consistently surpasses either individually.This supports using both spatial motion guidance and cross-attention scores for token selection.
  • Appearance Importance: Using the center template token is slightly more accurate and marginally faster than max-pooling over all template tokens.The paper attributes this to target-centered templates, where the center token is usually a clean foreground cue.
  • Scaling Context: The study compares accuracy and throughput when scaling motion-aware sparsification to a ViT-Base tracker and larger input resolutions.The table reports accuracy and FPS for the scaled configurations.
  • Benchmark Comparison: The UAV benchmark comparison distinguishes traditional correlation-filter trackers from deep trackers and marks the best and second-best overall results.The table uses a double rule and red/blue result highlighting for these distinctions.

A.3 Scaling to Larger Models

Motion-aware sparsification scales to a ViT-Base tracker and larger inputs, substantially increasing throughput while retaining reported tracking accuracy.

  • 215 FPS at 224^2 input with 30% retention nearly doubles throughput from 116 FPS, while achieving 69.8 LaSOT AUC and 70.2 GOT-10k AO.The sparse configuration uses a ViT-Base tracker.
  • 104 FPS at 378^2 input more than doubles throughput from 51 FPS, while achieving 70.4 AUC and 71.5 AO.The results show the same motion-aware sparsification strategy at higher resolution.

A.4 Additional Details on Global Box Decoding.

MaST evaluates two sparse-compatible global box-decoding alternatives, but both face optimization difficulties compared with the adopted design.

  • Loc Tokens appends four box tokens to the encoder, while Trans. Decoder uses four queries that cross-attend to search tokens after encoding.Both alternatives predict box coordinates globally and are naturally compatible with sparse tokens.
  • Both global localization designs converged much more slowly in image-pair IoU than the adopted approach.Additional training, head replacement, and teacher distillation did not resolve the optimization difficulty.
  • Table 12 groups efficient-tracking comparisons across three large-scale benchmarks by computational cost.Red and blue identify the best and second-best results within each group; gray FPS values indicate speeds below 20 FPS.

A.5 Comparison on UAV Benchmarks.

MaST variants deliver competitive or leading accuracy across UAVDT, UAV123, and DTB70 while maintaining low computation and strong edge-device throughput.

  • UAVDT: 63.0 success and 85.9 precision make MaST-small the top UAVDT tracker, surpassing ORTrack-DeiT by +2.9 and +2.5.MaST-small uses 1.82 G MACs versus 2.39 G and runs at 98 FPS versus 84 FPS on Jetson Orin Nano.
  • UAV123: 86.9 precision makes MaST-tiny best on UAV123, while its 66.6 success is second-highest at 0.836 G MACs.It runs at 22.6 FPS on Raspberry Pi 5 and 152 FPS on Jetson Orin Nano.
  • DTB70: 66.2 success and 85.6 precision make MaST-small competitive on DTB70, with second-highest success behind the 66.4 shared by two baselines.MaST-nano is the lightest variant at 0.585 G MACs and reaches 230 FPS on Jetson Nano.
  • Overall: Across the three UAV benchmarks, MaST provides a strong accuracy–efficiency trade-off on general and edge hardware.The reported results indicate favorable transfer of motion-aware sparse computation to UAV deployment scenarios.

A.6 Running Latency on More Platforms.

MaST’s latency and robustness analyses examine edge deployment, frame-rate effects, motion-aware token selection, and failure boundaries under difficult tracking conditions.

  • Latency and frame rate: At 240 FPS, MaST-tiny exceeds OSTrack-Tiny by +2.9 AUC, +4.0 PNorm, and +4.0 P, compared with smaller gains at 30 FPS.Higher input frame rates provide a clearer motion prior, while lower tracker latency improves temporal continuity exploitation.
  • Latency and frame rate: 152 FPS on Jetson Nano versus 83 FPS for OSTrack-Tiny makes high-frame-rate online tracking more practical in deployment.From 30 to 240 FPS, MaST-tiny gains +2.5 AUC while OSTrack-Tiny improves by only +0.7 AUC.
  • Motion-aware sparsification: 63.8 LaSOT AUC from appearance-attention and motion fusion exceeds 62.6 for motion alone and 60.5 for attention alone.The soft Gaussian prior still permits strong off-center appearance evidence to retain tokens.
  • Robustness: A larger search view improves challenging-attribute results, indicating that long occlusion and large displacement are constrained by local crop size and token budget.Under the same 4× search factor and 256-pixel input as FERMT, MaST-tiny is less robust to severe occlusion and fast motion; a 5× factor and 384-pixel input lets MaST-small surpass FERMT with less computation.
  • Robustness: With 30% tokens and a motion window, MaST closely matches the 100%-token model in rapid-motion visualization, whereas naive sparsification fails completely.The comparison attributes the rapid-motion behavior to the motion-aware component.
  • Failure cases: Low-quality UAV footage with severe appearance ambiguity can exceed even the full-token model’s capability, representing a limitation shared by existing trackers.This degradation is reported as inherent to the tracking conditions rather than caused by sparsification or motion-aware strategies.
  • Limitations: MaST inherits a local search region, so displacement beyond its radius or prolonged occlusion prevents relocation outside the local window.The authors suggest reallocating sparse-computation savings toward wider search or lightweight global context, while the current retention rate remains fixed for predictable latency.
Loading 2608.24365v1…