Source-linked AI summary
LightTrack: Finding Lightweight Neural Networks for Object Tracking via One-Shot Architecture Search
Bin Yan, Houwen Peng, Kan Wu, Dong Wang, Jianlong Fu, Huchuan Lu
TL;DR
Object tracking models have become too computationally and memory intensive for many resource-constrained deployments. LightTrack uses one-shot neural architecture search to design lightweight trackers, achieving state-of-the-art performance with substantially lower complexity and faster execution on restricted hardware.
Problem
Increasingly heavy and expensive tracking models limit deployment on resource-constrained hardware, motivating a better accuracy–complexity trade-off.
Method
LightTrack reformulates one-shot NAS for tracking and searches lightweight backbone and head architectures using tracking data, accuracy, and model complexity.
Results
LightTrack achieves state-of-the-art tracking performance with fewer Flops and parameters, including a 530M-Flop tracker that surpasses SiamRPN++ by 4.6% on VOT-19.
Takeaways & Limitations
The searched trackers can run in real time on diverse resource-restricted platforms, potentially narrowing the gap between academic tracking models and industrial deployment.
Takeaways & Limitations
The approach assumes object trackers receive model pre-training on an image-classification task for good initialization.
Abstract
from arXiv · showhide
Object tracking has achieved significant progress over the past few years. However, state-of-the-art trackers become increasingly heavy and expensive, which limits their deployments in resource-constrained applications. In this work, we present LightTrack, which uses neural architecture search (NAS) to design more lightweight and efficient object trackers. Comprehensive experiments show that our LightTrack is effective. It can find trackers that achieve superior performance compared to handcrafted SOTA trackers, such as SiamRPN++ and Ocean, while using much fewer model Flops and parameters. Moreover, when deployed on resource-constrained mobile chipsets, the discovered trackers run much faster. For example, on Snapdragon 845 Adreno GPU, LightTrack runs $12\times$ faster than Ocean, while using $13\times$ fewer parameters and $38\times$ fewer Flops. Such improvements might narrow the gap between academic models and industrial deployments in object tracking task. LightTrack is released at https://github.com/researchmm/LightTrack.
1. Introduction
LightTrack addresses the growing computational burden of object tracking by automating lightweight tracker design with neural architecture search. Its searched trackers achieve strong accuracy with substantially lower complexity and faster deployment on resource-limited hardware.
- Motivation: State-of-the-art trackers increasingly require substantial computation and memory, hindering deployment in applications such as camera drones, industrial robotics, and driving assistance.SiamRPN++ uses 7.1G Flops and 11.2M parameters, while Ocean uses 20.3G Flops and 25.9M parameters.
- Motivation: Model compression can reduce complexity but may degrade performance, whereas handcrafted compact-model design is engineering-intensive and dependent on human expertise.
- Approach: LightTrack automates lightweight tracker design with neural architecture search, addressing the challenge that tracking typically requires ImageNet pre-training while NAS needs task-specific feedback.
- Approach: Its one-shot search encodes candidate backbones and heads in supernets trained once, then evaluates architectures using tracking accuracy and model complexity while inheriting supernet weights.
- Results: 530M Flops LightTrack reaches 0.33 EAO on VOT-19, surpassing SiamRPN++ by 4.6% while reducing its 48.9G-Flop complexity by 98.9%.On Snapdragon 845 Adreno 630, it runs 12× faster than Ocean at 38.4 versus 3.2 fps, with 13× fewer parameters and 38× fewer Flops.
- Contributions: The work presents automated neural-architecture design for tracking and a lightweight search pipeline whose searched trackers achieve state-of-the-art performance on resource-limited platforms.
2. Related Work
Prior tracking research improved localization, feature representation, and online updating, but these advances increased computation and memory demands. One-shot NAS reduces search cost, while LightTrack extends architecture search to jointly design lightweight tracking backbones and heads.
- Object Tracking: Siamese trackers evolved from correspondence-based SiamFC and SINT toward improved box estimation, stronger backbones, and online update mechanisms.
- Object Tracking: SiamRPN++ and Ocean use ResNet-50 backbones to strengthen feature representation, but this design substantially increases computation and memory footprint.
- Efficiency Gap: The mobile setting commonly requires fewer than 600M Flops, yet SiamRPN++ uses 48.9G Flops and exceeds that workload by approximately 80 times.
- Efficiency Gap: The related-work gap is finding object trackers that balance accuracy and complexity for real-world deployment.
- Neural Architecture Search: One-shot NAS amortizes search by training one over-parameterized supernet whose weights are shared across candidate subnetworks.
- Neural Architecture Search: Unlike DetNAS, LightTrack targets object tracking, searches backbone and head architectures simultaneously, and introduces a tracking-specific lightweight search space.
3. Preliminaries on One-Shot NAS
One-shot NAS represents many candidate architectures as weight-sharing subnets within a single supernet. It trains the shared weights once and searches candidates using validation accuracy and complexity without separately training each subnet.
- One-Shot NAS: The architecture space A is encoded in a supernet N(A, W), with supernet weights W shared across all candidate subnetworks α.
- One-Shot NAS: Optimal architecture search is formulated as nested optimization: training supernet weights on the training loss and ranking subnetworks by validation accuracy.
- One-Shot NAS: Only the single supernet is trained, allowing subnetworks to inherit learned weights and be evaluated without separate training.This greatly speeds architecture performance estimation.
- Optimization: Single-path uniform sampling trains one randomly selected supernet path per batch to reduce memory usage and decouple supernet training from architecture optimization.
4. LightTrack
LightTrack addresses lightweight tracking architecture search by jointly optimizing backbone and head supernets with one-shot NAS. Its pipeline combines ImageNet pretraining, tracking-supernet training, and evolutionary search under deployment budgets.
- Method: LightTrack targets three challenges: combining ImageNet pretraining with tracking supervision, searching backbone and head jointly, and designing a lightweight tracking search space.The method uses one-shot NAS to address these requirements within a unified tracking architecture search.
- Backbone pretraining: ImageNet pretraining is performed once on a backbone supernet, whose shared weights initialize subsequent tracking architecture search.Weight sharing avoids individually pretraining every backbone candidate and reduces training costs by orders of magnitude.
- Tracking supernet: The tracking supernet jointly trains backbone and head weights on tracking data, with candidate architectures evaluated by inheriting the trained supernet weights.The head supernet represents localization architectures, while joint search considers the complete tracker rather than either component alone.
- Architecture search: Evolutionary search ranks candidate architectures using tracking accuracy and model-complexity constraints, including preset parameter and Flops budgets.The same trained supernet can support repeated searches under different deployment constraints.
- Search space: The lightweight search space uses depthwise separable convolutions, MBConv blocks, elastic head depth, and automatic selection of the backbone output feature layer.The backbone space contains about 7.8×10^10 candidates, while the head space contains about 3.9×10^8 candidates.
- Tracking supernet: The tracking supernet follows a Siamese design: exemplar and search images share backbone processing, their features are cross-correlated, and classification and regression heads localize the object.This structure supplies the tracking-task computation used during supernet training.
5. Experiments
Experiments across four tracking benchmarks and resource-constrained hardware show that LightTrack searches competitive lightweight trackers, while ablations identify benefits from searching backbone, feature-output, and head components.
- Experimental Setup: LightTrack evaluates performance on VOT-19, GOT-10K, TrackingNet, and LaSOT, and tests runtime on Apple, Huawei, and Xiaomi mobile platforms.Three resource-constrained variants are provided: Mobile, LargeA, and LargeB.
- Benchmark Results: LightTrack-Mobile achieves superior VOT-19 performance to offline SOTA trackers while using more than 10 times fewer model Flops and parameters.LightTrack-LargeB surpasses ATOM and DiMP by 5.6% and 3.6%, respectively.
- Benchmark Results: On GOT-10K, LightTrack-Mobile exceeds SiamFC++(G) and Ocean(off) by 1.6% and 1.9% AO, while LightTrack-LargeB beats DiMP-50 by 1.2% using 8× fewer parameters.
- Benchmark Results: On TrackingNet, LightTrack-Mobile reaches 69.5% precision, 0.8% above DiMP-50, with comparable Pnorm and AUC using 96% and 92% fewer parameters.On LaSOT, LightTrack-LargeB reaches a 0.555 success score, surpassing SiamFC++(G) and Ocean-offline by 1.2% and 2.9%.
- Runtime Results: 3–6× speedups over SiamRPN++ and 5–17× speedups over Ocean enable LightTrack to reach real-time operation on resource-limited mobile platforms.The comparisons cover Snapdragon 845, Apple A10 Fusion, and Kirin 985 GPU/DSP platforms.
- Ablation and Analysis: Searching the backbone, feature-output layer, and head improves the handcrafted baseline, with the searchable head adding 2.9% EAO and joint search performing best.The baseline EAO is 0.268, while searching the output feature layer raises performance to 0.307.
6. Conclusion
LightTrack reformulates one-shot NAS for object tracking and designs lightweight trackers that maintain state-of-the-art performance with substantially fewer computational resources. Experiments show real-time operation across resource-restricted platforms.
- LightTrack reformulates one-shot NAS specifically for object tracking and introduces an effective tracking-oriented search space.
- LightTrack achieves state-of-the-art performance while using much fewer Flops and parameters than existing trackers.
- LightTrack runs in real time on diverse resource-restricted platforms.