Source-linked AI summary
Spatially Supervised Recurrent Convolutional Neural Networks for Visual Object Tracking
Guanghan Ning, Zhi Zhang, Chen Huang, Zhihai He, Xiaobo Ren, Haohong Wang
TL;DR
Visual tracking must remain reliable under large appearance and motion changes, while existing approaches have limited feature robustness and spatiotemporal deep analysis was insufficiently studied. The paper proposes ROLO, combining YOLO features and preliminary locations with LSTM regression over location history and visual cues. Experiments report improved accuracy and robustness at low computational cost, with the best performance on most test sequences.
Problem
Visual tracking faces severe appearance and motion challenges, while handcrafted features lack robust semantic discrimination and spatiotemporal deep analysis was insufficiently studied.
Method
ROLO combines YOLO’s visual features and preliminary location inferences with LSTM-based spatiotemporal regression using historical locations and visual features.
Results
The tracker is reported as more accurate and robust than state-of-the-art methods while maintaining low computational cost, achieving the best performance on most test sequences.
Takeaways & Limitations
The approach can handle major occlusion and severe motion blur while exploiting both spatial and temporal information for tracking.
Takeaways & Limitations
Future work identifies efficient online learning for unseen dynamics and data association for multi-target tracking as unresolved directions.
Abstract
from arXiv · showhide
In this paper, we develop a new approach of spatially supervised recurrent convolutional neural networks for visual object tracking. Our recurrent convolutional network exploits the history of locations as well as the distinctive visual features learned by the deep neural networks. Inspired by recent bounding box regression methods for object detection, we study the regression capability of Long Short-Term Memory (LSTM) in the temporal domain, and propose to concatenate high-level visual features produced by convolutional networks with region information. In contrast to existing deep learning based trackers that use binary classification for region candidates, we use regression for direct prediction of the tracking locations both at the convolutional layer and at the recurrent unit. Our extensive experimental results and performance comparison with state-of-the-art tracking methods on challenging benchmark video tracking datasets shows that our tracker is more accurate and robust while maintaining low computational cost. For most test video sequences, our method achieves the best tracking performance, often outperforms the second best by a large margin.
1 Introduction
Visual tracking must handle substantial appearance and motion variation, while traditional handcrafted features provide limited semantic and discriminative information. The paper addresses this gap with a recurrent convolutional approach that combines spatial and temporal learning.
- Visual tracking is challenged by deformations, illumination changes, scale variation, abrupt motion, occlusion, blur, and background clutter.
- Handcrafted low-level features capture limited semantic information, are not robust to major appearance changes, and have limited discriminative power.
- Spatiotemporal analysis with deep neural networks for visual object tracking had not been adequately studied.
- The proposed recurrent convolutional model learns from both historical locations and robust visual features from past frames, unlike temporal methods using location history alone.
- The framework directly regresses coordinates or heatmaps instead of using local-region binary classifiers employed by related recurrent trackers.
- The modular network is trained end-to-end and is designed to extend ConvNet analysis into the spatiotemporal domain with low complexity.
2 System Overview
The system processes video frames with a deep network that combines YOLO visual features and preliminary locations with LSTM sequence modeling. Its probability model conditions each current location on the history of locations and input frames.
- The proposed network takes raw video frames as input and returns an object bounding-box coordinate for each frame.
- YOLO provides rich visual features and preliminary location inferences, while LSTM processes the resulting sequence information.
- At time t, B_t denotes the object location and X_t denotes the input frame; B_<t and X_≤t represent prior locations and frames through t.
- LSTM addresses conventional RNNs’ limited long-range context by using a memory cell and parameterized gates to accumulate and control state information.
- The system is motivated by regression-based detection and extends YOLO into the spatiotemporal domain with recurrent neural networks.
3 Our Proposed System
ROLO combines YOLO visual features and preliminary region proposals with LSTM sequence processing to regress object locations over time. Its design also supports direct coordinate or heatmap regression with spatial supervision.
- ROLO extends YOLO into the spatiotemporal domain by combining convolutional features and region information with recurrent processing.YOLO provides visual features and preliminary location inferences, while LSTM processes the sequence.
- Network Training of the Detection Module: The convolutional network is first pretrained for general feature learning, producing a 4096-dimensional visual feature vector.The feature vector is obtained from the first fully connected layer during CNN pretraining.
- Network Training of the Detection Module: YOLO predicts bounding-box locations, while class labels and confidence values are removed because tracking evaluation uses locations only.The framework sets S = 7, B = 2, and C = 20, then nullifies class label and confidence predictions.
- Network Training of the Tracking Module: The LSTM receives 4096-dimensional visual features, detection information, and the previous time-step state to train the tracking module.The tracking objective uses Mean Squared Error, with Adam used for stochastic optimization.
- Alternative Heatmap: An alternative heatmap representation converts predicted locations into a 32-by-32 spatial map concatenated with visual features before LSTM input.The heatmap provides confidence at multiple spatial locations and supports visualization of intermediate results.
- LSTM performs regression both within each concatenated feature unit and across sequences of frames to predict the next-frame feature vector.YOLO region proposals provide soft spatial supervision that helps connect visual features with locations.
4 Experimental Results
Experiments evaluate ROLO on challenging benchmark videos, unseen sequences, occlusions, robustness protocols, and step-size settings. Results indicate strong generalization and stable tracking when detections fail, while revealing sensitivity to training dynamics and sequence length.
- Benchmark evaluation: ROLO is compared with 10 distinct trackers on 30 challenging publicly available video sequences, using a harder OTB-30 subset.The comparison includes nine benchmark-leading trackers, CNN-SVM, and a modified SORT variant.
- Generalization: On unseen sequences, ROLO tracks objects outside the pretrained or training classes and remains stable when YOLO detection is degraded by motion blur.The reported behavior is attributed to interpreting high-level visual features and using spatio-temporal history.
- Occlusion robustness: ROLO continues tracking through occlusion, including frames where the detection module fails, and can follow the correct target during near-complete occlusion with similar distractors present.The heatmap model retains confidence in the true target by combining location history with visual features.
- Robustness and sensitivity: ROLO’s OPE, TRE, and SRE evaluations assess generalization and temporal and spatial robustness, while step-size experiments examine the accuracy–runtime trade-off.Figure 9 reports average IOU scores and fps under varied step sizes.
- Quantitative results: Training with auxiliary frames sharing similar dynamics improves ROLO performance, while adding more sequence frames can increase performance even without additional ground-truth boxes.The experiments also report that limited benchmark training data constrains learning of motion dynamics.
5 Conclusion and Future Work
The paper presents ROLO as a spatially and temporally deep tracker that addresses occlusion and motion blur while achieving accurate, robust, and computationally efficient tracking. Future work targets improved optimization, efficient online learning, and multi-target tracking.
- ROLO extends deep neural network learning and analysis into the spatiotemporal domain for visual object tracking.
- The tracker is designed to tackle major occlusion and severe motion blur while maintaining low computational cost.
- Future Work: Future research will study two stacked LSTMs to optimize heatmap and location cost functions separately for further performance improvement.
- Future Work: The authors also plan efficient online learning for unseen dynamics and data association techniques for multi-target tracking.