Source-linked AI summary
End-to-end Flow Correlation Tracking with Spatial-temporal Attention
Zheng Zhu, Wei Wu, Wei Zou, Junjie Yan
TL;DR
Existing DCF trackers largely emphasize current-frame appearance, leaving motion and inter-frame information underused in challenges such as occlusion and deformation. FlowTrack jointly learns flow and tracking, warps and aggregates historical features with current ones using spatial-temporal attention, and achieves favorable results across OTB2013, OTB2015, VOT2015, and VOT2016.
Problem
Most existing DCF trackers mainly use current-frame appearance and hardly benefit from motion or inter-frame information, limiting their handling of occlusion and deformation.
Method
FlowTrack jointly learns optical flow estimation, feature extraction, aggregation, and correlation-filter tracking while warping and adaptively aggregating historical features with current ones.
Results
FlowTrack achieves favorable accuracy and robustness against state-of-the-art methods on OTB2013, OTB2015, VOT2015, and VOT2016, including 1st EAO rankings on VOT2015 and VOT2016 at 12FPS.
Takeaways & Limitations
End-to-end use of consecutive-frame flow information is presented as a way to improve feature representation and tracking accuracy across challenging benchmarks.
Abstract
from arXiv · showhide
Discriminative correlation filters (DCF) with deep convolutional features have achieved favorable performance in recent tracking benchmarks. However, most of existing DCF trackers only consider appearance features of current frame, and hardly benefit from motion and inter-frame information. The lack of temporal information degrades the tracking performance during challenges such as partial occlusion and deformation. In this work, we focus on making use of the rich flow information in consecutive frames to improve the feature representation and the tracking accuracy. Firstly, individual components, including optical flow estimation, feature extraction, aggregation and correlation filter tracking are formulated as special layers in network. To the best of our knowledge, this is the first work to jointly train flow and tracking task in a deep learning framework. Then the historical feature maps at predefined intervals are warped and aggregated with current ones by the guiding of flow. For adaptive aggregation, we propose a novel spatial-temporal attention mechanism. Extensive experiments are performed on four challenging tracking datasets: OTB2013, OTB2015, VOT2015 and VOT2016, and the proposed method achieves superior results on these benchmarks.
1. Introduction
The paper addresses the limited use of motion and inter-frame information in DCF-based tracking by jointly learning flow and tracking, warping historical features, and adaptively aggregating them. FlowTrack performs favorably across four challenging tracking benchmarks.
- Motivation: Most existing DCF trackers rely mainly on current-frame appearance and do not fully exploit motion or inter-frame information during occlusion and deformation.Some flow-based trackers use off-the-shelf flow that is not trained end-to-end.
- Approach: FlowTrack formulates optical-flow estimation, feature extraction, aggregation, and correlation-filter tracking as network layers for end-to-end learning.The paper describes this as the first work, to the authors’ knowledge, to jointly train flow and tracking in a deep learning framework.
- Approach: Historical feature maps are warped to a specified frame using flow information and aggregated with current features for subsequent correlation-filter tracking.Warping addresses spatial misalignment caused by video motion before feature fusion.
- Results: 0.689 and 0.655 AUC were obtained on OTB2013 and OTB2015, respectively, while FlowTrack ranked 1st in EAO on both VOT2015 and VOT2016 at 12FPS.These evaluations cover four challenging object-tracking benchmarks.
- Approach: A spatial-temporal attention mechanism adaptively aggregates warped historical and current feature maps.The mechanism is intended to combine diverse information from different viewpoints, deformations, and illumination conditions.
2. Related works
Prior work combines DCFs with CNN features, builds Siamese or jointly trained tracking networks, and applies optical flow in related vision tasks. The paper distinguishes FlowTrack by aggregating flow and Siamese tracking end-to-end to use motion and inter-frame information.
- DCF-based trackers: DCF trackers have evolved through scale, color, long-term, and boundary-effect improvements, while retaining an emphasis on efficiency and extensibility.Examples include SAMF and fDSST for scale changes, Staple for color information, and SRDCF for boundary effects.
- CNN-based trackers: CNN-based DCF trackers replace handcrafted features with CNN features, but many components remain separately learned or use features pretrained for other tasks.The paper notes that this separation may produce suboptimal tracking results.
- CNN-based trackers: SiamFC, CCOT, ECO, CREST, and UCT model tracking with convolutional or Siamese architectures, but the cited trackers mainly use current-frame appearance features.FlowTrack instead aggregates flow and Siamese tracking in an end-to-end framework.
- Optical flow in visual recognition: Optical flow has been used to align heatmaps, predict future frames, model deformation, and accelerate or improve video recognition tasks.The related examples include pose estimation, segmentation, and video detection.
3. End-to-end flow correlation tracking
FlowTrack integrates optical flow, feature extraction, feature aggregation, and correlation-filter tracking into a differentiable Siamese network. Historical feature maps are flow-warped and adaptively combined with current features using spatial-temporal attention.
- End-to-end architecture: Historical frames are flow-warped to a specified frame, while current-frame features are extracted separately before both branches enter the correlation-filter layer.The historical branch uses FlowNet and FeatureNet; the current branch uses FeatureNet.
- End-to-end architecture: The framework formulates optical-flow estimation, feature extraction, aggregation, and correlation-filter tracking as jointly trainable network layers.This design unifies components that were previously learned separately within an end-to-end framework.
- Optical-flow aggregation: Warping aligns neighboring feature maps using flow correspondences and bilinear interpolation, enabling gradients to propagate through both features and flow.The flow field maps location p in one frame to p+δp in the specified frame.
- Optical-flow aggregation: Aggregated historical maps provide complementary viewpoint, deformation, and illumination information that can enhance the tracked object's appearance representation.The method addresses spatial misalignment across frames through flow-based warping before aggregation.
- Spatial-temporal attention: Spatial attention assigns location- and channel-dependent weights using cosine similarity between warped features and specified-frame features.A bottleneck sub-network first projects features into an embedding, and SoftMax normalizes weights across nearby frames.
- Spatial-temporal attention: Temporal attention re-calibrates aggregated temporal channels through global pooling and fully connected self-gating, counteracting spatial attention's preference for the specified frame.The mechanism learns channel importance from channel dependence and re-weights the original feature maps.
4. Experiments
The experiments evaluate FlowTrack on four challenging visual-tracking benchmarks using reported results for fair comparison.
- Datasets: Experiments use OTB2013, OTB2015, VOT2015, and VOT2016 as four challenging tracking datasets.OTB2013 and OTB2015 contain 50 and 100 videos, while VOT2015 and VOT2016 each contain 60 videos.
4.1. Implementation details
The implementation uses dedicated feature, flow, embedding, and temporal-attention subnetworks, with five-frame aggregation and a reported tracking speed of 12 FPS.
- Network configuration: FeatureNet uses three convolution layers, while FlowNet follows the implementation in.The FeatureNet layers are 3 × 3 × 128, 3 × 3 × 128, and 3 × 3 × 96.
- Network configuration: The spatial-attention embedding subnetwork contains three convolution layers, and temporal attention uses three fully connected layers with ReLU and Sigmoid activations.The temporal-attention layers are configured as 1 × 1 × 128, 1 × 1 × 128, and 1 × 1 × 6.
- Training and runtime: Aggregation uses five frames, and the tracker operates at 12 FPS on an Nvidia GTX TITAN X GPU.Training data comes from the VID training and validation sets.
4.2. Results on OTB
On OTB2013 and OTB2015, FlowTrack outperforms state-of-the-art trackers in precision and success measures, including challenging attributes.
- Datasets and metrics: OTB2013 contains 50 sequences, while OTB2015 extends it to 100 sequences and evaluates trackers with precision and success plots.The representative precision is measured at 20 pixels, and success plots are ranked by area under the curve (AUC).
- Interpretation: The authors attribute the results to flow-based feature enhancement, end-to-end component coupling, and efficient updating and scale handling.Unlike the proposed framework, SINT+ uses off-the-shelf optical flow without end-to-end training.
- OTB2015: On OTB2015, FlowTrack achieves a success score of 0.655, exceeding MCPF at 0.628 and CREST at 0.623.Attribute analyses report stronger handling of occlusion, illumination variation, and background clutter than other compared trackers.
4.3. Results on VOT
FlowTrack ranks first on VOT2015 and VOT2016 under EAO, with additional accuracy, robustness, and speed results reported for VOT2016.
- VOT2015: VOT2015 evaluates 60 challenging videos using expected average overlap (EAO), accuracy, and robustness.EAO measures expected no-reset overlap over a short-term sequence-length distribution.
- VOT2015: FlowTrack ranks 1st among 61 VOT2015 trackers according to EAO and also ranks 1st in accuracy and failures.The authors attribute this performance to associating flow information with end-to-end training.
- VOT2016: FlowTrack ranks 1st among 70 VOT2016 trackers according to EAO.VOT2016 uses the same datasets as VOT2015 but with re-annotated ground truth.
- VOT2016: At 12 FPS, FlowTrack runs 40 times faster than CCOT at 0.3 FPS on VOT2016.Its accuracy ranks 1st and robustness ranks 2nd among representative trackers.
4.4. Ablation analyses
Ablation results show that flow information, end-to-end training, flow aggregation, and temporal attention each contribute to FlowTrack’s performance.
- Experimental design: The ablation compares a no-flow baseline, fixed-flow training, time-decay fusion, and a model without temporal attention.These variations isolate the proposed flow and attention components.
- Overall ablation findings: Every ablated variation performs worse than the full FlowTr algorithm, indicating that each tracking component contributes to performance.The full model is denoted FlowTr in the ablation analysis.
- Flow information: Adding and assembling flow information improves all evaluation criteria by more than 6% relative to the no-flow model.This comparison is reported between the no-flow baseline and FlowTr.
- End-to-end training: Fixing FlowNet can reduce VOT performance compared with no flow, supporting the necessity of end-to-end training.The reported comparison is among no flow, fixed flow, and FlowTr.
- Aggregation and attention: The proposed flow aggregation increases EAO in VOT2015 and VOT2016 by nearly 8% over time-decay fusion, while temporal attention further improves tracking.The comparison is between decay and FlowTr.
4.5. Qualitative Results
Qualitative examples show FlowTrack retaining targets under deformation, illumination and pose variation, scale change, and background clutter where comparison trackers fail or drift.
- Qualitative comparisons: FlowTrack successfully tracks targets through severe deformation, illumination and pose variation, scale changes, and background clutter.The examples are drawn from the singer2, skating1, carscale, and bolt2 sequences.
- Qualitative comparisons: Compared trackers lose targets, fail to fit scale changes, drift to background, or drift to distracters in the highlighted sequences.FlowTrack is reported as successful in each corresponding challenging case.
5. Conclusions
The work proposes an end-to-end tracking framework that uses flow information from consecutive frames and validates it on OTB and VOT datasets.
- FlowTrack uses rich flow information from consecutive frames within an end-to-end tracking framework.
- Frames sampled at certain intervals are warped using flow information and aggregated before correlation filter tracking.
- A spatial-temporal attention mechanism adaptively aggregates the warped frame features.
- The approach is evaluated on OTB and VOT datasets.