PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Spatial-Aware Regressions for Visual Tracking

Chong Sun, Dong Wang, Huchuan Lu, Ming-Hsuan Yang

arXiv:1706.07457v2cs.CV

TL;DR

Visual tracking needs models that use target spatial structure without the limitations of conventional CNN and KRR trackers. LSART combines a cross-patch KRR model with a spatially regularized CNN, and reports promising performance, especially on VOT-2017.

  • Problem

    CNN trackers can overfit, while KRR trackers may be insufficiently discriminative and existing kernels overlook target spatial layouts.

  • Method

    LSART combines cross-patch KRR with learnable similarity weights and a spatially regularized CNN whose channels focus on localized regions, then combines their heat maps.

  • Results

    LSART achieves promising tracking performance, especially on the VOT-2017 public dataset.

  • Takeaways & Limitations

    The complementary spatial-aware regressions provide a unified tracker that uses holistic object information and localized target regions.

  • Takeaways & Limitations

    The conventional alternating solver for KRRCPS is computationally expensive, with O(N^3) and O(dN^2) update complexities.

Abstract

from arXiv · show

In this paper, we analyze the spatial information of deep features, and propose two complementary regressions for robust visual tracking. First, we propose a kernelized ridge regression model wherein the kernel value is defined as the weighted sum of similarity scores of all pairs of patches between two samples. We show that this model can be formulated as a neural network and thus can be efficiently solved. Second, we propose a fully convolutional neural network with spatially regularized kernels, through which the filter kernel corresponding to each output channel is forced to focus on a specific region of the target. Distance transform pooling is further exploited to determine the effectiveness of each output channel of the convolution layer. The outputs from the kernelized ridge regression model and the fully convolutional neural network are combined to obtain the ultimate response. Experimental results on two benchmark datasets validate the effectiveness of the proposed method.

1. Introduction

LSART addresses complementary weaknesses in CNN- and KRR-based trackers by combining spatial-aware regressions: KRR models holistic structure, while CNN filters focus on localized regions. The resulting tracker is reported to achieve promising performance, especially on VOT-2017.

  • Motivation: CNN trackers can overfit because their many convolutional parameters are difficult to fine-tune, whereas KRR trackers may lack discriminative power with correlated samples.Existing KRR methods also treat target parts equally and ignore relationships among parts.
  • Method: The KRR and CNN heat maps are combined to generate the final heat map for target localization.Experiments on popular datasets are reported to outperform other state-of-the-art methods.
  • Method: The KRR component uses cross-patch similarities with learnable weights and is reformulated as a neural network for efficient optimization.The learned weights adaptively emphasize reliable regions, while the model jointly optimizes regression coefficients and similarity weights.
  • Method: The CNN component imposes spatially regularized filter kernels so each output channel focuses on a localized target region.A two-stream training network is designed to avoid overfitting and consider rotation information.
  • Method: LSART combines spatial-aware KRR and CNN models because their complementary spatial focuses address their inherent limitations.The KRR model focuses on the holistic object, while the CNN model focuses on small, localized regions.
  • Results: LSART achieves very promising tracking performance, especially on the VOT-2017 public dataset.Figure 1 describes comparable results with state-of-the-art algorithms on OTB.

2. Related Work

Visual tracking research emphasizes discriminative appearance models, including correlation filters, KRR, and CNN trackers. These approaches offer efficiency or strong performance but may overlook target spatial layouts.

  • Appearance Models: Visual tracking algorithms mainly design robust appearance models, broadly categorized as generative or discriminative.Recent work increasingly prefers discriminative appearance models alongside deep neural networks and correlation filters.
  • Correlation Filters: Correlation filters are attractive for efficiency and robustness, and can be viewed as KRR models accelerated in the frequency domain.Kernelized correlation filters such as CSK extend correlation-filter approaches with kernel functions.
  • CNN Trackers: CNN-based trackers have achieved good recent performance using architectures that learn representations and classifiers or transfer pretrained models.Examples include shallow convolutional networks, transferred fully convolutional networks, Siamese networks, and multi-domain CNNs.
  • CNN Trackers: CNN-based trackers often exploit global target information and therefore ignore spatial layouts.This limitation motivates using spatially regularized convolutional filters.

3. Spatial-Aware KRR

The spatial-aware KRR model uses cross-patch similarities and learnable weights, then reformulates its optimization as a neural network. This network produces localization heat maps through cropped feature regions and convolutional operations.

  • KRR Formulation: Conventional ridge regression represents the model with a linear weight vector, regularization, and sample labels, while KRR uses kernel values between feature samples.Existing kernels do not fully account for target spatial layouts.
  • Cross-Patch Similarity: The proposed kernel divides each sample into M patches and computes similarity across all patch pairs between two samples.Each patch-pair similarity receives a learnable weight.
  • Cross-Patch Similarity: Learnable similarity weights let KRRCPS emphasize reliable regions, while considering more patch pairs enhances discriminative ability.The model can represent both regression coefficients and patch reliability.
  • Optimization: Alternating optimization of αt and βt is computationally expensive, with complexity O(N^3) for αt and O(dN^2) for βt.The conventional solver is therefore unsuitable for efficient online updates.
  • Network Structure: The equivalent network crops target-centered feature regions, applies convolutional operations to patch-specific maps, and outputs a heat map for localization.Its modules correspond to the operations of the reformulated regression model.
  • Network Reformulation: Reformulating KRRCPS as a neural network enables backpropagation with forward and backward complexity O(dN).The reformulation also supports integration with deep feature extraction networks.

4. Spatial-Aware CNN

The spatial-aware CNN constrains convolution filters to localized regions, uses distance transform pooling to weight response maps, and applies two-stream training for rotation robustness and reduced overfitting.

  • CNNSRK Architecture: Each output channel produces a target-localization heat map, whose grouped responses are combined by distance transform pooling into a final heat map.The network reshapes inputs to 46 × 46 × C, uses 5 × 5 and 3 × 3 convolutions, and groups C1/4 response maps.
  • Spatially Regularized Kernels: The CNN uses spatially regularized filter kernels so different kernels focus on localized regions of the target.Unlike constraints on the input feature map, the method constrains filter kernels and fixes the filter mask during tracking.
  • Distance Transform Pooling: Distance transform pooling estimates response-map reliability, with larger learned ϖ indicating greater reliability and ϖ = θ = 0 eliminating input influence.The layer generalizes max pooling and combines response maps according to learned distance-transform parameters.
  • Two-Stream Training Strategy: The two-stream network shares convolution weights, combines original and rotated-target responses through max-out pooling, and learns rotation-aware parameters in stages.The upper branch uses the original feature map, while the lower branch uses the feature map corresponding to the rotated target.
  • Two-Stream Training Strategy: Severe in-plane rotation is defined as a rotation angle larger than 90 degrees, where two-stream training improves tracking results.The reported visual results compare tracking with and without the two-stream training process.
  • Two-Stream Training Strategy: Two-stream training also addresses overfitting by decomposing a network with 1,281,000 parameters when C = 512 and C1 = 100.The authors motivate staged learning because the original network is difficult to train with limited visual-tracking data.

5. Tracking with Spatial-Aware KRR and CNN

The tracker combines spatial-aware KRR and CNN responses for target localization, then estimates scale and refines the bounding box. Both models are finetuned against an ideal Gaussian heat map.

  • Response Combination: The tracking framework combines spatial-aware KRR and CNN models to exploit holistic and localized target information.The KRR and CNN heat maps are combined using a trade-off parameter γ, and the highest final heat-map score gives the target location.
  • Scale Estimation: After location estimation, the tracker evaluates candidate scales and selects the scale with the largest score.Each candidate feature map is cropped or padded, reshaped to H × W, and passed through a fully connected scale-estimation layer.
  • Scale Estimation: Bounding box regression further refines the tracking result after the optimal location and scale are obtained.The refinement follows scale estimation.
  • Online Finetuning: The KRR and CNN models are finetuned with stochastic gradient descent using an L2 loss to fit an ideal Gaussian heat map.The CNN uses the two-stream network during parameter updates, while the KRR updates its model parameters and auxiliary quantities.
  • Scale Estimation: When scale variation is detected, the scale-estimation network is updated with a Gaussian-target squared-error loss.The loss compares predicted scale scores fS(T(Xt, sl)) with Gaussian targets ysl.

6. Experimental Results

Experiments on OTB-2015 and VOT-2017 evaluate LSART against state-of-the-art trackers and separately assess its spatial-aware components. LSART achieves strong benchmark performance, while ablations show gains from combining cross-patch similarity and spatially regularized kernels.

  • OTB-2015 Dataset: LSART achieves 92.3% precision and 67.2% AUC on OTB-2015, ranking first in precision and second in AUC.The tracker has comparable overall performance to ECO.
  • OTB-2015 Dataset: On OTB-2015 attributes, LSART improves the second-best tracker by 4.3% for deformation and ECO by 1.8% for in-plane rotation.The authors attribute these gains to adaptive reliability estimation and a two-stream network learning rotation information.
  • VOT-2017 Public Dataset: On VOT-2017, LSART ranks first in EAO, R, and AO while maintaining competitive accuracy.It gains 12.94% relative EAO over CFCF and 15.36% over ECO.
  • VOT-2017 Public Dataset: LSART achieves comparable AO performance to MCPF and improves ECO by a relative gain of 8.71% under the OPE rule.The comparison uses AO values reported in the last column of Table 1.
  • Ablation Studies: The direct KRR-CNN baseline scores 0.841 in precision and 0.606 in AUC, whereas LSART improves it by 9.75% and 10.89%, respectively.The ablation compares Baseline, Baseline+CPS, Baseline+SRK, and Baseline+CPS+SRK variants on OTB-2015.
  • Ablation Studies: The CPS module contributes relative gains of 5.35% in precision and 6.93% in success plots over the baseline.The SRK module is evaluated separately by comparing Baseline+SRK with Baseline.

7. Conclusion

LSART combines spatial-aware KRR and CNN regressions into a robust online tracker, with experiments on two benchmarks showing promising tracking performance, especially on VOT-2017.

  • LSART combines spatial-aware KRR and spatial-aware CNN models to exploit their complementary properties for visual tracking.The KRR model uses cross-patch similarity, while the CNN focuses on localized target regions through spatially regularized filter kernels.
  • The cross-patch similarity KRR considers target interior structure and adaptively determines the importance of similarity scores between patches.Its neural-network reformulation enables more efficient optimization.
  • The spatially regularized CNN focuses on localized regions, while distance transform pooling determines convolutional-layer reliability.
  • The two regressions are combined to generate a final heat map for target localization.
  • Experimental results on two recent benchmarks demonstrate promising tracking performance, especially on the VOT-2017 public dataset.
Loading 1706.07457v2…