Learn to Match: Automatic Matching Network Design for Visual Tracking

Zhipeng Zhang; Yihao Liu; Xiao Wang; Bing Li; Weiming Hu

Source-linked AI summary

Learn to Match: Automatic Matching Network Design for Visual Tracking

Zhipeng Zhang, Yihao Liu, Xiao Wang, Bing Li, Weiming Hu

arXiv:2108.00803v1Aug 2, 2021cs.CV

TL;DR

Siamese tracking relies on heuristic matching-network designs that require expert effort and may lack robustness across challenging environments. This paper introduces six feature-fusion operators and Binary Channel Manipulation to search and combine matching networks, and reports gains over the Ocean baseline across five benchmarks. The authors conclude that cross-correlation is not always optimal and that the searched networks are robust on short- and long-term benchmarks.

Problem
Heuristic matching-network design requires substantial human effort and is difficult to guarantee robust tracking across all challenging environments.
Method
The paper introduces six matching operators and uses Binary Channel Manipulation to automatically select and combine them for classification and regression branches.
Results
The proposed framework surpasses the Ocean baseline on OTB100, LaSOT, GOT10K, TrackingNet, and TNL2K.
Takeaways & Limitations
The analyses indicate that depthwise cross-correlation is not requisite and that appropriate matching operators can provide performance gains and robustness across short- and long-term benchmarks.
Takeaways & Limitations
The bilevel-optimization derivation is beyond the scope of the work and relies on an approximation described in prior work.

Abstract

from arXiv · show

Siamese tracking has achieved groundbreaking performance in recent years, where the essence is the efficient matching operator cross-correlation and its variants. Besides the remarkable success, it is important to note that the heuristic matching network design relies heavily on expert experience. Moreover, we experimentally find that one sole matching operator is difficult to guarantee stable tracking in all challenging environments. Thus, in this work, we introduce six novel matching operators from the perspective of feature fusion instead of explicit similarity learning, namely Concatenation, Pointwise-Addition, Pairwise-Relation, FiLM, Simple-Transformer and Transductive-Guidance, to explore more feasibility on matching operator selection. The analyses reveal these operators' selective adaptability on different environment degradation types, which inspires us to combine them to explore complementary features. To this end, we propose binary channel manipulation (BCM) to search for the optimal combination of these operators. BCM determines to retrain or discard one operator by learning its contribution to other tracking steps. By inserting the learned matching networks to a strong baseline tracker Ocean, our model achieves favorable gains by $67.2 \rightarrow 71.4$, $52.6 \rightarrow 58.3$, $70.3 \rightarrow 76.0$ success on OTB100, LaSOT, and TrackingNet, respectively. Notably, Our tracker, dubbed AutoMatch, uses less than half of training data/time than the baseline tracker, and runs at 50 FPS using PyTorch. Code and model will be released at https://github.com/JudasDie/SOTS.

1. Introduction

Siamese tracking has advanced efficient state estimation, but its heuristic matching-network design remains labor-intensive and difficult to make robust across challenging environments. The paper introduces novel matching operators and Binary Channel Manipulation to automatically search and combine them, improving a strong baseline across benchmarks.

Generic object tracking estimates an arbitrary object's location and scale in a video sequence.
Heuristic matching-network design requires substantial expert effort and cannot readily guarantee robustness across challenging environments.The paper motivates automatic search as an alternative to manually selecting operators for different circumstances.
Six novel operators—Concatenation, Pointwise-Addition, Pairwise-Relation, FiLM, Simple-Transformer, and Transductive-Guidance—expand matching-operator selection beyond cross-correlation.Their performances are compared under different environmental degradation types, revealing distinct resilience patterns.
Replacing Ocean's cross-correlation with concatenation yields a 1.2-point OTB100 success-score gain.
Binary Channel Manipulation automatically selects and combines matching operators by learning binary channel-level contributions to subsequent tracking steps.The search space contains seven operators, and Gumbel-Softmax discretizes learnable manipulators while preserving differentiable training.
The searched matching networks surpass the Ocean baseline on OTB100, LaSOT, GOT10K, TrackingNet, and TNL2K.The tracker also outperforms DiMP and KYS on all reported criteria of the evaluated datasets.

2. Related Work

Related work frames Siamese tracking as efficient matching between exemplar and candidate images, with cross-correlation and feature-based alternatives supporting localization and state estimation.

Matching-based tracking predicts foreground probability as a one-shot matching problem between exemplar and candidate images.Prior methods include learned matching functions, dot products, concatenated features, and target-information injection.
Figure 2 presents four matching operators: Concatenation, Pointwise-Addition, Pairwise-Relation, and FiLM.
The related-work context contrasts conventional similarity matching with feature-fusion operators explored in this paper.
SiamFC introduced cross-correlation as a similarity metric, while later Siamese trackers extended the framework for scale estimation and deeper backbones.

3. Analysis of Matching Operators

The paper instantiates seven matching operators for Siamese tracking and analyzes how their feature fusion choices respond to different tracking conditions. Results show operator-specific strengths, motivating automatic combination rather than selecting one universally best operator.

Operator instantiations: Seven operators are explored: depthwise cross-correlation plus Concatenation, Pointwise-Addition, Pairwise-Relation, FiLM, Simple-Transformer, and Transductive-Guidance.These operators inject exemplar information into candidate features through distinct fusion or relation-learning mechanisms.
Operator instantiations: FiLM generates candidate features through an exemplar-conditioned affine transformation, R = γF_x + β.The coefficient γ and bias β are produced from the exemplar feature and have size 1 × 1 × C.
Operator instantiations: Simple-Transformer uses candidate features as queries and exemplar features as keys and values in multi-head attention.Transductive-Guidance instead predicts exemplar–candidate affinity and propagates the first-frame pseudo mask to provide spatial guidance for localization.
Individual-operator analysis: On OTB100, vanilla Ocean with depthwise cross-correlation achieves 67.2 overall success, while Simple-Transformer and Transductive-Guidance score 65.8 and 65.0.The other four operators are comparable to or better than depthwise cross-correlation, showing that replacing it can help or hurt depending on the operator.
Complementarity analysis: Operator strengths vary by degradation: Concatenation leads on most challenging factors but trails other operators on SV, OPR, OV, and LR.Activation maps also separate operators that focus on the target from those that exploit more context information.
Complementarity analysis: Because operators show different resilience across challenging factors and image contents, AutoMatch learns to select and combine them instead of manually searching for one operator.The framework uses binary channel manipulation to adaptively choose matching functions.

4. Methodology

AutoMatch replaces heuristic matching-network selection with a searchable space of matching operators and uses binary channel manipulation to retain task-dependent combinations. The selected response maps are concatenated and passed to classification and regression networks for target-state prediction.

Framework: AutoMatch extracts exemplar and candidate features, applies a matching network, and feeds the resulting representations to classification and regression branches for target-state prediction.The matching network is the searched component within the standard feature extraction, matching, and localization pipeline.
Search Space: The search space contains optional matching operators applied to exemplar and candidate features, producing a response set for subsequent selection.The response set consists of operator-generated response features.
Binary Channel Manipulation: Binary channel manipulation assigns each response channel a learnable manipulator whose value indicates that channel’s contribution to later target-state estimation.The weighted response maps are aggregated by concatenation into a feature used by the subsequent target-estimation network.
Binary Decision: Gumbel-Softmax converts the manipulators’ binary decisions into differentiable training variables, using soft values for gradients and hard values during the forward pass.The binary formulation uses two class probabilities and a temperature-controlled softmax approximation.
Bilevel Optimization: BCM jointly learns channel manipulators and other network weights through bilevel optimization of validation loss after training parameters on training loss.The method uses a one-step approximation to speed bilevel optimization, with ϵ as the inner-optimization learning rate.
Final Selection: The classification and regression branches are searched separately for task-dependent matching networks, and the two operators with maximum potential are retained after training.The retained operators’ response maps are concatenated before the following tracking steps.

5. Experiments

Experiments apply the searched matching networks to Ocean across five benchmarks, showing consistent gains and efficient operation. Ablations indicate that BCM's channel-level search outperforms scalar manipulation and NAS-like cell search.

Implementation Details: The searched matching networks are applied to Ocean's classification and regression branches after a two-stage search-and-retraining procedure.The baseline uses a modified ResNet50, and the search selects target-dependent networks for both state-estimation branches.
State-of-the-art Comparison: 71.4 AUC on OTB100 exceeds Ocean's 67.2 by 4.2 points, while the tracker runs at 50 FPS versus Ocean's 58 FPS.The result also exceeds SiamAttn's previous best AUC of 71.2.
State-of-the-art Comparison: 58.3 success on LaSOT improves over Ocean by 5.7 points, with 7.3 points higher precision and 50 FPS versus DiMP's 43 FPS.LaSOT evaluates long-term tracking on 280 testing videos.
State-of-the-art Comparison: 76.0 success on TrackingNet surpasses Ocean by 5.7 points and MAMLTrack by 0.3 points.On GOT10K, gains over Ocean are 6 points in AO, 7.1 points in SR0.5, and 7.8 points in SR0.75.
Ablation and Analysis: Channel-level BCM search achieves higher success than scalar operator manipulation, while the NAS-like cell reaches 55.7 success on LaSOT at 35 FPS.Scalar manipulation achieves 69.5 on OTB100 and 54.7 on LaSOT; the NAS-like alternative is inferior in both performance and speed.

6. Conclusion

The paper introduces six matching operations and uses BCM to search matching networks for Siamese tracking's classification and regression branches. Experiments report robustness across short- and long-term benchmarks, while identifying cross-correlation as non-optimal in this setting.

6. Conclusion: Six novel operations expand matching-operator selection beyond classical depthwise cross-correlation in Siamese tracking.The operations are evaluated for feature matching and selected for complementary behavior.
6. Conclusion: BCM simultaneously searches matching networks for the classification and regression branches and applies them to a baseline tracker.The method uses binary channel manipulation to select and combine operators.
6. Conclusion: Experiments show robustness on both short-term and long-term tracking benchmarks.The conclusion reports this as the outcome of applying the learned matching networks to the baseline tracker.