PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning the Model Update for Siamese Trackers

Lichao Zhang, Abel Gonzalez-Garcia, Joost van de Weijer, Martin Danelljan, Fahad Shahbaz Khan

arXiv:1908.00855v2cs.CV

TL;DR

Siamese trackers rely on a simple linear template update that cannot adapt fully to changing appearances. UpdateNet learns this update from initial, accumulated, and current templates, improving tracking across four benchmarks over standard updating.

  • Problem

    Siamese trackers’ linear update rule uses a fixed rate and simple template combination, limiting adaptation to changing appearance, occlusion, and drift.

  • Method

    UpdateNet is a convolutional neural network that learns updated templates from the initial, accumulated, and current-frame templates for integration into Siamese trackers.

  • Results

    Experiments on VOT2016, VOT2018, LaSOT, and TrackingNet show significantly improved tracking performance over standard linear updating or no update.

  • Takeaways & Limitations

    The learned update strategy provides an efficient, general approach that can improve existing Siamese trackers across multiple tracking benchmarks.

  • Takeaways & Limitations

    UpdateNet receives ground-truth information only from the object location in the initial frame; subsequent inputs rely on predicted locations.

Abstract

from arXiv · show

Siamese approaches address the visual tracking problem by extracting an appearance template from the current frame, which is used to localize the target in the next frame. In general, this template is linearly combined with the accumulated template from the previous frame, resulting in an exponential decay of information over time. While such an approach to updating has led to improved results, its simplicity limits the potential gain likely to be obtained by learning to update. Therefore, we propose to replace the handcrafted update function with a method which learns to update. We use a convolutional neural network, called UpdateNet, which given the initial template, the accumulated template and the template of the current frame aims to estimate the optimal template for the next frame. The UpdateNet is compact and can easily be integrated into existing Siamese trackers. We demonstrate the generality of the proposed approach by applying it to two Siamese trackers, SiamFC and DaSiamRPN. Extensive experiments on VOT2016, VOT2018, LaSOT, and TrackingNet datasets demonstrate that our UpdateNet effectively predicts the new target template, outperforming the standard linear update. On the large-scale TrackingNet dataset, our UpdateNet improves the results of DaSiamRPN with an absolute gain of 3.9% in terms of success score.

1. Introduction

Siamese trackers are efficient but struggle with appearance changes because fixed or linearly updated templates cannot adapt to varied tracking conditions. UpdateNet learns an adaptive update from initial, accumulated, and current-frame templates, improving SiamFC and DaSiamRPN performance on multiple benchmarks.

  • Tracking challenges: Visual tracking must localize a target throughout a video despite appearance changes, occlusions, distractors, motion blur, and illumination changes.These conditions make generic visual object tracking highly challenging.
  • Siamese tracking: Siamese trackers match an object template with a search-region feature representation while providing competitive accuracy and computational efficiency.The paper focuses on Siamese tracking methods rather than tracking-by-detection methods.
  • Template updating: Fixed templates can cause early tracker failure when target appearance changes substantially, motivating template adaptation.The original Siamese tracker initializes its template in the first frame and keeps it fixed thereafter.
  • Template updating: Linear running-average updates assume a constant appearance-change rate, although update requirements vary considerably across tracking situations.Recent Siamese trackers use a fixed learning rate, but the paper notes that different situations require different update behavior.
  • Proposed approach and evaluation: UpdateNet learns an adaptive template update from the initial ground-truth template, accumulated template, and current-frame feature template at the predicted object location.It is evaluated with SiamFC and DaSiamRPN on benchmarks including VOT2018 and LaSOT, where enhanced updating improves tracking performance.

2. Related work

Related work centers on two tracking frameworks: tracking-by-detection, which learns discriminative classifiers online, and template matching, commonly implemented with Siamese networks. Existing template-update methods use linear averaging, stored-template selection, recurrent memory, or transformations based only on the initial template, each with stated efficiency or adaptation limitations.

  • Tracking Frameworks: Tracking-by-detection formulates target localization as classification and learns a discriminative classifier online from target and background image patches.
  • Tracking Frameworks: Template-matching trackers commonly use Siamese networks with spatial cross-correlation and two streams for template and search-region features.
  • Template Updating: Linear interpolation remains a limitation: even with more samples, correlation filters are updated by averaging the filters corresponding to those samples.A conservative update strategy applied only every five frames can increase tracking efficiency.
  • Template Updating: Prior adaptive methods trade off capability and complexity: LSTM memory is computationally expensive, reinforcement learning selects one stored template without accumulating multiple frames, and initial-template transformations ignore historical variations.

3. Updating the object template

This section replaces fixed-rate template averaging with UpdateNet, a learned adaptive updater for Siamese trackers. UpdateNet uses initial, accumulated, and current templates to predict a template suited to the next frame.

  • 3.1. Standard update: Standard updating uses a running average with exponentially decaying weights and typically fixes the update rate at a small value such as γ = 0.01.The template sample comes from the current frame, while the accumulated template summarizes prior frames.
  • 3.1. Standard update: Fixed averaging cannot adapt its rate across videos or time, and it updates every template channel and spatial dimension equally.These limitations are problematic when updating needs vary, including during camera motion or partial occlusion.
  • 3.2. Learning to update: UpdateNet learns a generic function φ that updates the accumulated template using the initial ground-truth, previous accumulated, and current-frame templates.The function can respond to differences between current and accumulated templates while retaining the initial template as a reliable reference.
  • 3.3. Tracking framework with UpdateNet: During tracking, the accumulated template predicts the current target location, whose extracted features are concatenated with the initial template and passed through convolutional layers.The network outputs the new accumulated template; on the first frame, current and previous accumulated templates equal the initial template.
  • 3.3. Tracking framework with UpdateNet: Only the initial-frame object location is ground-truth input during tracking, so UpdateNet uses residual learning anchored by the initial template.A skip connection adds the initial ground-truth template to the network output while historical appearances still inform the update.
  • 3.4. Training UpdateNet: UpdateNet is trained to make its predicted template match the ground-truth template extracted from the next frame by minimizing their Euclidean distance.The next-frame ground-truth template is treated as the optimal template for searching in that frame.
  • 3.4. Training UpdateNet: Training uses imperfect current-frame localizations rather than ground-truth locations, then progressively regenerates accumulated templates and predictions with the model from the previous stage.This avoids biasing UpdateNet toward unrealistically small changes and forms inputs closer to online tracking conditions.

4. Experiments … 4.4. Ablation study

The experiments train UpdateNet on LaSOT, evaluate it across standard tracking benchmarks, and integrate it into SiamFC and DaSiamRPN without changing their other components. Ablations show that residual, multi-stage learning with an initial-template skip connection improves SiamFC performance over no updating.

  • 4.2. Evaluation datasets and protocols: Evaluation covers VOT2018/16, LaSOT, and TrackingNet using their standard tracking benchmarks and protocols.VOT uses EAO, LaSOT reports precision and success plots, and TrackingNet uses precision, normalized precision, and success.
  • 4.2. Evaluation datasets and protocols: VOT2018 contains 60 public testing sequences and 21,356 frames, with failures re-initialized five frames later under the protocol.Expected Average Overlap (EAO) combines accuracy and robustness for ranking.
  • 4.2. Evaluation datasets and protocols: TrackingNet comprises 30,643 videos, including 30,132 training and 511 testing videos, and reports precision, normalized precision, and success.The videos average 470,9 frames.
  • 4.3. Implementation details: SiamFC and DaSiamRPN are used as base trackers with a modified AlexNet backbone, changing only the updating component.SiamFC originally performs no object update, while DaSiamRPN uses its original no-update version; SiamFC receives γ = 0.0102 from CFNet for linear updating during training.
  • 4.3. Implementation details: UpdateNet is a compact two-layer convolutional network with 1 × 1 convolutions, ReLU, and tracker-specific channel dimensions.Its template dimensions use C = 256 for SiamFC and C = 512 for DaSiamRPN.
  • 4.4. Ablation study: 7.4% total improvement is achieved over original SiamFC with no update when UpdateNet uses residual training, multi-stage training, and a skip connection from T GT_0.The first UpdateNet stage is beneficial because the learned update is grounded on a reliable object sample; the paper uses three training stages thereafter.

4.5. Analysis on representation update · 4.6. Generality and tracking speed

UpdateNet improves template updating by adapting more closely to ground-truth feature changes than linear updating. Across SiamFC and DaSiamRPN, it improves tracking performance with only a very small temporal overhead.

  • 4.5. Analysis on representation update: Accumulated-template visualizations compare ground-truth, no-update, linear-update, and UpdateNet strategies for SiamFC.Figure 3 shows search regions, ground-truth boxes, response maps, and the top four template channels.
  • 4.5. Analysis on representation update: Update strategies can interfere with the learned feature correlations that Siamese trackers use for localization.The analysis examines whether template updating damages the correlation abilities of learned features.
  • 4.5. Analysis on representation update: The linear strategy cannot match the updating rate required by ground-truth feature changes, whereas UpdateNet is substantially more adaptive.Change rate is quantified between templates from contiguous frames and averaged across all 60 VOT2018 videos.
  • 4.6. Generality and tracking speed: UpdateNet generalizes to SiamFC and DaSiamRPN, improving their VOT2018 tracking performance with a very small temporal overhead.Figure 5 evaluates EAO against tracking speed, measured in frames per second on a logarithmic axis.
  • 4.6. Generality and tracking speed: UpdateNet substantially improves tracking performance without significantly affecting speed when combined with two Siamese trackers.The comparison covers state-of-the-art methods using EAO versus speed on VOT2018.
  • 4.6. Generality and tracking speed: On VOT2018, the proposed approach achieves superior EAO performance compared with the state-of-the-art methods.Figure 6 presents the EAO comparison for the proposed method and competing trackers.

4.7. Fine-tuning the linear update rate … 4.10. TrackingNet dataset

Experiments show that a fixed linear update rate is unsuitable across videos, whereas UpdateNet provides a more effective, adaptive update across Siamese architectures and datasets. It improves DaSiamRPN on LaSOT and TrackingNet, including gains of 3.4%, 1.9%, and 3.9% across TrackingNet’s three metrics.

  • 4.7. Fine-tuning the linear update rate: Linear-update performance continuously and rapidly decreases as the update rate increases, indicating that no fixed general rate suits all videos.Figure 7 reports EAO scores as a function of update rate for DaSiamRPN and SiamFC.
  • 4.8. Comparison with other updating strategies: UpdateNet achieves a more effective update and higher tracking performance than DSiam and MemTrack while remaining applicable to different Siamese architectures.The comparison is conducted on VOT2016; DSiam and MemTrack use SiamFC as their base tracker.
  • 4.9. LaSOT dataset: On LaSOT, UpdateNet enhances DaSiamRPN’s updating capabilities and significantly boosts performance on all measures.The evaluation follows the official protocol on long-term testing sequences, where appearance variations and template departures are common.
  • 4.9. LaSOT dataset: The learned-update tracker surpasses all state-of-the-art trackers on LaSOT, supporting adaptive updating for accurate object localization.The comparison includes the top-10 trackers reported for the LaSOT testing set.
  • 4.10. TrackingNet dataset: 3.4%, 1.9%, and 3.9% are UpdateNet+DaSiamRPN’s absolute gains over DaSiamRPN in TrackingNet precision, normalized precision, and success, respectively.UpdateNet significantly improves all three TrackingNet metrics.
  • 4.10. TrackingNet dataset: TrackingNet results show that learning the model update benefits Siamese trackers across several datasets and evaluation measures.The reported evaluation uses TrackingNet’s three testing-set metrics.

5. Conclusions

The paper replaces Siamese trackers’ simple linear template update with a learned optimization-based update using UpdateNet, which can integrate into existing trackers.

  • 5. Conclusions: UpdateNet learns to update the accumulated appearance template using the first-frame template, current-frame template, and previous accumulated template.It replaces the usual linear update rule by learning the updating step as an optimization problem.

Supplementary Material · S1. Difference with offline weighted fusion

This section compares UpdateNet with an offline weighted fusion of three templates. On VOT2018, UpdateNet substantially outperforms both offline fusion and the baseline linear update.

  • S1. Difference with offline weighted fusion: The experiment tests whether learning an offline weighted fusion of UpdateNet’s three template inputs can improve upon the linear update.This comparison is designed to assess more sophisticated updating mechanisms.
  • S1. Difference with offline weighted fusion: UpdateNet uses a convolutional neural network that leverages previous templates to predict an accumulated template resembling the real one.Its fusion mechanism depends on the actual input features and can adapt accordingly.
  • S1. Difference with offline weighted fusion: The offline fusion combines the initial, accumulated, and current templates using learned weights initialized to 0, 0.9898, and 0.0102, respectively.The weights follow the default settings for the linear update and are trained with UpdateNet’s process until convergence.
  • S1. Difference with offline weighted fusion: 0.198 EAO is achieved by SiamFC with offline weighted fusion on VOT2018.This result is slightly higher than the baseline linear update’s 0.188 EAO.
  • S1. Difference with offline weighted fusion: 0.262 EAO is achieved by SiamFC with UpdateNet on VOT2018, compared with 0.198 for offline weighted fusion.The passage characterizes UpdateNet as significantly better than offline weighted fusion.
  • S1. Difference with offline weighted fusion: The offline weighted fusion learns a high value for αcurr, which the authors identify as one reason for UpdateNet’s superior results.The passage states that this makes the tracker likely to succumb to a further condition, but the supplied text is truncated.

S2. Visualization of updating templates

Figure S2 visualizes accumulated SiamFC templates produced by linear updating and UpdateNet across additional exemplar videos. UpdateNet’s templates more closely resemble the ground truth and yield more accurate cross-correlation response maps, including in highlighted channels from sequences A and F.

  • Visualization setup: Figure S2 compares accumulated SiamFC templates from linear update and UpdateNet across additional exemplar videos.Four of the most dynamic ground-truth channels are split visually for ease of comparison.
  • Qualitative comparison: UpdateNet learns templates that more closely resemble the ground truth and predict more accurate response maps for cross-correlation.This behavior is observed across the additional exemplar videos shown in Figure S2.
  • Qualitative comparison: In sequence A frame 125 channel 106 and sequence F frame 42 channel 121, UpdateNet and ground-truth templates share the same highlighted region, unlike linear-update templates.The linear-update templates do not resemble the ground truth in either sequence.

S3. Change rate for update

UpdateNet learns an adaptive template-update strategy whose change rate stays close to the real template and follows its trends, unlike linear update’s constant change rate. Additional results cover six example VOT2018 videos.

  • Change rate for update: UpdateNet provides an adaptive update strategy with a change rate close to that of the real template.The change rate δ is calculated between templates of contiguous frames.
  • Change rate for update: Linear update offers only a constant change rate, whereas UpdateNet’s change rate follows the same trends as the ground-truth template.The passage gives this trend agreement as an example of UpdateNet’s behavior.
  • Change rate for update: The supplementary change-rate results include six example videos from VOT2018.Figure S3 presents the change rate between contiguous frames.
Loading 1908.00855v2…