PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Dynamic Memory Networks for Object Tracking

Tianyu Yang, Antoni B. Chan

arXiv:1803.07268v2cs.CV

TL;DR

Fast template-matching trackers lack effective adaptation to target appearance changes, while tracking-by-detection requires expensive online fine-tuning. MemTrack uses an LSTM-controlled external memory with attention and gated residual template learning to adapt feed-forward; experiments report favorable performance at 50 fps.

  • Problem

    Template-matching trackers are fast but lack effective adaptation to target appearance variations, limiting their tracking accuracy relative to state-of-the-art methods.

  • Method

    MemTrack uses an LSTM-controlled external addressable memory with attention and channel-wise gated residual template learning to update matching templates without online fine-tuning.

  • Results

    MemTrack performs favorably against state-of-the-art tracking methods while retaining real-time speed of 50 fps.

  • Takeaways & Limitations

    External memory lets the tracker adapt to appearance variations while preserving the initial template and allowing memory capacity to grow with task requirements.

Abstract

from arXiv · show

Template-matching methods for visual tracking have gained popularity recently due to their comparable performance and fast speed. However, they lack effective ways to adapt to changes in the target object's appearance, making their tracking accuracy still far from state-of-the-art. In this paper, we propose a dynamic memory network to adapt the template to the target's appearance variations during tracking. An LSTM is used as a memory controller, where the input is the search feature map and the outputs are the control signals for the reading and writing process of the memory block. As the location of the target is at first unknown in the search feature map, an attention mechanism is applied to concentrate the LSTM input on the potential target. To prevent aggressive model adaptivity, we apply gated residual template learning to control the amount of retrieved memory that is used to combine with the initial template. Unlike tracking-by-detection methods where the object's information is maintained by the weight parameters of neural networks, which requires expensive online fine-tuning to be adaptable, our tracker runs completely feed-forward and adapts to the target's appearance changes by updating the external memory. Moreover, unlike other tracking methods where the model capacity is fixed after offline training --- the capacity of our tracker can be easily enlarged as the memory requirements of a task increase, which is favorable for memorizing long-term object information. Extensive experiments on OTB and VOT demonstrates that our tracker MemTrack performs favorably against state-of-the-art tracking methods while retaining real-time speed of 50 fps.

1 Introduction

MemTrack addresses the limited appearance adaptation of fast template-matching trackers with an external-memory network that updates templates feed-forward. Its gated residual design preserves the initial template while adapting through retrieved memory, achieving favorable tracking performance at 50 fps.

  • Motivation: Template-matching trackers are fast because they avoid neural-network parameter updates, but their adaptation to target appearance changes remains limited.Tracking-by-detection methods can achieve high accuracy but typically require expensive online fine-tuning; cited examples run at about 1 fps.
  • Dynamic memory network: MemTrack stores and recalls target information from external memory to maintain object-appearance variations during template matching.The memory capacity can be enlarged independently of the neural-network parameters, supporting longer-term appearance information.
  • Dynamic memory network: An LSTM with attention controls memory access by concentrating on potential target regions in the search feature map.The attention mechanism helps generate read and write control signals when the target location is initially unknown.
  • Gated residual template learning: Gated residual template learning combines retrieved memory with the initial template while controlling the added appearance variation channel-wise.This design prevents excessive model updating while retaining the initial template’s conservative target information.
  • Evaluation: 50 fps: MemTrack performs favorably against state-of-the-art tracking methods while retaining real-time speed.The tracker requires no online fine-tuning after offline training.

2 Related Work

MemTrack extends template-matching tracking with addressable external memory, distinguishing its adaptation mechanism from both conventional matching trackers and neural memory approaches. The design addresses unknown target location, high-dimensional visual features, and controlled template updating.

  • Template-Matching Trackers: Template-matching trackers use an initial or recent target patch as the matching model, but existing online updating provides limited adaptation.SiamFC uses only the first-frame template, while interpolation-based updating yields modest accuracy improvements.
  • Template-Matching Trackers: MemTrack enlarges adaptive capacity by adding addressable memory whose size is independent of the neural-network feature extractor.This separates memory capacity from the fixed network size and supports increasing memory requirements.
  • Memory Networks: Unlike LSTM-based tracking memory, the proposed external memory uses addressable storage to manage object-appearance changes beyond hidden-state capacity.The comparison identifies RFL as a related approach that maintains appearance variations in LSTM memory states.
  • Memory Networks: Attention highlights target information before LSTM memory retrieval because the target location is initially unknown in the search image.Average pooling reduces the feature representation used to generate an efficient memory-addressing key.
  • Memory Networks: Channel-wise gated residual template learning updates the matching model while preserving information from the initial target template.The method retrieves related templates and controls how much of their appearance variation contributes to matching.

3 Dynamic Memory Networks for Tracking

MemTrack combines CNN features, attention-controlled memory access, and gated residual templates to adapt template matching to appearance changes. Its memory-writing mechanism selectively preserves, updates, or replaces stored templates.

  • 3.1 Feature Extraction: MemTrack extracts search features with a fully convolutional network and uses an attentional LSTM to control memory reading and writing.The predicted target box supplies the object patch used to compute a new template for memory writing.
  • 3.2 Attention Scheme: Attention weights aggregate search-image patches into an attended feature vector that helps the LSTM generate a memory read key despite the unknown target location.Average pooling reduces each patch before the weighted aggregation; the implementation uses 6 × 6 × 256 patches matching the template size.
  • 3.4 Memory Reading: The controller retrieves templates by weighting memory slots according to cosine similarity between the read key and stored memory keys.The LSTM also outputs read strength, which indicates the reliability of the generated read key.
  • 3.5 Residual Template Learning: A channel-wise residual gate controls how much of the retrieved template is added to the initial template, limiting over-adaptation while updating target parts separately.The paper links this design to feature channels that respond to different object parts and reports a large performance improvement from channel-wise residual learning.
  • 3.6 Memory Writing: Memory writing distinguishes unreliable templates, small appearance changes, and large appearance changes through write, read, and allocation gates.The gates interpolate between writing nothing, updating a previously read slot, and overwriting a newly allocated slot; infrequently accessed slots receive new templates.

4 Implementation Details

The tracker is trained offline on video clips with augmented appearance variation and implemented for real-time inference. Its ablation studies evaluate tracker variants and memory sizes on OTB-2015.

  • 4 Implementation Details: The ablation study compares tracker variants and memory sizes of 1, 2, 4, 8, and 16 slots using OTB-2015 success plots.The memory-size experiment evaluates how increasing external memory affects tracking performance.
  • 4 Implementation Details: Training uses the ILSVRC VID dataset, Adam optimization, batches of 8 video clips of length 16, and offline augmentation of appearance variation.Uniform temporal sampling and augmentation simulate fast motion, background change, object jitter, and low frame rate.
  • 4 Implementation Details: 50 fps is achieved with a TensorFlow implementation on four Intel i7-7700 CPUs and one NVIDIA GTX 1080 Ti.The reported runtime is for the complete implemented algorithm on the stated hardware.

5 Experiments

Experiments evaluate MemTrack across OTB-2013, OTB-2015, and VOT-2016, including ablations of attention, memory design, residual learning, and memory size. MemTrack generally outperforms real-time and baseline trackers while operating at 50 fps.

  • 5.1 Ablation Studies: MemTrack uses attention, dynamic memory, and residual template learning; ablations show each component contributes to tracking performance.Removing attention, replacing dynamic memory with a queue, using hard template reading, or removing residual gates decreases performance.
  • 5.1 Ablation Studies: Tracking accuracy increases with memory size and saturates at 8 memory slots, which is selected as the default for runtime and memory efficiency.
  • 5.2 Comparison Results: 4.9% precision and 5.8% success improvements over SiamFC are reported on OTB-2013, where MemTrack achieves the best success AUC among recent real-time trackers.
  • 5.2 Comparison Results: 6.4% precision and 7.6% success improvements over SiamFC are reported on OTB-2015, where MemTrack outperforms the other real-time methods in both measures.MemTrack also has similar AUC performance to several non-real-time trackers running at about 1 fps.
  • 5.2 Comparison Results: 10.7% higher AUC than SiamFC is reported for OTB-2015 low-resolution sequences, and MemTrack outperforms all compared trackers across eight challenging attributes.
  • 5.2 Comparison Results: MemTrack exceeds the reported VOT-2016 SOTA EAO bound with 0.273 while running at 50 fps, though CCOT, TCNN, and DeepSRDCF achieve higher EAO.

6 Conclusion

MemTrack uses an external addressable memory network to adapt matching templates to appearance variations without online fine-tuning. Its channel-wise gated residual learning preserves initial target information while supporting real-time tracking at 50 fps.

  • MemTrack uses an external addressable memory block to adapt matching templates to object appearance variations.
  • An attention-controlled LSTM parameterizes the memory interactions during tracking.
  • Channel-wise gated residual template learning preserves conservative initial-target information while adapting each feature channel.
  • 50 fps is achieved after offline training without online fine-tuning.
  • Extensive experiments on standard tracking benchmarks demonstrate MemTrack's effectiveness.
Loading 1803.07268v2…