Source-linked AI summary
Modeling and Propagating CNNs in a Tree Structure for Visual Tracking
Hyeonseob Nam, Mooyeol Baek, Bohyung Han
TL;DR
Visual tracking needs appearance models that remain reliable under abrupt changes, occlusion, and tracking failures. The paper uses multiple CNNs in a tree, combining their scores for state estimation and updating models along reliable paths. It reports outperformance of state-of-the-art techniques on OTB and VOT2015 while limiting extra cost through shared convolutional parameters.
Problem
Existing online trackers often assume smoothly changing appearances, while challenging changes can leave models insufficiently discriminative and diverse.
Method
The method maintains multiple CNN appearance models in a tree, estimates states from weighted CNN scores, and updates models along selected paths.
Results
The algorithm outperforms state-of-the-art techniques on both OTB and VOT2015 benchmarks.
Takeaways & Limitations
Tree-structured CNN models provide multi-modality and reliability while shared convolutional parameters avoid substantial additional memory and computation.
Takeaways & Limitations
TCNN is not outstanding for out-of-view situations because it uses local candidate sampling without a re-detection module.
Abstract
from arXiv · showhide
We present an online visual tracking algorithm by managing multiple target appearance models in a tree structure. The proposed algorithm employs Convolutional Neural Networks (CNNs) to represent target appearances, where multiple CNNs collaborate to estimate target states and determine the desirable paths for online model updates in the tree. By maintaining multiple CNNs in diverse branches of tree structure, it is convenient to deal with multi-modality in target appearances and preserve model reliability through smooth updates along tree paths. Since multiple CNNs share all parameters in convolutional layers, it takes advantage of multiple models with little extra cost by saving memory space and avoiding redundant network evaluations. The final target state is estimated by sampling target candidates around the state in the previous frame and identifying the best sample in terms of a weighted average score from a set of active CNNs. Our algorithm illustrates outstanding performance compared to the state-of-the-art techniques in challenging datasets such as online tracking benchmark and visual object tracking challenge.
1. Introduction
The paper addresses the difficulty of maintaining reliable, diverse target appearance models when abrupt changes violate smooth temporal updates. It proposes tree-structured multiple CNNs for robust online tracking and reports outstanding benchmark accuracy.
- Motivation: Abrupt motion, occlusion, illumination variation, and deformation can break the smooth-appearance assumption used by many online trackers.Existing models may therefore need to handle appearance changes that are not well represented by sequential temporal updates.
- Motivation: Sequential online updates may produce models that are insufficiently discriminative and diverse for challenging target appearances.The paper identifies limited temporal-order learning as a weakness of existing multiple-model and multimodal approaches.
- Approach: The proposed algorithm maintains multiple CNN appearance models in a tree and updates them online along selected paths.Shared convolutional representations support multiple models while path histories preserve distinct appearance changes.
- Approach: Multiple models capture diverse appearances and support more robust tracking during appearance changes, occlusions, and temporary tracking failures.The tree structure is intended to preserve separate appearance histories while maintaining reliable updates.
- Results: The method outperforms state-of-the-art methods by large margins on standard tracking benchmarks.The introduction characterizes the reported benchmark accuracy as outstanding.
2. Related Work
Related tracking work includes discriminative tracking-by-detection, multiple appearance models, and CNN-based representations. The paper situates its approach among methods that address appearance variation while noting challenges in applying CNNs to online learning.
- Scope: The paper reviews these lines of work to position its method within tracking-by-detection, multiple-model tracking, and CNN representation learning.Its related-work scope explicitly focuses on these three areas.
- Tracking-by-detection: Tracking-by-detection formulates tracking as discriminative object classification between targets and surrounding backgrounds.These methods learn classifiers across video frames to construct robust target models under dynamic appearance changes.
- CNN representations: CNN-based tracking is difficult to train online because noisy labels and few examples can cause overfitting and weaker performance than hand-crafted features.The related-work discussion emphasizes that deep representations are effective in recognition but not straightforward to adapt for online tracking.
- Multiple appearance models: Prior generative trackers use multiple templates, observation models, or frame patches to handle target appearance variations and tracking failures.Examples include sparse-representation templates, MCMC-integrated observation models, and patch matching across frames.
3. Algorithm Overview
The algorithm combines state estimation and online model updating through multiple CNNs organized in a tree. Candidate likelihoods use CNN weights derived from path reliability, while updates follow the highest-weight contributing model.
- Tree-structured models: Multiple CNN appearance models are organized in a tree to preserve consistency and represent multimodal target appearances.State estimation and model updating both require interaction among the CNNs.
- State estimation: For each new frame, candidate samples are drawn around the previous target state and scored using a weighted average from multiple CNNs.The target state is selected by finding the candidate with maximum likelihood.
- State estimation: CNN weights depend on the reliability of the tree path along which each CNN was updated.This links contribution to state estimation with the reliability of the model’s update history.
- Model update: After a predefined number of frames, a new CNN is derived from the highest-weight contributing CNN to promote smooth updates and reliable models.The update chooses the model with the highest contribution to the current state estimate.
- Novelty: The tree differs from a CNN candidate pool by explicitly preserving multimodality and maximizing model reliability through maintenance of hierarchical relationships.The paper contrasts this objective with prototype-distance selection and prior offline or graph-based propagation approaches.
4. Proposed Algorithm
The proposed tracker organizes multiple CNN appearance models in a tree, combining their scores for state estimation and selecting reliable paths for online updates. Shared convolutional parameters and feature reuse reduce the extra cost of maintaining multiple models.
- Tree construction: A tree vertex stores one CNN, while directed edges encode relationships between models through appearance affinity.The affinity is computed from consecutive frames used to train the child model and predicted positive scores from the parent CNN.
- Target state estimation: The tracker samples target candidates around the previous state and selects the candidate with the highest weighted average score from active CNNs.The active set consists of N candidate samples and multiple CNNs contributing to current-frame estimation.
- Target state estimation: CNN weights combine affinity to the current frame with path-based reliability, penalizing models whose update histories contain unreliable edges.Reliability is computed recursively from the bottleneck edge, while affinity uses the maximum positive score over candidates.
- Model update: Multiple CNNs in diverse tree branches capture multimodal appearances while smooth path updates preserve model reliability.The method retains CNNs updated from problematic frames because background-following models can help during severe occlusion.
- Model update: A new CNN node is created after 10 consecutive frames without model updates, using a parent selected to maximize its reliability.The new CNN is fine-tuned from the selected parent with samples from the new node's frames and the parent's frame set; the active set retains the 10 most recent CNNs.
- CNN architecture and efficiency: Only fully connected layers are updated, while shared convolutional layers and cached conv3 features avoid redundant computation across CNNs.The network uses transferred VGG-M convolutional layers and randomly initialized fully connected layers; maintaining multiple CNNs therefore adds little time or space cost.
5. Experiment
Experiments on OTB50, OTB100, and VOT2015 evaluate TCNN against state-of-the-art trackers and internal variants. TCNN generally achieves strong performance, while its limitations are most apparent in strict localization and out-of-view tracking.
- OTB evaluation: TCNN outperforms all compared trackers by substantial margins on OTB50 and OTB100.
- OTB evaluation: TCNN is slightly less accurate than MUSTer and SRDCF at strict thresholds, where CNN features are less effective for tight localization.
- OTB evaluation: TCNN handles all eight OTB50 challenge attributes effectively compared with existing state-of-the-art methods.The attributes include rotation, deformation, motion blur, low resolution, illumination variation, fast motion, occlusion, and out of view.
- OTB evaluation: TCNN is not outstanding for out-of-view targets because local candidate sampling lacks a re-detection module.
- Internal analysis: Internal comparisons show that model diversity, tree-based updates, and the combined TCNN state-estimation strategy each improve tracking performance.Multiple models outperform a single sequentially updated model; tree updates outperform single-path updates; TCNN outperforms all tested variants.
- Internal analysis: Bounding box regression improves localization, with TCNN without regression reaching precision/success pairs of (0.923, 0.669) on OTB50 and (0.870, 0.634) on OTB100.
- Internal analysis: The tree maintains multiple reliable paths and isolates frames with substantially different appearances in local branches, helping avoid drift.
- VOT2015 evaluation: On VOT2015, TCNN outperforms most compared algorithms across evaluation metrics and remains consistently strong on both OTB and VOT2015.DeepSRDCF and EBT show comparable VOT2015 results, while TCNN records an OTB50 success rate of 0.682 versus 0.649 and 0.581.
6. Conclusion
The proposed tracker maintains multiple CNNs in a tree to model multi-modal and reliable target appearances while estimating states from their weighted scores. Shared convolutional parameters limit the added memory and computation, and the method outperforms state-of-the-art techniques on OTB and VOT2015.
- Multiple CNNs maintained in a tree help represent multi-modal and reliable target appearances.
- Target state estimation uses a weighted average of scores from multiple CNNs, with the tree determining each CNN’s contribution.
- Shared convolutional-layer parameters allow multiple CNNs without substantial increases in memory or computation.
- The tracking algorithm outperforms state-of-the-art techniques on both OTB and VOT2015 benchmarks.