Source-linked AI summary
Know Your Surroundings: Exploiting Scene Information for Object Tracking
Goutam Bhat, Martin Danelljan, Luc Van Gool, Radu Timofte
TL;DR
Appearance-only trackers can fail with distractors and rapid target changes, motivating the use of surrounding-scene information. The paper represents that information as dense state vectors, propagates it through frames, and combines it with appearance cues; it achieves state-of-the-art results on three benchmarks, including 63.6% AO on GOT-10k.
Problem
Appearance models alone can be insufficient for robust tracking when distractors are present or the target changes appearance quickly.
Method
The tracker represents scene information as dense localized state vectors, propagates them through frames using dense correspondence, and combines them with target appearance output.
Results
The tracker sets a new state-of-the-art on 3 tracking benchmarks and achieves 63.6% AO on GOT-10k.
Takeaways & Limitations
Scene information provides complementary cues that benefit tracking even when integrated with a strong appearance model.
Takeaways & Limitations
A failure case occurs when the appearance model misses occlusion by a white dog, causing incorrect state-vector updates and tracking drift to the dog.
Abstract
from arXiv · showhide
Current state-of-the-art trackers only rely on a target appearance model in order to localize the object in each frame. Such approaches are however prone to fail in case of e.g. fast appearance changes or presence of distractor objects, where a target appearance model alone is insufficient for robust tracking. Having the knowledge about the presence and locations of other objects in the surrounding scene can be highly beneficial in such cases. This scene information can be propagated through the sequence and used to, for instance, explicitly avoid distractor objects and eliminate target candidate regions. In this work, we propose a novel tracking architecture which can utilize scene information for tracking. Our tracker represents such information as dense localized state vectors, which can encode, for example, if the local region is target, background, or distractor. These state vectors are propagated through the sequence and combined with the appearance model output to localize the target. Our network is learned to effectively utilize the scene information by directly maximizing tracking performance on video segments. The proposed approach sets a new state-of-the-art on 3 tracking benchmarks, achieving an AO score of 63.6% on the recent GOT-10k dataset.
1 Introduction
Appearance-only tracking struggles with distractors and rapid target changes because it does not preserve information about surrounding objects. The proposed tracker propagates dense scene information and combines it with target appearance to improve localization, achieving state-of-the-art results across five benchmarks.
- Motivation: Appearance-only trackers can fail in cluttered scenes or when the target changes appearance quickly.Distractors may resemble previous target appearances, while updated appearance models do not capture other objects’ locations and characteristics.
- Approach: The tracker represents local scene regions as dense state vectors encoding target, background, or distractor information.These vectors are propagated through consecutive frames using dense correspondence maps.
- Approach: The predictor combines propagated scene information with the target appearance model to determine the target location.A recurrent neural network updates the state representation with new information.
- Training: The network is trained to directly maximize tracking performance on complete video segments.This training objective is part of the proposed tracking architecture.
- Results: The approach achieves state-of-the-art results on all five evaluated benchmarks.The benchmarks are VOT2018, GOT-10k, TrackingNet, OTB-100, and NFS.
- Results: 63.6% AO on GOT-10k, outperforming the previous best approach by 2.5%.The paper also reports an ablation study of key tracking components.
2 Related Work
Related tracking methods primarily learn target appearance models, sometimes incorporating background or motion cues to improve robustness. The proposed approach differs by explicitly propagating information about background objects and structures rather than using those cues only to refine the target model.
- Appearance-based tracking: Most tracking methods learn an appearance model from the target’s first frame.The related work includes discriminative correlation filters, deep target-background classifiers, and Siamese similarity models.
- Appearance-based tracking: Background information is often incorporated into appearance models, but a single target model can remain insufficient when distractors resemble prior target appearances.Fast motion can also make rapid adaptation to new distractors difficult.
- Motion cues: Some trackers use optical flow to complement features or warp and aggregate feature maps for target appearance modeling.These approaches use motion information to improve the robustness of the target model.
- Positioning: Unlike these methods, the proposed approach explicitly propagates dense motion information about background objects and scene structures to complement the target model.Its distinction is the use of motion for scene representation, not only target-model refinement.
- Recurrent models: Other approaches use recurrent networks to regress target locations, process proposals, or update target models for appearance changes.Examples include RNN- and LSTM-based tracking architectures.
3 Proposed Method
The proposed tracker complements a target appearance model with propagated scene information encoded in localized state vectors. It propagates, combines, and updates these states across frames to improve target localization.
- The tracker propagates scene information alongside the target appearance model to improve target prediction in each frame.Scene information is encoded as localized state vectors that can represent target, background, or distractor regions.
- Each region in the target neighborhood maintains a state vector that moves with objects through the sequence.States are initialized from the first-frame target annotation and propagated to compensate for object and camera motion.
- The predictor fuses propagated states, propagation reliability, and the current appearance score into final target confidence scores.The target is localized at the spatial location with the highest fused confidence score.
- A recurrent update module uses current predictions and appearance scores to refresh states, reset incorrect states, or flag newly entered distractors.Updated states are then used for tracking in the next frame.
- Dense correspondences between consecutive frames are estimated from feature correlations to propagate previous-frame states to current-frame locations.The method constructs a partial 4D cost volume using 3 × 3 feature windows and a maximum displacement assumption.
- The scene-aware architecture can be integrated with any appearance model; the work uses DiMP as its target model component.DiMP is selected because of its strong performance and supplies the appearance model used by the tracker.
4 Experiments
Experiments evaluate the tracker across five benchmarks, including ablations of scene information and comparisons with state-of-the-art methods. The approach improves over the appearance-only baseline and achieves leading results on several datasets.
- Benchmark setup: The tracker is evaluated on five benchmarks: VOT2018, GOT-10k, TrackingNet, OTB-100, and NFS.It operates at around 20 FPS on a single Nvidia RTX 2080 GPU.
- Ablation study: The ablation study uses 200 combined NFS and OTB-100 videos, evaluates overlap precision, and averages results over five runs.Reported metrics are AUC and OP0.5.
- Ablation study: 1.3% AUC improvement is obtained by integrating scene information compared with using only the target appearance model.The corresponding OP0.5 improvement is 1.6%.
- State-of-the-art comparison: 5.0% relative EAO gain over DiMP-50 is achieved on VOT2018.The benchmark compares expected average overlap, accuracy, and robustness.
- State-of-the-art comparison: 63.6 AO is achieved on GOT-10k, where the approach obtains the best results in average overlap and success rates at thresholds 0.5 and 0.75.The result is reported on the GOT-10k test set.
- State-of-the-art comparison: The approach performs similarly to DiMP-50 on TrackingNet, achieving an AUC score of 74.0%.TrackingNet reports precision, normalized precision, and success.
5 Conclusions
The paper proposes tracking with propagated scene information represented as dense localized state vectors and combined with appearance predictions. Evaluations on five benchmarks establish new state-of-the-art results on three of them.
- Conclusion: The tracker represents scene information as dense localized state vectors propagated through the sequence and combined with appearance-model output.The states encode information about target, background, or distractor regions.
- Conclusion: The proposed approach sets a new state-of-the-art on 3 of 5 tracking benchmarks.The conclusion attributes the result to exploiting scene information for tracking.
Supplementary Material
The supplementary material adds architectural details, VOT2018 results, and qualitative comparisons with the DiMP-50 baseline.
- Supplementary Material: Section 6 provides additional details about the tracking architecture.
- Supplementary Material: Section 7 contains detailed VOT2018 results, while Section 8 provides qualitative comparisons with DiMP-50.
6 Network details
The tracker initializes dense state vectors from the first-frame target annotation, propagates them, and updates them with a ConvGRU-based state module. A predictor processes propagated states, reliability scores, and appearance outputs using shared convolutional components.
- State initializer: The state initializer converts a Gaussian target-center label map into initial state vectors through a 3×3 convolution and tanh activation.The input is the first-frame target annotation B0.
- State update: The state update module uses a convolutional gated recurrent unit to update state vectors with new information.The implementation uses 3 × 3 kernels for convolution layers and sigmoid and hyperbolic tangent activations.
- Cost-volume processing: The cost-volume processing network takes individual 18 × 18 × 1 slices, uses 3 × 3 convolutions, and applies batch normalization.The same architecture processes the initial correspondence φ′.
- Predictor module: The predictor concatenates propagated states, reliability scores, and appearance outputs before processing them with convolutional blocks.Its inputs have shapes 18 × 18 × 8, 18 × 18 × 1, and 18 × 18 × 1, respectively.
7 Detailed Results on VOT2018
On VOT2018, the tracker is evaluated with expected average overlap across sequence lengths and achieves the best reported comparison against recent state-of-the-art trackers.
- Evaluation and comparison: The evaluation plots expected average overlap between tracker predictions and groundtruth for different sequence lengths.The EAO score is the average over typical sequence lengths.
- Evaluation and comparison: 0.462 EAO is achieved on VOT2018, outperforming DiMP-50 by a relative improvement of 5%.The benchmark contains 60 challenging videos, and EAO ranks trackers by averaging expected overlap over typical sequence lengths.
8 Qualitative Results
Qualitative comparisons show that propagated scene information helps tracking amid appearance-similar distractors and fast target appearance changes. The examples also include a failure caused by incorrect state updates during an occlusion.
- Distractors and appearance changes: Scene-aware tracking handles distractor objects that are difficult to distinguish from the target using appearance alone.The comparison is against DiMP-50, which uses only an appearance model.
- Comparison setup: The qualitative figure compares tracking outputs from the proposed approach and the DiMP-50 baseline across example sequences.The rows include cases involving distractors, fast appearance changes, and one failure case.
- Distractors and appearance changes: Propagated scene information helps eliminate target candidate regions when the target undergoes fast appearance changes.Keeping track of background regions simplifies target localization in the cited examples.
- Failure case: The tracker fails when its appearance model misses occlusion by a white dog, causing incorrect state-vector updates and subsequent tracking of the dog.This is shown as the last-row failure case.