Source-linked AI summary
Dense Feature Aggregation and Pruning for RGBT Tracking
Yabin Zhu, Chenglong Li, Bin Luo, Jin Tang, Xiao Wang
TL;DR
RGBT tracking requires effective fusion of complementary RGB and thermal information, but existing approaches remain limited by unreliable or insufficient representations. DAPNet recursively aggregates features across all layers and collaboratively prunes multimodal channels, achieving state-of-the-art performance on two benchmark datasets.
Problem
Effective fusion of RGB and thermal information remains an unresolved core challenge in RGBT tracking.
Method
DAPNet recursively aggregates shallow-to-deep features and collaboratively prunes multimodal channels using global average pooling and weighted random selection.
Results
The tracker outperforms state-of-the-art RGB and RGBT trackers on the GTOT and RGBT234 benchmark datasets.
Takeaways & Limitations
Dense aggregation and collaborative pruning provide robust multimodal representations for handling appearance changes, background clutter, and partial occlusion in RGBT tracking.
Abstract
from arXiv · showhide
How to perform effective information fusion of different modalities is a core factor in boosting the performance of RGBT tracking. This paper presents a novel deep fusion algorithm based on the representations from an end-to-end trained convolutional neural network. To deploy the complementarity of features of all layers, we propose a recursive strategy to densely aggregate these features that yield robust representations of target objects in each modality. In different modalities, we propose to prune the densely aggregated features of all modalities in a collaborative way. In a specific, we employ the operations of global average pooling and weighted random selection to perform channel scoring and selection, which could remove redundant and noisy features to achieve more robust feature representation. Experimental results on two RGBT tracking benchmark datasets suggest that our tracker achieves clear state-of-the-art against other RGB and RGBT tracking methods.
1 INTRODUCTION
RGBT tracking still faces an unresolved challenge in effectively fusing RGB and thermal information. DAPNet addresses this with dense all-layer aggregation and collaborative feature pruning for robust target representations.
- RGBT tracking seeks to follow an instance across sequential frames using RGB and thermal infrared information, but effective cross-modal fusion remains unresolved.
- Handcrafted fusion methods have limited generality, while unreliable modality scores and redundant information can impair fusion under source failures.
- DAPNet recursively aggregates features from all network layers, combining shallow spatial details with deep semantic information for more accurate localization.
- Collaborative feature pruning removes noisy and redundant channels because only a few convolutional filters may be active for a particular target.
- Channel scoring and selection use global average pooling and weighted random selection during training, while pruning is removed during online tracking.
- The framework combines dense aggregation and pruning in an end-to-end network and outperforms other state-of-the-art trackers on two RGBT benchmarks.
2 RELATED WORK
Prior RGBT tracking research has emphasized sparse representation and feature aggregation to manage multimodal information and improve visual representations.
- Recent RGBT trackers mainly use sparse representation because it can suppress noise and errors.
- Feature aggregation has become popular in visual tracking for enhancing network performance and feature representations.
- Existing approaches aggregate RGB and thermal feature maps or combine handcrafted low-level features with hierarchical deep features.
3 PROPOSED APPROACH
The proposed tracker combines dense multimodal feature aggregation with collaborative channel pruning, then uses the resulting network for candidate-based tracking. Training uses pretrained convolutional layers, stochastic gradient descent, and cross-dataset experiments, while testing removes pruning and selects candidates by positive score.
- Network architecture: The network contains a dense feature aggregation module, a feature pruning module, and three fully connected layers for binary classification.It uses shared VGG-M conv1–3 backbones for RGB and thermal modalities, removes pooling after conv2, and applies dilated convolution at rate 3.
- Dense feature aggregation: Aggregation blocks recursively combine backbone and preceding aggregation outputs to propagate shallow spatial details and deep semantic information across both modalities.Different scales are aligned with max pooling, followed by 1×1 convolution, ReLU, and local response normalization.
- Feature pruning: Feature pruning scores channels with global average pooling and selects channels collaboratively using weighted random selection to suppress noisy and redundant representations.The method targets discriminative channels for localization rather than applying ordinary dropout solely as a regularizer.
- Feature pruning: Weighted random selection retains M = N ∗wrs_ratio channels by choosing the largest computed key values after assigning each channel a random number.The pruning algorithm calculates channel scores, generates random values, computes keys, and obtains the selected feature channels.
- Training and evaluation: During training, the whole network is optimized with SGD using mini-batches sampled from video sequences, while separate cross-dataset experiments train on RGBT234 or GTOT.The supplied training description specifies eight randomly chosen frames per mini-batch and experiments using 77 RGBT234 sequences or all 50 GTOT sequences.
- Tracker details: At test time, the feature pruning module is removed, convolutional filters are fixed, fully connected layers are fine-tuned, and the candidate with the maximum positive score is selected.Bounding-box regression is trained only on the first frame and applied when the estimated target state satisfies f +(z∗t ) > 0.5.
4 EXPERIMENTS
Experiments on GTOT and RGBT234 evaluate DAPNet using PR and SR against RGB and RGBT trackers, implementation baselines, variants, and qualitative sequences. DAPNet outperforms competing methods broadly, while dense aggregation and pruning support performance in challenging conditions.
- Evaluation Setting: DAPNet is evaluated on GTOT and RGBT234 using precision rate and success rate, with RGBT234 containing 234 videos and about 234,000 frames.PR measures location accuracy within a threshold, while SR measures overlap success across varying thresholds.
- Evaluation Setting: The tracker is compared with RGB and RGBT methods, including MDNet-based baselines using either six-channel concatenation or conv3 feature concatenation.The two MDNet+RGBT implementations test direct input fusion and separate convolutional feature fusion.
- Comparison with RGB Trackers: 7.0%/7.4% PR/SR gains over MDNet and 11.1%/8.9% gains over DAT are reported on the evaluated RGB-tracker comparison.These gains are reported as precision-rate/success-rate improvements.
- Comparison with RGBT Trackers: 4.1%/4.5% PR/SR gains over DAT are reported on RGBT234, alongside 4.4%/4.2% over MDNet+RGBT1 and 4.6%/6.5% over SGT.The paper states that DAPNet clearly outperforms the compared RGB and RGBT methods in all metrics on RGBT234.
- Attribute-based Performance: DAPNet performs best on most RGBT234 challenge attributes except no occlusion, partial occlusion, low illumination, background clutter, and scale variation.The reported attributes also include heavy occlusion, low resolution, thermal crossover, deformation, fast motion, motion blur, and camera moving.
- Ablation Study: Ablations compare DAPNet without feature aggregation and pruning, without aggregation, and without pruning to assess the two main components.The reported comparison identifies a substantial boost from introducing dense feature aggregation relative to the version without both components.
- Qualitative Performance: Qualitative sequences show better performance in high illumination, motion blur, scale variation, background clutter, and partial occlusion than four comparison trackers.Thermal images distinguish a target invisible in RGB, while DAPNet retains the target under partial occlusion and background clutter.
5 CONCLUSION
The paper concludes that its end-to-end network combines dense feature aggregation with collaborative feature pruning to improve RGBT tracking under challenging conditions.
- The network uses dense feature aggregation to provide powerful RGBT representations for target objects.
- Collaborative feature pruning selects discriminative feature maps from both modalities to enhance RGBT features.
- Experiments on two benchmark datasets show significantly improved tracking performance under background clutter and partial occlusion.
- Future work will pursue wider and deeper networks, improved target representations, and real-time performance.