Source-linked AI summary
Material Based Object Tracking in Hyperspectral Videos: Benchmark and Algorithms
Fengchao Xiong, Jun Zhou, Yuntao Qian
TL;DR
Color-based tracking can fail when appearance is ambiguous, motivating better use of material information in hyperspectral videos. The paper builds a paired benchmark, represents material through spectral-spatial gradients and abundances, and embeds these features in correlation filters. Experiments report improved tracking performance, including a 61.2% AUC, while also exposing sensitivity to illumination-related fixed endmembers.
Problem
Color-based trackers can drift in cluttered or changing scenes, while hyperspectral tracking lacks diverse benchmarks and effective ways to extract material information.
Method
The paper constructs paired hyperspectral/color videos and embeds SSHMG and abundance-based material features with adaptive weighting into correlation-filter tracking.
Results
The proposed tracker ranks top across thresholds with an AUC of 61.2%, and combining abundances with SSHMG outperforms SSHMG and HOG in average distance precision.
Takeaways & Limitations
Experiments show that hyperspectral material properties have potential to improve tracking in background clutter, rotation, and deformation scenes.
Abstract
from arXiv · showhide
Traditional color images only depict color intensities in red, green and blue channels, often making object trackers fail in challenging scenarios, e.g., background clutter and rapid changes of target appearance. Alternatively, material information of targets contained in a large amount of bands of hyperspectral images (HSI) is more robust to these difficult conditions. In this paper, we conduct a comprehensive study on how material information can be utilized to boost object tracking from three aspects: benchmark dataset, material feature representation and material based tracking. In terms of benchmark, we construct a dataset of fully-annotated videos, which contain both hyperspectral and color sequences of the same scene. Material information is represented by spectral-spatial histogram of multidimensional gradient, which describes the 3D local spectral-spatial structure in an HSI, and fractional abundances of constituted material components which encode the underlying material distribution. These two types of features are embedded into correlation filters, yielding material based tracking. Experimental results on the collected benchmark dataset show the potentials and advantages of material based object tracking.
I. INTRODUCTION
Traditional color-video trackers can drift when appearance cues are ambiguous, while hyperspectral material information offers additional discrimination. The paper addresses this gap by introducing a benchmark and material-aware tracking features, supported by video-rate paired hyperspectral and RGB data.
- Color-video trackers can drift under background clutter, similar target and background colors, rotation, and deformation.
- Hyperspectral spectra provide material-constitution details that increase discrimination between objects with similar visible colors.The figure contrasts plastic and cloth toys whose pixels appear similar in color but have different spectral responses.
- Hyperspectral tracking remains challenging because many spectral bands, lower spatial resolution, and conventional feature limitations hinder robust material extraction.
- The work introduces a diverse hyperspectral tracking benchmark to address the shortage of datasets covering close-range challenges such as rotation, deformation, and illumination variation.
- The paper proposes SSHMG and abundance features to capture local spectral-spatial structure and underlying material distributions, with online learning to adapt their relative importance.
- The dataset includes hyperspectral and RGB videos acquired at 25 frames per second from closely matched viewpoints for comparison with color-based trackers.Each hyperspectral frame is a 3D cube with spatial dimensions and a spectral-band dimension.
B. Data Annotation
The dataset pairs 35 color and 35 hyperspectral videos, averaging 500 frames per sequence, and covers diverse targets and challenging tracking attributes. The proposed SSHMG extracts local spectral-spatial gradient structure from HSI cubes, aggregates orientation features, and normalizes them into block descriptors.
- B. Data Annotation: Hyperspectral and color videos are annotated independently with upright bounding boxes because target locations may differ slightly between the two captures.
- B. Data Annotation: The dataset contains 35 color videos and 35 hyperspectral videos, with an average of 500 frames per sequence.
- B. Data Annotation: Videos cover diverse targets, scenarios, activities, and content, including vehicles, faces, people, generic objects, and animals.
- B. Data Annotation: All videos are labeled using eleven challenging attributes, including illumination, scale, occlusion, deformation, motion blur, fast motion, rotations, out-of-view, clutter, and low resolution.
- C. Data Attributes: MHT weights SSHMG and abundance features by reliability, learns correlation filters, sums multi-channel responses, and updates filters using BACF.
- A. Spectral-spatial Histogram of Multidimensional Gradients: SSHMG computes multidimensional gradients in spatial and spectral directions, represents them by magnitude and orientations, and aggregates orientations within local 3D cubes.
- A. Spectral-spatial Histogram of Multidimensional Gradients: The cube-level features are concatenated over overlapping 2×2 spatial blocks, normalized to unit length, and truncated to limit feature-vector maxima.
B. Material Distribution Learning
Material distribution learning uses hyperspectral unmixing to represent mixed pixels through endmembers and abundances. The method adopts semi-supervised endmember selection from an offline spectral library, followed by efficient abundance estimation.
- B. Material Distribution Learning: Hyperspectral unmixing decomposes mixed pixels into spectral signatures called endmembers and their associated per-pixel material proportions called abundances.
- B. Material Distribution Learning: The paper adopts semi-supervised unmixing to balance tracking speed and endmember accuracy, selecting endmembers from an offline library built from many HSIs.
- B. Material Distribution Learning: The spectral library clusters VCA-extracted endmembers with K-means, and cluster centers provide spectral reflectances for subsequent processing.
- B. Material Distribution Learning: CLSUnSAL estimates material contributions from the predefined library, after which HySime selects the top R spectral signatures as scene endmembers.
- B. Material Distribution Learning: SPU estimates abundances from the selected endmembers because of its computational efficiency and reported superior performance.
C. Material Based Object Tracking Method
The method embeds material features in a BACF correlation-filter tracker, using group-wise reliability to weight feature groups during optimization, detection, and updating.
- Tracking formulation: BACF learns correlation filters from material features, with spatial regularization and feature weights incorporated into the objective.The formulation uses SSHMG and abundances as material features, while a binary matrix crops the central patch.
- Optimization: FFT reformulates the filter-learning problem in the frequency domain, where an augmented-Lagrangian procedure solves three subproblems.The formulation introduces Fourier-domain variables, an auxiliary matrix, a regularization factor, and a multiplier.
- Reliability weighting: Group-wise reliability represents the importance of feature groups and adaptively suppresses less reliable features.All abundance channels jointly encode the distribution of underlying materials, forming one physical-property group.
- Model optimization: Feature reliability is initialized equally, recomputed after each frame, and used when learning the correlation filters.The filters are learned with the reliability-weighted features under the BACF framework.
- Model detection: During detection, reliability-weighted SSHMG and abundances are evaluated across multiple candidate resolutions to handle scale changes.Weighted channel responses are summed to obtain the final response, and interpolation estimates location and scale with sub-grid precision.
- Model update: During updating, group reliability scores and models are updated autoregressively to accommodate scale, pose, and rotation changes.The update uses a learning rate η.
IV. EXPERIMENTS
The experiments evaluate material-based tracking from feature representation, comparisons with state-of-the-art trackers, and attribute- and quality-based analyses.
- Experimental scope: The study first examines the proposed tracking method from a feature-representation perspective.
- Experimental scope: The evaluation compares hand-crafted, deep-feature, and hyperspectral trackers using automatically configured parameters.
- Experimental scope: Attribute-based and quality-based comparisons are also reported.
A. Experimental Setting
The experimental setting uses precision, success, and AUC protocols, with fixed SSHMG and optimization parameters for tracker evaluation.
- Parameter setting: SSHMG uses α = 0.2, z = 4, nθ = 9, and nφ = 4, while the learning rate is η = 0.0023.Other parameters match BACF settings.
- Evaluation protocols: Three protocols—precision plot, success plot, and AUC—measure tracker performance.Precision reports localization within a distance threshold, while success reports bounding-box overlap performance.
- Evaluation protocols: Average distance precision is reported at a threshold of 20 pixels.
B. Effectiveness of Proposed Material Feature
Feature comparisons show that spectral-spatial structure and material abundances are more effective for tracking than raw spectra or generic color-image descriptors.
- Feature comparison: The comparison evaluates spectrum, HOG, BHOG, abundances, SSHMG, and their combination MHT.Raw spectral responses, spatial gradients, spectral-spatial gradients, and material distributions are tested as tracking features.
- Feature comparison: Raw spectral features perform worst because they are sensitive to illumination changes.
- Feature comparison: 0.4% AUC gain is achieved by SSHMG compared with HOG.
- Feature comparison: MHT combines SSHMG with abundances and dominates SSHMG and HOG by a relatively large margin in average distance precision.The reported explanation is the hybrid benefit of local spectral-spatial structure and underlying material distribution.
- Feature comparison: Abundances improve performance by exploiting underlying material distribution, while HOG and BHOG benefit from local spatial structure.
C. Quantitative Comparison with Hand-crafted Feature-based Trackers.
MHT is compared with hand-crafted trackers on RGB, hyperspectral, and false-color videos, achieving the strongest reported performance among these alternatives.
- 7.2% and 3.4% AUC gains are obtained over BACF on color and hyperspectral videos, respectively.
- MHT uses spectral-spatial and material-distribution information to represent image content and learn correlation filters from more reliable features.
- MHT significantly outperforms the other compared trackers on both color and false-color videos in the reported comparisons.
- MHT outperforms CNHT and DeepHKCF because it combines local spectral-spatial texture information with detailed material information without reducing HSI data to false color.
- MHT outperforms the compared hand-crafted feature-based trackers on RGB videos.
- 61.2% AUC is achieved by MHT on hyperspectral or corresponding false-color videos, the best accuracy reported in Fig. 7.
F. Attribute-based Evaluation
Attribute-based evaluation shows that MHT ranks first on five of eleven attributes and is particularly effective when color or spatial appearance is unreliable, though illumination variation remains difficult.
- Material information helps MHT discriminate targets from backgrounds in background-clutter situations where robust color features are difficult to extract.
- Underlying material information is more robust than spatial information when targets are partly or fully deteriorated under DEF, OPR and IPR attributes.
- MHT fails to outperform color trackers in IV because its fixed endmembers are related to illumination conditions.
- MHT shows higher robustness than competing trackers in qualitative sequences involving rotation, deformation, clutter, and low-resolution targets.
V. CONCLUSION
The paper introduces a hyperspectral tracking benchmark and represents material properties with SSHMG and abundance features. Experiments show potential for material-based tracking in challenging scenes.
- The paper introduces a benchmark dataset for object tracking in hyperspectral videos and studies tracking using material information.
- SSHMG encodes local spectral-spatial texture, while abundance features describe constituent material distributions through hyperspectral unmixing.
- Extensive benchmark experiments show that material properties contribute to object tracking, especially in background clutter, rotation, and deformation scenes.
- The results indicate that material information in HSIs has strong potential for object tracking.
APPENDIX A
The appendix describes dataset visualization, sensor calibration, sequence registration, and conversion of hyperspectral data into false-color images for comparison.
- The benchmark displays first-frame ground-truth bounding boxes and labels sequences using 11 challenging attributes.
- Spectral calibration combines dark-frame subtraction with spectral correction to reduce sensor noise and unwanted second-order responses.
- Color and hyperspectral sequences are manually registered using matched key points and a geometrical transformation applied to subsequent color frames.
- Hyperspectral videos are converted to false-color images through CIE color matching functions that weight wavelengths when generating color channels.