PaperlayerView exact revision on arXiv

Source-linked AI summary

RGB-T Object Tracking:Benchmark and Baseline

Chenglong Li, Xinyan Liang, Yijuan Lu, Nan Zhao, Jin Tang

arXiv:1805.08982v1cs.CV

TL;DR

RGB-T tracking lacks a sufficiently comprehensive evaluation platform despite complementary visible and thermal information. The paper introduces RGBT234 and a graph-based tracker with jointly learned patch, graph, and modality weights. Experiments report strong performance, including leading MPR and EAO results and advantages over several deep-feature trackers.

  • Problem

    RGB-T tracking lacks a comprehensive benchmark despite complementary visible and thermal information that supports tracking across challenging conditions.

  • Method

    The paper builds RGBT234 and learns patch-based RGB-T object representations with dynamically learned graph and modality weights in a unified ADMM framework.

  • Results

    SGT performs best in MPR and EAO, while outperforming ECO, C-COT, CFnet, and CFnet+RGBT in reported MPR/EAO comparisons.

  • Takeaways & Limitations

    The benchmark supports large-scale, occlusion-sensitive RGB-T evaluation, while the experiments support adaptive fusion and thermal information for tracking.

Abstract

from arXiv · show

RGB-Thermal (RGB-T) object tracking receives more and more attention due to the strongly complementary benefits of thermal information to visible data. However, RGB-T research is limited by lacking a comprehensive evaluation platform. In this paper, we propose a large-scale video benchmark dataset for RGB-T tracking.It has three major advantages over existing ones: 1) Its size is sufficiently large for large-scale performance evaluation (total frame number: 234K, maximum frame per sequence: 8K). 2) The alignment between RGB-T sequence pairs is highly accurate, which does not need pre- or post-processing. 3) The occlusion levels are annotated for occlusion-sensitive performance analysis of different tracking algorithms.Moreover, we propose a novel graph-based approach to learn a robust object representation for RGB-T tracking. In particular, the tracked object is represented with a graph with image patches as nodes. This graph including graph structure, node weights and edge weights is dynamically learned in a unified ADMM (alternating direction method of multipliers)-based optimization framework, in which the modality weights are also incorporated for adaptive fusion of multiple source data.Extensive experiments on the large-scale dataset are executed to demonstrate the effectiveness of the proposed tracker against other state-of-the-art tracking methods. We also provide new insights and potential research directions to the field of RGB-T object tracking.

I. INTRODUCTION

RGB-T tracking combines complementary visible and thermal information to address difficult imaging conditions, but progress is constrained by limited benchmarks. The paper introduces RGBT234 and a graph-based tracker that jointly learns patch structure and modality reliability.

  • RGB-T data combines thermal robustness under poor illumination and weather with RGB advantages during thermal crossover and through glass.
  • RGBT234 contains 234 paired videos, about 234K total frames, sequences up to 8K frames, annotations, baseline trackers, and five evaluation metrics.
  • RGBT234 provides highly accurate cross-modal alignment without pre- or post-processing and annotates no, partial, and heavy occlusion levels.
  • The tracker partitions target boxes into patches, weights patches to suppress background pollution, and uses graph-based RGB-T feature learning.
  • Graph structure, edge weights, node weights, sparse representations, and modality weights are jointly learned in one ADMM-based optimization framework.
  • Experiments evaluate the proposed approach extensively on the benchmark and report its effectiveness alongside new insights and future research directions.

II. RELATED WORK

Prior RGB-T tracking research used small or biased datasets and explored sparse, structured, and deep tracking approaches. The paper responds with the larger RGBT234 benchmark and a graph-learning RGB-T tracker.

  • RGB-T tracking datasets such as OSU-CT, LITIV, and GTOT have limited size, diversity, or coverage for comprehensive evaluation.
  • RGBT234 expands prior benchmark resources with 234 video pairs, challenging videos, baseline algorithms, attributes, and evaluation metrics.
  • Earlier RGB-T methods concatenate or fuse modality representations, while modality weighting addresses perturbation or malfunction in individual sources.
  • Patch weighting within structured SVM tracking suppresses background effects, and the paper extends this line with graph learning for RGB-T tracking.

1) Structured SVM based Trackers:

The benchmark is built from paired RGB-T videos captured and annotated to support aligned, attribute-sensitive tracking evaluation. Its surrounding methods emphasize patch- and graph-based representation learning.

  • The dataset section covers imaging hardware, dataset statistics, annotations, baseline RGB and RGB-T trackers, and evaluation metrics.
  • The collection contains 234 RGB-T videos, each pairing one RGB video with one thermal video, for large-scale evaluation.
  • The imaging system uses a turnable platform and two cameras whose optical axes are aligned in parallel with a collimator.
  • Annotations are produced in two stages using more reliable source data first, then adjusting minimum bounding boxes in the less reliable modality.
  • Each sequence receives attributes for attribute-sensitive analysis, and targets are annotated in every frame, including full occlusions.

3) Advantages over existing Datasets:

RGBT234 is designed for broad, modality-aware evaluation through aligned annotations, challenging attributes, multiple tracker families, and complementary metrics.

  • RGBT234 compares RGB, thermal, and RGB-T datasets while addressing the environmental weaknesses of single-modality benchmarks.
  • Attribute annotations and their distribution support performance analysis across visual challenges represented in the dataset.
  • The benchmark contains about 234K annotated frames, sequences up to 8K frames, accurate alignment without preprocessing, and occlusion labels.
  • The evaluation includes RGB trackers, RGB-T trackers, deep and non-deep baselines, and modality-specific implementations for comprehensive comparison.
  • Five metrics measure location precision, overlap success, accuracy, failures, and expected average overlap.

IV. PROPOSED APPROACH

The proposed approach jointly learns patch structure, patch weights, sparse representations, and modality reliability to build a robust RGB-T object descriptor. It uses dynamically learned graphs rather than fixed neighborhood structures and optimizes the components in a unified framework.

  • Joint optimization: The final model jointly optimizes modality weights, sparse representations, and graph structure, including edge and node weights.This combines the component objectives into one proposed formulation.
  • Feature construction: Each bounding box is partitioned into non-overlapping patches represented by modality-specific feature matrices, with RGB-T treated as the two-modality case.The framework is formulated for M modalities and specializes to RGB-T when M = 2.
  • Dynamic graph learning: Sparse self-representation dynamically learns graph relationships among patches, while noise terms improve robustness to corrupted patches.Sparse constraints select informative higher-order neighbors and encourage shared cross-modality representation patterns.
  • Adaptive modality weighting: Modality weights represent source reliability, assigning larger weights to modalities with smaller sparse reconstruction errors while regularization prevents degenerate solutions.This enables adaptive fusion when individual sources are perturbed or malfunctioning.
  • Graph affinity: The learned affinity matrix encodes same-class probabilities from joint sparse-representation distances, with constraints enforcing valid affinities and discouraging overfitting.The formulation replaces direct use of sparse coefficients with a learned graph affinity.
  • Patch weighting: Patch weights are computed semi-supervised from a shrunk-box foreground initialization using graph smoothness and fitting constraints.The initial vector assigns foreground labels to patches inside the shrunk bounding-box region.

2) Optimization:

An ADMM algorithm efficiently optimizes the proposed non-convex model by solving convex subproblems with closed-form solutions. The patch-weight transformation uses a fixed sigmoid parameter.

  • Optimization: ADMM optimizes the model efficiently, with each variable's subproblem convex and admitting a closed-form solution.The resulting limit point satisfies Nash equilibrium conditions.
  • Optimization: The final weight of patch i is computed as ˆs_i = 1/(1 + exp (−σs_i)), with σ fixed to 37.The tracking performance is reported as insensitive to parameter variations.

1) Feature Representation:

The tracker combines weighted patch features and modality reliability into a bounding-box descriptor, then uses structured SVM for target localization and model updating. Structured constraints use IoU-based margins and confidence thresholding to reduce false-labeling effects.

  • Feature Representation: Patch weights suppress background effects, while modality weights adaptively integrate RGB and thermal source data into the target descriptor.The corresponding weighted patch features form the final bounding-box representation.
  • Translation estimation: Structured SVM selects the target bounding box from candidates sampled within a search window around the previous location.The classifier combines the learned object representation with structured prediction.
  • Translation estimation: The structured constraint requires the target box to score above nearby boxes by a margin determined by their IoU overlap.The margin is Δ(y_t, y) = 1 − IoU(y_t, y).
  • Translation estimation: The classifier is updated only when tracking confidence exceeds a threshold, with confidence defined by average similarity to positive support vectors.This update rule limits the effects of unreliable tracking results.

4) Scale estimation:

Scale estimation evaluates a pyramid of target regions centered at the estimated translation and selects the scale with the highest classification score. The evaluation section also reports experiments across 14 algorithms and analyzes the proposed approach's scale limitation.

  • Scale estimation: A target pyramid samples regions of multiple scales around the estimated translation location.Each region has size bW × bH and is resized to W × H before scoring.
  • Scale estimation: The optimal target scale is selected by maximizing classification scores over the resized image regions.This extends translation estimation with explicit scale search.
  • Evaluation: The evaluation reports results for 14 tracking algorithms on RGBT234 and analyzes challenging factors and limitations.The source code, dataset, baseline trackers, and evaluation results are stated to be available for free academic usage.

A. Parameter Settings

The evaluation fixes experimental settings and compares SGT with RGB-T and deep-feature trackers using multiple overall and attribute-based metrics on RGBT234.

  • A. Parameter Settings: 64 non-overlapping patches balance tracking accuracy and efficiency, with color and gradient histograms extracted for each patch.Thermal patches use gray values; frames are scaled so the bounding-box minimum side is 32 pixels.
  • A. Parameter Settings: The evaluation reports maximum success and precision plots alongside Accuracy, Robustness, and Expected Average Overlap.Table V contains the three additional metrics, while Fig. 6 presents MPR and MSR plots.
  • A. Parameter Settings: SGT performs best in MPR and EAO but trails CSR-DCF+RGBT in MSR by 1.8%.Its lower MSR and Accuracy indicate difficulty handling scale variations.
  • A. Parameter Settings: SGT, SOWP+RGBT, and CSR-DCF+RGBT rank among the top three RGB-T methods and significantly outperform the other evaluated methods.The comparison attributes gains to weighted patch representations, segmentation, and adaptive RGB-thermal fusion.
  • A. Parameter Settings: For partial occlusion, SGT and SOWP+RGBT perform best in MPR, whereas heavy occlusion causes larger performance decreases for these methods.The authors associate the heavy-occlusion drop with unreliable initial seeds and inaccurate background-suppression weights.

2) Low Illumination:

The attribute evaluation highlights thermal data under low illumination, weighted patch representations under deformation, and complementary tracker behavior under thermal crossover and fast motion.

  • 2) Low Illumination:: Thermal information is critical under low illumination because visible-spectrum imaging quality becomes poor.SGT, SOWP+RGBT, and CSR-DCF+RGBT significantly outperform several RGB-only and other comparison methods in this condition.
  • 3) Low Resolution:: SGT achieves superior MPR over C-COT and ECO for very low-resolution objects.The passage identifies non-deep features as a potentially better direction for low-resolution tracking.
  • 5) Thermal Crossover:: C-COT, ECO, and SGT rank in the top three on thermal crossover, while SOWP outperforms SOWP+RGBT.The comparison suggests direct RGB-thermal fusion can be ineffective when one source is noisy.
  • 1) Occlusion:: Weighted patch representations alleviate background effects during partial occlusion, but heavy occlusion makes their initial seeds unreliable.Under heavy occlusion, inaccurate patch weights can weaken background suppression.
  • 6) Fast Motion:: Structured SVM-based SGT benefits from a flexible search window on fast motion, outperforming ECO and C-COT in the reported comparison.The passage links fast-motion performance to tracker search strategies.

7) Scale Variation:

Across scale variation and related attributes, the evaluation favors methods with strong appearance discrimination, weighted representations, and modality-aware handling, while identifying direct fusion and scale handling as limitations.

  • 7) Scale Variation:: Correlation-filter and deep-learning methods handle scale variations well through discriminative representations of slightly different appearances.ECO, C-COT, CSR-DCF+RGBT, and SRDCF are cited as examples.
  • 9) Motion Blur:: Deep-learning and weighted-patch trackers including ECO, C-COT, SGT, and SOWP-RGBT excel in motion-blur MPR and MSR.The passage emphasizes feature quality as critical under appearance degradation.
  • 8) Camera Moving:: SGT adapts well to camera movement, while CFnet+RGBT provides little gain over CFnet in that condition.Camera movement is described as introducing motion variation and blur.
  • 10) Background Clutter:: SGT and SOWP+RGBT outperform other trackers on background clutter with a clear margin.The authors link this outcome to weighted patches and thermal information mitigating background effects.
  • VI. CONCLUDING REMARK: The evaluation identifies thermal data, adaptive fusion, background suppression, and powerful feature representations as important RGB-T tracking components.It also notes that direct RGB-thermal integration can sometimes be ineffective and that non-deep features may suit low-resolution objects.
Loading 1805.08982v1…