PaperlayerView exact revision on arXiv

Source-linked AI summary

Synthetic data generation for end-to-end thermal infrared tracking

Lichao Zhang, Abel Gonzalez-Garcia, Joost van de Weijer, Martin Danelljan, Fahad Shahbaz Khan

arXiv:1806.01013v2cs.CV

TL;DR

TIR tracking lacks the large labeled datasets that support deep feature learning, so handcrafted features remain common. The paper translates labeled RGB tracking data into synthetic TIR sequences, trains end-to-end features, and reports state-of-the-art performance when combining the learned network with motion features.

  • Problem

    TIR tracking lacks large labeled datasets, and most state-of-the-art methods consequently rely on handcrafted features.

  • Method

    The paper uses paired and unpaired image-to-image translation to convert labeled RGB tracking data into synthetic TIR sequences with transferred annotations for end-to-end training.

  • Results

    An end-to-end tracker trained with synthetic TIR data sets a new state-of-the-art on VOT-TIR2017 with an EAO of 0.448, surpassing DSLT’s 0.401.

  • Takeaways & Limitations

    Synthetic labeled TIR data enables end-to-end TIR features that outperform features trained on available real TIR sequences, with further gains from combining both sources.

Abstract

from arXiv · show

The usage of both off-the-shelf and end-to-end trained deep networks have significantly improved performance of visual tracking on RGB videos. However, the lack of large labeled datasets hampers the usage of convolutional neural networks for tracking in thermal infrared (TIR) images. Therefore, most state of the art methods on tracking for TIR data are still based on handcrafted features. To address this problem, we propose to use image-to-image translation models. These models allow us to translate the abundantly available labeled RGB data to synthetic TIR data. We explore both the usage of paired and unpaired image translation models for this purpose. These methods provide us with a large labeled dataset of synthetic TIR sequences, on which we can train end-to-end optimal features for tracking. To the best of our knowledge we are the first to train end-to-end features for TIR tracking. We perform extensive experiments on VOT-TIR2017 dataset. We show that a network trained on a large dataset of synthetic TIR data obtains better performance than one trained on the available real TIR data. Combining both data sources leads to further improvement. In addition, when we combine the network with motion features we outperform the state of the art with a relative gain of over 10%, clearly showing the efficiency of using synthetic data to train end-to-end TIR trackers.

I. INTRODUCTION

TIR tracking is important but constrained by scarce labeled data, leaving handcrafted features dominant. The paper transfers labeled RGB tracking data into synthetic TIR data to enable end-to-end feature learning.

  • TIR tracking supports surveillance, robotics, and defense applications, including operation in darkness and reduced sensitivity to illumination variation.
  • Large labeled datasets are crucial for learning discriminative tracking features, but TIR datasets are small and lack an ImageNet counterpart.
  • Most leading TIR trackers therefore still rely on handcrafted features, including the top three VOT-TIR2017 trackers.
  • The proposed framework translates labeled RGB tracking videos into synthetic TIR videos while transferring their tracking annotations.
  • End-to-end TIR feature training is presented as a contribution, with synthetic-only training outperforming trackers trained on available labeled TIR data.
  • Combining synthetic-data training with motion features achieves state-of-the-art performance on the TIR tracking challenge.

II. RELATED WORK

Prior TIR tracking work mainly extends correlation-filter methods with handcrafted, multimodal, or motion features. Related data-generation studies motivate using synthetic modalities as labeled data for the scarce TIR domain.

  • Discriminative correlation filters learn target-background classifiers from example patches and efficiently exploit spatial shifts through circular correlation.
  • Recent DCF advances address boundary effects, scale estimation, contextual information, and long-term memory.
  • Deep CNN features improve robustness to geometric variation over handcrafted features, largely because they are learned from large RGB datasets.
  • Leading TIR trackers use handcrafted features such as edges, HOG, intensity, and motion, with DSLT winning the VOT-TIR2017 challenge.
  • Other approaches fuse grayscale and TIR inputs, while prior modality-generation work used synthetic data for classification or pretraining rather than as labeled data for scarce-domain tracking.

C. Adversarial image-to-image translation

The paper uses image-to-image translation to address scarce labeled TIR video data, comparing paired and unpaired translation while motivating domain-specific feature learning.

  • GAN-based conditional generation supports image-to-image translation by conditioning synthesis on an input image.
  • pix2pix uses paired training images and combines adversarial and target-matching losses, whereas CycleGAN extends translation to unpaired domains through cycle consistency.
  • Labeled TIR videos are scarce and largely limited to one object class, while many labeled RGB tracking videos are available.
  • RGB-pretrained features are unlikely to be optimal for TIR because the modalities differ substantially.
  • The method transfers labeled RGB datasets into synthetic TIR datasets while retaining tracking annotations for end-to-end feature training.

A. Correlation Filter Tracking

The section describes correlation-filter tracking, including conventional DCFs, end-to-end feature learning, and ECO's continuous multi-channel optimization. These methods learn filters and feature representations from target samples to produce detection scores for tracking.

  • Correlation filters: Conventional DCFs learn linear filters that discriminate target appearance from background using feature-map samples.The target location is predicted by applying the learned filter to a sample feature map.
  • End-to-end feature learning: End-to-end trackers use a two-branch Siamese network with shared convolutional weights and a differentiable correlation-filter layer.One branch computes the optimal filter, which is applied to the other branch to obtain a response map.
  • End-to-end feature learning: The CFNet procedure is applied to TIR tracking because its Fourier-domain correlation-filter layer supports stable and fast network training.The paper uses this procedure to generate synthetic TIR tracking data for training.
  • Efficient Convolution Operators: ECO combines shallow and deep features by learning a multi-channel continuous convolution filter and a dimensionality-reducing projection matrix.The projection matrix and filter are learned jointly in the first frame, then subsequent normal equations are solved efficiently with Conjugate Gradients.
  • Efficient Convolution Operators: ECO predicts target locations with a score operator that interpolates feature samples into a continuous domain before applying the learned projection and filter.Its training samples contain multiple feature layers with varying spatial resolutions, and labels are periodically repeated Gaussian functions centered at the sample location.
  • Efficient Convolution Operators: The ECO objective regularizes the projection and filter while weighting training samples according to a learning-rate parameter.Spatial regularization mitigates periodic repetition, and Gaussian label functions encode desired target scores across spatial locations.

V. GENERATING TIR IMAGES

This section introduces image-to-image translation methods for transferring RGB images into synthetic TIR data.

  • The section compares image-to-image translation methods for transferring RGB data to synthetic TIR data.

A. Image-to-image translation methods

The paper uses paired pix2pix and unpaired CycleGAN models to transform labeled RGB videos into labeled TIR videos. These models use conditional GANs, while pix2pix additionally relies on paired supervision and an L1 loss.

  • Pix2pix requires matching RGB–TIR frames, whereas CycleGAN can train with unpaired RGB and TIR videos.Paired data can come from multispectral datasets such as KAIST; unpaired training permits broader modality combinations.
  • Both translation methods condition GAN generator and discriminator networks on input images to map RGB frames toward the TIR modality.The generator produces samples resembling the target distribution while the discriminator distinguishes real from generated samples.
  • pix2pix: Pix2pix uses a U-Net encoder–decoder generator with skip connections and a convolutional PatchGAN discriminator that classifies local image patches.The PatchGAN design is suited to modifying image textures or styles.
  • pix2pix: Pix2pix conditions both networks on the input image and combines a conditional GAN objective with an L1 loss in a weighted final objective.The L1 term is included to increase output sharpness, while the generator minimizes and discriminator maximizes the final objective.
  • pix2pix: The paper applies pix2pix independently to each video frame and disables dropout at test time because stochastic outputs make videos less stable.

2) Unpaired - CycleGAN:

CycleGAN enables RGB-to-TIR translation without paired examples by enforcing cycle consistency. The evaluation uses unseen KAIST test frames and a three-video test subset for the translation models.

  • Unpaired translation: CycleGAN addresses limited paired data by learning domain mappings whose forward and reverse transformations preserve the original image.The method adds a cycle-consistency loss to the standard generator framework.
  • Unpaired translation: CycleGAN is applied independently to each frame, with dropout removed at test time to produce more stable video outputs.
  • Evaluation: Fig. 4 compares pix2pix and CycleGAN on KAIST test-set frames that were not seen during training.
  • Evaluation: The translation models are tested using a subset of three videos from the official KAIST test set.

B. Datasets

The paper uses paired and unpaired RGB–TIR datasets to train image-translation models and generate synthetic TIR tracking data. Paired translation with pix2pix is selected because it more closely matches real TIR images than CycleGAN.

  • Training data: The paired datasets include KAIST and several pedestrian datasets, totaling 87K aligned image pairs.KAIST is the largest and most relevant paired dataset described.
  • Training data: The unpaired training collection combines RGB tracking sequences with TIR datasets and contains about 230K images, nearly 5× the paired collection.CycleGAN uses this additional unpaired data.
  • Translation quality: Pix2pix achieves an average translation distance of 35.3 versus 69.5 for CycleGAN on a random 10% KAIST test subset.The comparison uses Euclidean distance to TIR ground-truth images.
  • Translation quality: The authors select pix2pix because its paired training signal produces translations that are qualitatively and quantitatively closer to ground-truth TIR images.The choice holds despite CycleGAN having a larger training collection.
  • Translation quality: Synthetic TIR gradient statistics closely follow real TIR statistics, with only small differences at low gradient magnitudes.The comparison is made on the KAIST test set.

VI. EXPERIMENTAL RESULTS

The tracker is trained and evaluated using real TIR datasets, generated TIR data, and the VOT-TIR2017 benchmark. Evaluation follows the benchmark’s accuracy, robustness, EAO, and OPE success measures.

  • Training datasets: Real-data training uses BU-TIV, ASL, and OTCBVS, while generated-data training uses RGB videos from VOT2016, VOT2017, and OTB.The datasets provide annotated real TIR examples and synthetic TIR tracking data.
  • Evaluation benchmark: VOT-TIR2017 contains 25 TIR videos averaging 740 frames per sequence, totaling 13,863 manually annotated frames.Each frame has one bounding box around a particular object instance.
  • Evaluation metrics: The benchmark reports accuracy as predicted-to-ground-truth overlap and robustness as the average number of tracking failures per sequence.These measures are combined into EAO, the primary ranking metric.
  • Evaluation metrics: EAO estimates expected average overlap for a tracker on sequences of a fixed short length and is the main measure used to rank trackers.The evaluation also includes OPE success plots.

C. Implementation details

The experiments use CFNet-style deep features and ECO tracking with multiple CNN backbones, then compare pretrained, real-data, generated-data, and combined fine-tuning.

  • Network architectures: The tracker is evaluated with AlexNet, VGG-M, and ResNet-50 backbones, using reduced network stride to obtain larger feature maps for correlation filters.The stride reduction is applied consistently across trained models.
  • Adding motion features: The table compares tracker variants with and without motion features on VOT-TIR2017 using ResNet-50, with the best results obtained from combining real and generated TIR data.Boldface marks the best results in the table.
  • Training configurations: ECO is the base tracker, with pretrained and real-data models as baselines and a generated-plus-real model trained on both data sources.The combined model uses pix2pix-generated TIR data alongside real TIR data.
  • Network architectures: Fine-tuning improves all base networks, and generated data provides complementary gains beyond real-data fine-tuning, especially for ResNet-50.The authors use ResNet-50 in subsequent experiments.

1 Success plots of OPE

OPE success plots show progressively stronger performance from pretrained to real-data and generated-data fine-tuning, with the best results from jointly using real and generated data. Increasing the synthetic-data share also improves EAO throughout the tested range.

  • Success plots: Under OPE, handcrafted features perform clearly worse than deep features, while pretrained deep features achieve higher success rates at mid-range overlap thresholds.Real-data fine-tuning provides a small additional boost.
  • Success plots: Generated-data fine-tuning improves OPE performance beyond real-data fine-tuning, and simultaneous use of both data types performs best.The same ordering is reported for the success plots shown in Fig. 8.
  • Synthetic-data amount: Increasing the proportion of synthetic TIR data monotonically improves the generated-plus-real tracker’s EAO across the tested percentages.The all-generated-data point represents 90% of the training set and is not saturated.
  • Motion features: Adding motion features substantially improves all tracker models, while generated-data models outperform pretrained and real-only models.The combined generated-and-real model achieves the strongest reported performance.
  • Qualitative results: Improved features learned from generated TIR data support accurate and robust tracking under occlusion, scale change, and out-of-plane rotation.This qualitative comparison contrasts the pretrained ECO baseline with the generated-data tracker.

H. State-of-the-art Comparison

On VOT-TIR2017, the proposed tracker surpasses established TIR and CNN-based trackers, with strong performance across challenging attributes and conditions. Its synthetic-data-trained features support precise localization, though failures remain under tiny targets, clutter, scale change, and heavy occlusion.

  • 0.448 EAO sets a new state-of-the-art on VOT-TIR2017, surpassing DSLT's 0.401.
  • The tracker achieves superior accuracy and second-best robustness, improving robustness over EBT in every video except trees2.
  • Qualitative comparisons show accurate tracking under scale change and occlusion, but failures occur with tiny cluttered targets or severe scale change combined with heavy occlusion.
  • It outperforms state-of-the-art trackers on most challenging attributes, including heavy camera motion, motion change, and occlusion.
  • The approach achieves the best results on 4 of 6 attributes, while EBT and DSLT remain superior for size change.
  • The method combines synthetic and real TIR data to train end-to-end features, then obtains state-of-the-art results with motion features.
Loading 1806.01013v2…