PaperlayerView exact revision on arXiv

Source-linked AI summary

PTB-TIR: A Thermal Infrared Pedestrian Tracking Benchmark

Qiao Liu, Zhenyu He, Xin Li, Yuan Zheng

arXiv:1801.05944v3cs.CV

TL;DR

The paper addresses the absence of a benchmark for fair TIR pedestrian-tracker evaluation and develops a dataset with manual annotations and attribute labels. It evaluates nine public trackers and conducts component-level experiments, finding that feature extraction is most influential while motion and observation effects depend on feature strength.

  • Problem

    Fair comparison of TIR pedestrian trackers is difficult because representative benchmark data are lacking, despite the importance of understanding tracker strengths and weaknesses.

  • Method

    The paper builds a 60-sequence manually annotated dataset, evaluates nine public trackers, and compares feature extractors, motion models, and observation models.

  • Results

    Feature extraction is the most important component; motion and observation models have minor effects with strong features, while stronger observation models help more when features are weak.

  • Takeaways & Limitations

    The benchmark and component findings provide information for understanding TIR tracker performance and guidelines for future research.

Abstract

from arXiv · show

Thermal infrared (TIR) pedestrian tracking is one of the important components among numerous applications of computer vision, which has a major advantage: it can track pedestrians in total darkness. The ability to evaluate the TIR pedestrian tracker fairly, on a benchmark dataset, is significant for the development of this field. However, there is not a benchmark dataset. In this paper, we develop a TIR pedestrian tracking dataset for the TIR pedestrian tracker evaluation. The dataset includes 60 thermal sequences with manual annotations. Each sequence has nine attribute labels for the attribute based evaluation. In addition to the dataset, we carry out the large-scale evaluation experiments on our benchmark dataset using nine publicly available trackers. The experimental results help us understand the strengths and weaknesses of these trackers.In addition, in order to gain more insight into the TIR pedestrian tracker, we divide its functions into three components: feature extractor, motion model, and observation model. Then, we conduct three comparison experiments on our benchmark dataset to validate how each component affects the tracker's performance. The findings of these experiments provide some guidelines for future research. The dataset and evaluation toolkit can be downloaded at {https://github.com/QiaoLiuHit/PTB-TIR_Evaluation_toolkit}.

I. INTRODUCTION

The paper addresses the lack of a sufficiently diverse benchmark for fair TIR pedestrian-tracker evaluation. It constructs a 60-sequence dataset, evaluates nine public trackers, and separately studies how feature, motion, and observation components affect performance.

  • I. INTRODUCTION: Existing TIR datasets are unsuitable for fair tracker comparison because they are small, lack capture diversity, or provide insufficient evaluation support.The paper motivates a representative benchmark to compare tracker strengths and weaknesses fairly.
  • I. INTRODUCTION: Three validation experiments separately examine feature extractors, motion models, and observation models.The experiments compare features, motion models, and observation models to provide insight for future tracker development.
  • I. INTRODUCTION: The proposed benchmark contains 60 manually annotated thermal sequences collected for fair TIR pedestrian-tracking evaluation.The dataset supports evaluation across nine attribute subsets.
  • I. INTRODUCTION: Nine publicly available trackers are evaluated at large scale on the benchmark.This evaluation is intended to characterize the strengths and weaknesses of different trackers.

II. RELATED WORK

Earlier TIR tracking datasets could be used for testing but lacked the scale, diversity, annotations, or tooling needed for standardized fair evaluation. The paper therefore introduces a larger benchmark with 60 annotated sequences.

  • II. RELATED WORK: No standard specialized TIR pedestrian-tracking dataset existed, although several datasets could be used for basic testing.The paper frames the need as a benchmarking and comparison problem rather than a complete absence of usable data.
  • II. RELATED WORK: Existing datasets vary from 5 to 18 usable pedestrian videos or sequences and often use limited scenes, devices, or resolutions.OSU Color-Thermal has 6 videos, Terravic has 11 suitable pedestrian sequences, PDT-ATV has 8 sequences, and BU-TIV has 5 usable videos.
  • II. RELATED WORK: PDT-ATV provides object ground truths without sequence attributes, while BU-TIV lacks tracking evaluation code.These omissions further limit standardized tracker comparison.
  • II. RELATED WORK: The paper collects a large-scale benchmark with 60 annotated sequences to enable fair tracker comparison and evaluation.The benchmark is presented as a remedy for the limitations of the previously used datasets.

B. TIR Pedestrian Tracking Methods

The paper describes TIR trackers through feature extraction, candidate generation, and target selection, while emphasizing that each component can affect final tracking performance. It motivates component-level validation to determine their respective effects.

  • B. TIR Pedestrian Tracking Methods: Feature extractors represent the target using intensity, local descriptors, texture cues, or fused features to improve discrimination under challenges such as occlusion.Examples include dense SIFT and LBP, alongside intensity, edge, ROI, SURF, and HOG features.
  • B. TIR Pedestrian Tracking Methods: Motion models generate target candidates through probabilistic estimation or exhaustive search.Examples include Kalman and particle filters, sliding windows, and radius sliding windows.
  • B. TIR Pedestrian Tracking Methods: Observation models select the tracked target from candidates, commonly by training a binary classifier.The paper cites mean-shift and random-forest approaches as examples of observation models.
  • B. TIR Pedestrian Tracking Methods: TIR pedestrian trackers are divided into feature extractor, motion model, and observation model components.The paper evaluates these components separately because whole-tracker performance does not reveal each component’s effectiveness.
  • B. TIR Pedestrian Tracking Methods: The proposed validation experiments compare alternatives within each component to determine how feature, motion, and observation choices affect tracking results.This component-level analysis is intended to provide guidance for future research.

III. TIR PEDESTRIAN TRACKING BENCHMARK

The benchmark consists of a TIR pedestrian dataset and an evaluation methodology, which together form the proposed tracking benchmark.

  • III. TIR PEDESTRIAN TRACKING BENCHMARK: The benchmark section introduces the TIR pedestrian dataset and then presents the evaluation methodology for TIR pedestrian tracking.These two parts constitute the proposed benchmark.

A. Dataset

PTB-TIR collects 60 manually annotated thermal sequences from diverse devices, scenes, and shooting times, with attribute subsets for challenge-specific evaluation. It uses standard tracking metrics and robustness, success, and speed protocols to compare trackers fairly.

  • Dataset construction: 60 thermal sequences are manually annotated from different devices, scenes, and shooting times to ensure dataset diversity.The dataset combines sequences from multiple existing thermal datasets and online sources.
  • Annotations: Ground truth records each target’s external bounding box using its left-corner point, width, and height.Several first-frame annotations are shown in Fig. 1.
  • Attributes: Nine attribute subsets characterize sequence challenges and reveal a tracker’s ability to handle each corresponding condition.The attributes distribution is illustrated for the entire dataset and a scale-variation subset.
  • Evaluation metrics: Precision and success rate quantify tracking performance, with success-plot area under the curve used to rank trackers.The metrics derive from center location error and bounding-box overlap.

IV. EVALUATION EXPERIMENTS

The benchmark evaluates nine publicly available trackers using precision and success plots under OPE, TRE, and SRE protocols. SRDCF ranks first overall, while the comparisons highlight the importance of background information, scale estimation, and regression-kernel complexity.

  • Robustness evaluation: TRE performance exceeds OPE, whereas SRE performance is lower because initialization errors tend to cause tracker drift.TRE varies initial frames, while SRE varies the initial bounding box.
  • Overall performance: SRDCF ranks first in success plots, exceeding DSST by more than 5% in OPE and SRE and by about 3% in TRE.The authors suggest its superior performance benefits from incorporating more background information.
  • Overall performance: DSST outperforms KCF by more than 10%, indicating the importance of scale estimation for TIR pedestrian tracking.Both trackers are correlation-filter-based methods evaluated with their original parameters.
  • Tracker comparisons: MCFTS achieves a promising result despite deep features learned from RGB images, suggesting potential for improvement with sufficient thermal-image training data.The authors note that MCFTS uses only a simple combination of its deep network and KCF.
  • Tracker comparisons: TGPR performs about 5% better than RR, supporting the authors’ view that a more complex kernel benefits regression-based TIR tracking.TGPR and RR are the Gaussian- and ridge-regression-based trackers, respectively.

B. Attribute-based Evaluation

Nine trackers are evaluated on attribute-specific subsets to assess their ability to handle distinct challenges. SRDCF performs best on almost all subsets, while MCFTS is stronger than DSST on occlusion and out-of-view cases despite DSST’s higher overall performance.

  • Evaluation scope: Attribute-subset performance measures a tracker’s ability to handle the corresponding challenge, with TRE plots reported for eight subsets.OPE and SRE plots are provided in the supplemental material.
  • Attribute-based results: SRDCF achieves the best performance on almost all attribute subsets.The authors attribute this to its handling of the boundary effect caused by cyclic shifts.
  • Attribute-based results: MCFTS outperforms DSST on occlusion and out-of-view subsets, although DSST has higher overall performance than MCFTS.The comparison indicates that tracker rankings vary across specific challenges and overall evaluation.

C. Speed Comparison

The paper compares feature extractors and tracker components through benchmark experiments, finding that feature choice has the largest effect on TIR tracking performance.

  • C. Speed Comparison: The paper evaluates tracker speed using average FPS on TRE results under a fixed PC configuration and reports the comparison in Table IV.The setup uses an Intel i7-6700K CPU, 32G RAM, and a GeForce GTX 1080 GPU.
  • C. Speed Comparison: KCF is fast, DSST exceeds real-time speed, SRDCF reaches half real-time speed despite best precision and success, and deep trackers run at low frame rates.KCF and DSST benefit from Fourier-domain computation, whereas HDT and MCFTS incur high deep-feature extraction costs.
  • A. Feature Extractor: HOG gives ridge regression the best performance in Fig. 5a, while KCF with HOG outperforms KCF with three tested deep features.The results indicate that single-layer features from RGB-trained networks are not well suited to TIR tracking.
  • A. Feature Extractor: Local gradient features improve performance over gray features by about 20%, while LBP achieves the second-best result and remains close to the best performance.Gray features have low discriminative ability because pedestrians can have intensities similar to their backgrounds; texture features become useful as thermal image quality improves.
  • A. Feature Extractor: Feature extractor choice has a major effect on TIR pedestrian tracking, and selecting or developing a strong feature can dramatically improve performance.The comparison across Figs. 5–7 identifies feature extraction as the most important tracker component.

B. Motion Model.

The motion-model experiment compares particle filtering, sliding-window search, and radius sliding-window search under weak and strong features. Motion-model differences are small with strong features, making faster search preferable in that condition.

  • B. Motion Model: The experiment tests three commonly used motion models on a baseline tracker with two different features, using ridge regression as the baseline.The comparison is reported in Fig. 6.
  • B. Motion Model: With a strong HOG feature, the three motion models perform similarly, whereas particle filtering is much worse than sliding-window search with the weak gray feature.The authors attribute the weak-feature degradation to gray features lacking discriminative capacity, which can cause particle-filter drift.
  • B. Motion Model: Different motion models have a minor effect when the feature is strong enough, so a faster strategy such as sliding-window search is more helpful for TIR tracking.The tested models are particle filter, sliding window, and radius sliding window.

C. Observation Model.

The observation-model experiment compares four models with weak and strong features. Stronger observation models help when features are weak, but observation-model differences become minor when features are strong.

  • C. Observation Model: The experiment tests logistic regression, ridge regression, SVM, and structured output SVM using two different features.The four models are ranked according to their classification ability and evaluated in Fig. 7.
  • C. Observation Model: With the weak gray feature, SOSVM performs best and exceeds ridge regression by more than 10%; with HOG, ridge regression performs best against SVM and SOSVM.The observation models have similar performance when HOG is used.
  • C. Observation Model: A strong observation model can improve tracking with weak features, while different observation models have only a minor performance gap when the feature is strong.The experiment therefore shows that observation-model effects depend on feature strength.

VI. CONCLUSION AND FUTURE WORK

The paper concludes that PTB-TIR supports fair TIR pedestrian-tracker evaluation and component-level analysis. Its findings emphasize feature strength, scale estimation, background information, and the conditional value of deep and observation models.

  • VI. CONCLUSION AND FUTURE WORK: The benchmark contains 60 annotated thermal sequences and supports large-scale evaluation of nine publicly available trackers.The dataset was developed specifically for TIR pedestrian-tracker evaluation.
  • VI. CONCLUSION AND FUTURE WORK: Scale estimation and background information are identified as important for tracking, while deep learning-based trackers have potential for superior TIR performance.Scale estimation can greatly improve tracking performance, and background information can enhance model discrimination.
  • VI. CONCLUSION AND FUTURE WORK: Feature extraction is the most important component; motion and observation models have minor effects with strong features, while stronger observation models often help with weak features.The conclusion synthesizes the three component-level validation experiments.
  • VI. CONCLUSION AND FUTURE WORK: Future work will extend the dataset with more thermal sequences and investigate additional challenge factors in TIR pedestrian tracking.This defines the paper’s stated scope boundary and next research direction.
Loading 1801.05944v3…