PaperlayerView exact revision on arXiv

Source-linked AI summary

TFDet: Target-Aware Fusion for RGB-T Pedestrian Detection

Xue Zhang, Xiaohan Zhang, Jiangtao Wang, Jiacheng Ying, Zehua Sheng, Heng Yu, Chunguang Li, Hui-Liang Shen

arXiv:2305.16580v4cs.CV

TL;DR

RGB-only pedestrian detection can degrade under low-light conditions, and multispectral fusion can still produce noisy features associated with false positives. TFDet uses target-aware fusion and feature refinement; it reports state-of-the-art performance across pedestrian and multispectral object benchmarks, with comparable computational efficiency.

  • Problem

    RGB-only detection degrades in low light, while prior multispectral fusion can produce noisy features associated with false positives and performance degradation.

    Source: §I · p. 1

  • Method

    TFDet uses target-aware fusion with FFM to collect complementary features, FRM to distinguish pedestrian and background features, and correlation-maximum loss to enhance feature contrast.

    Source: §V · p. 12

  • Results

    TFDet achieves state-of-the-art performance on two multispectral pedestrian and two multispectral object benchmarks, with greater superiority under low-light conditions and comparable computational efficiency.

    Source: §I · p. 2

  • Takeaways & Limitations

    The strategy can be applied to one-stage and two-stage detectors and extended to multispectral multiclass object detection.

    Source: §I · p. 2

Abstract

from arXiv · show

Pedestrian detection plays a critical role in computer vision as it contributes to ensuring traffic safety. Existing methods that rely solely on RGB images suffer from performance degradation under low-light conditions due to the lack of useful information. To address this issue, recent multispectral detection approaches have combined thermal images to provide complementary information and have obtained enhanced performances. Nevertheless, few approaches focus on the negative effects of false positives caused by noisy fused feature maps. Different from them, we comprehensively analyze the impacts of false positives on the detection performance and find that enhancing feature contrast can significantly reduce these false positives. In this paper, we propose a novel target-aware fusion strategy for multispectral pedestrian detection, named TFDet. TFDet achieves state-of-the-art performance on two multispectral pedestrian benchmarks, KAIST and LLVIP. TFDet can easily extend to multi-class object detection scenarios. It outperforms the previous best approaches on two multispectral object detection benchmarks, FLIR and M3FD. Importantly, TFDet has comparable inference efficiency to the previous approaches, and has remarkably good detection performance even under low-light conditions, which is a significant advancement for ensuring road safety.

I. INTRODUCTION

RGB-only detection degrades in low light, while multispectral fusion combines complementary information but can leave noisy features that lead to false positives. TFDet introduces target-aware fusion to refine target and non-target features and reports state-of-the-art performance across four datasets.

I. INTRODUCTION

  • Prior fusion methods combine complementary modalities but may neglect target-versus-non-target distinction, producing noisy features that lead to false positives and degraded performance.The introduction notes that the previous studies largely focus on combining complementary features from both modalities.Source: §I · p. 1
  • The strategy is presented as flexible across one-stage and two-stage detectors and extensible to multiclass object detection.The introduction reports state-of-the-art performance on two multispectral pedestrian and two multispectral object detection benchmarks.Source: §I · p. 2
  • TFDet’s target-aware strategy combines multispectral features and refines representations to enhance target regions while suppressing non-target regions.The paper also describes a feature refinement module and a feature contrast enhancement unit.Source: §I · pp. 1–2
  • TFDet achieves state-of-the-art performance on four challenging datasets.The introduction also reports inference time comparable to previous state-of-the-art counterparts.Source: §I · p. 2

II. RELATED WORK

  • The related-work section reviews general object detection, pedestrian detection, and multispectral pedestrian detection.Source: §II · p. 2

A. Object Detection and Pedestrian Detection

This section reviews object-detection designs and multispectral pedestrian fusion, then introduces TFDet’s modules and their empirical-analysis approach. It frames noisy fused features and background false positives as a gap in prior work.

A. Object Detection and Pedestrian Detection

  • One-stage detectors predict boxes directly, whereas two-stage detectors use proposals in a coarse-to-fine process; the former tend to be faster and the latter more accurate.Faster R-CNN is given as a two-stage example, with an RPN generating proposals for R-CNN.Source: A. Object Detection and Pedestrian… · p. 2

B. Multispectral Pedestrian Detection

  • Multispectral pedestrian detection fuses RGB and thermal data, but prior approaches often overlook noisy fused features and their potential to generate background false positives.The section reviews illumination-aware, attention-based, and non-local fusion, as well as box-level-mask approaches.Source: B. Multispectral Pedestrian Detection · p. 3

III. METHOD

  • The method section introduces FFM, FRM, and correlation-maximum loss, and studies module roles empirically because rigorous theoretical analysis of learned representations remains challenging.It describes FFM’s effect after its structure and examines FRM and the loss together because they complement one another in contrast enhancement during training.Source: §III · p. 3

A. Motivation

The motivation is that background false positives persist despite pedestrian recall, and removing certain false positives improves detection performance. TFDet addresses noisy features through target-aware fusion and contrast enhancement grounded in mask-feature correlation.

A. Motivation

  • Removing false positives progressively lowers log-average miss rate, with the sharpest decreases after removing high-confidence or low-IoU false positives.The authors suggest high-score false positives may be concentrated in background regions and be more detrimental.Source: A. Motivation · p. 3
  • Shape similarity between background objects and pedestrians can confuse the model, producing noisy features and false positives.The motivating example has a correct target-region prediction alongside three background false positives.Source: A. Motivation · pp. 3–4
  • TFDet combines complementary features with target/non-target refinement and contrast-enhancement loss; its t-SNE visualization shows tighter same-class clusters and separation between classes.The strategy is reported to significantly reduce false positives.Source: A. Motivation · p. 4

B. Foundation of Feature Contrast Enhancement

  • The contrast-enhancement foundation correlates a noisy fused feature with a box-level mask, assigning ones inside ground-truth boxes and zeros elsewhere; the mask is predicted because ground-truth masks are unavailable at test time.The predicted mask and each normalized feature channel are compared by cosine similarity, with inner products indicating similarity.Source: B. Foundation of Feature Contrast… · p. 4
  • A projection matrix transforms channel correlations to preserve semantic information, and the correlation-maximum loss uses segmentation and negative-correlation losses to supervise the predicted mask and processed correlations.Each processed correlation represents mask similarity to a weighted average of fused features and is encouraged to approach 1.0.Source: B. Foundation of Feature Contrast… · p. 4

C. Architecture of TFDet

TFDet processes paired RGB and thermal images through a backbone, target-aware fusion in the neck, and detection heads. The design supports different detector heads and uses a weight-sharing backbone to save computation and parameters.

C. Architecture of TFDet

  • The backbone extracts paired RGB and thermal features, the neck’s fusion strategy produces fused features, and the head predicts boxes and scores.The architecture comprises backbone, neck, and head; the neck first applies FFM and then FRM.Source: C. Architecture of TFDet · p. 4, Figure 5 · p. 5
  • The discriminative fused feature feeds either a one-stage or two-stage detection head, while the correlation-maximum loss is used only during training.Examples include a YOLO head or a Faster R-CNN head with RPN and R-CNN.Source: C. Architecture of TFDet · p. 5, Figure 5 · p. 5
  • The fusion strategy is independent of CNN structure and is applied at different pyramid levels; the chosen weight-sharing CNN backbone saves computation and parameters, helping mitigate potential overfitting.The detailed description shows one pyramid level because the fusion process is identical across levels.Source: C. Architecture of TFDet · p. 5

D. Feature Fusion Module (FFM)

The FFM uses parallel- and cross-channel similarities in paired multispectral features to guide fusion. Its design combines corresponding channels adaptively, then recalibrates features across channels.

D. Feature Fusion Module (FFM)

  • Diagonal entries indicate strong same-position similarity, while large off-diagonal entries indicate similarity across different channels.ANR and AIR both exceed 1 on KAIST and LLVIP, indicating parallel-channel similarity; ANR is consistently smaller than AIR, implying cross-channel similarity.Source: D. Feature Fusion Module (FFM) · pp. 5–6
  • ANR and AIR compare diagonal entries with off-diagonal means and medians, respectively, to summarize differences in feature relations.The median denominator avoids influence from outliers.Source: D. Feature Fusion Module (FFM) · pp. 5–6
  • The FFM first fuses corresponding channels with modified deformable convolution, riffle shuffle, and group-wise convolution, then uses GlobalCC to recalibrate channel features.The deformable convolution adaptively learns receptive fields; GlobalCC constructs channel interdependencies and applies modulation weights to produce the initially fused feature.Source: D. Feature Fusion Module (FFM) · p. 6
  • The FFM with Adaptive-RP and GlobalCC outperforms variants using Fixed-RP or ConvCC, respectively.The supplied prose attributes GlobalCC’s advantage to capturing a global receptive field through global average pooling, unlike ConvCC’s fixed-sized kernel.Source: D. Feature Fusion Module (FFM) · p. 6, Table II · p. 6

E. Feature Refinement Module (FRM)

The FRM refines initially fused features to increase contrast between foreground and background. It predicts a box-level mask, uses its correlation with fused features to refine them, and supervises the mask and correlation.

E. Feature Refinement Module (FRM)

  • The FRM takes initially fused features Fx as input and outputs contrast-enhanced features Fy.It is designed to enhance contrast between foreground and background features.Source: E. Feature Refinement Module (FRM) · p. 6
  • A segmentation branch predicts a box-level mask from Fx, using a mask label formed by filling annotated foreground boxes with 1s and other areas with 0s.The predicted mask is supervised during refinement.Source: E. Feature Refinement Module (FRM) · p. 6
  • The FRM correlates the predicted mask with Fx, projects that correlation, and uses it to scale Fx into the refined feature Fy.Channel-wise product performs the scaling; the process supervises the predicted mask and maximizes the processed correlation.Source: E. Feature Refinement Module (FRM) · p. 6
  • For multi-class detection, the FRM constructs its mask label by marking all target regions as 1 and background as 0.The paper states that the same process can then enhance feature contrast in this setting.Source: E. Feature Refinement Module (FRM) · p. 6

F. Correlation-Maximum Loss Function

The correlation-maximum loss combines mask segmentation supervision with a negative-correlation term that encourages processed correlations toward 1. The paper reports that FRM and this loss enhance feature contrast and reduce false positives across datasets.

F. Correlation-Maximum Loss Function

  • The loss is intended to make the predicted box-level mask accurate and the processed correlation between that mask and the initially fused feature high.The segmentation component uses binary cross-entropy and Dice losses.Source: F. Correlation-Maximum Loss Function · p. 7
  • Lcorr max combines segmentation loss for predicted-mask accuracy with negative-correlation loss to raise processed-correlation entries toward 1.The loss is applied alongside final detection classification and regression losses during training.Source: F. Correlation-Maximum Loss Function · p. 7
  • The negative-correlation loss is constructed in the log domain because the logarithm often gives larger gradients when correlation entries are near zero than l1- or l2-based losses.The paper combines correlation-maximum losses from multiple CNN stages with final detection losses.Source: F. Correlation-Maximum Loss Function · p. 7
  • The method with FRM and Lcorr max significantly reduces false positives across different datasets.The comparison reports that the variant without them produces noisy feature maps and false positives; Fig. 10 counts false positives across entire datasets.Source: F. Correlation-Maximum Loss Function · pp. 7–8, Figure 10 · p. 8

IV. EXPERIMENTS

The experiments evaluate TFDet on four multispectral detection datasets, covering pedestrian and multi-class object detection, and compare detection performance and inference time with prior state-of-the-art approaches.

IV. EXPERIMENTS

  • The experiments cover four multispectral datasets: two for pedestrian detection and two for multi-class object detection.The evaluation also compares detection performance and inference time with prior state-of-the-art approaches and includes ablation studies.Source: §IV · p. 8

A. Datasets

The experiments use four aligned multispectral benchmarks, covering pedestrian detection on KAIST and LLVIP and multi-class object detection on FLIR and M3FD. Detector, backbone, training, and evaluation choices vary by benchmark, with prior-work settings used where specified.

A. Datasets

  • KAIST provides 7,601 training and 2,252 validation RGB-thermal pairs; LLVIP provides 12,025 training and 3,463 validation pairs and is captured under low-light conditions.Image resolutions are 512 × 640 for KAIST and 1,024 × 1,280 for LLVIP.Source: A. Datasets · p. 8
  • FLIR uses an updated aligned three-category dataset with 4,129 training and 1,013 validation pairs, while M3FD has six categories and 2,905 training and 1,295 validation pairs.M3FD uses a recent-work split and images are resized and padded to 640 × 640.Source: A. Datasets · p. 8

B. Implementation Details

  • Experiments use Faster R-CNN and YOLOv5 as two-stage and one-stage detectors, respectively, with prior-work detector, backbone, and metric settings adopted for fair comparison.The KAIST comparison uses MR versus FPPI curves and qualitative scenes mark ground-truth boxes in green and predictions in red.Source: B. Implementation Details · p. 8, Figure 11 · p. 9, Figure 12 · p. 10
  • KAIST uses Faster R-CNN with VGG-16 and log-average miss rate; the comparison table reports MR across scene and distance subsets.Lower MR indicates better detection performance; the scene subsets are All-Day, Day, and Night, and distance subsets are Near, Medium, and Far.Source: B. Implementation Details · p. 8, Table III · p. 9
  • LLVIP evaluates Faster R-CNN with ResNet-50 and YOLOv5-Large using AP and AP50, with YOLOv5 trained for 100 epochs.AP averages ten IoU thresholds from 0.50 to 0.95; higher AP indicates better performance.Source: B. Implementation Details · p. 8
  • FLIR uses Faster R-CNN with Swin-Transformer-Tiny and AdamW, while M3FD uses YOLOv5-Small trained for 36 epochs and reimplements comparison methods on the same split.FLIR uses AP and random flipping and cropping; M3FD image-fusion methods are evaluated by training on their generated fused images under the same settings.Source: B. Implementation Details · p. 10

C. Comparison on the KAIST Dataset

On KAIST, TFDet outperforms the compared single-modality and multispectral methods, with notable results on the Night subset. Its miss-rate curves and qualitative examples also indicate fewer missed pedestrians and unwanted detections in the reported comparisons.

C. Comparison on the KAIST Dataset

  • TFDet achieves the best KAIST detection performance, gaining 9.46% over the best single-modality approach and 0.65% over multispectral state-of-the-art LG-FAPF.The paper reports an MR below 5% after 12 training epochs.Source: C. Comparison on the KAIST Dataset · p. 10
  • TFDet shows particular superiority on the Night subset, where RGB images are generally of poor visual quality.The paper also identifies an MR below 5% with 12 epochs of training as a first to its knowledge.Source: C. Comparison on the KAIST Dataset · p. 10
  • TFDet misses fewer pedestrians even at low FPPI, corresponding to a high confidence threshold.The comparison uses miss-rate versus FPPI curves on the KAIST test set.Source: C. Comparison on the KAIST Dataset · p. 10
  • In two day and night scenes, TFDet detects all pedestrians while AR-CNN, MBNet, and ProbEn show false positives and missed pedestrians.The reported false positives include background boxes and replicated boxes on pedestrians.Source: C. Comparison on the KAIST Dataset · p. 10

D. Comparison on the LLVIP Dataset

TFDet-YOLOv5 leads the reported LLVIP comparisons, and TFDet also outperforms prior methods on the FLIR and M3FD multi-class benchmarks. The KAIST inference-time comparison reports speed comparable to DCMNet.

D. Comparison on the LLVIP Dataset

  • TFDet-YOLOv5 exceeds the best single-modality approach by 4.1% AP and the best multispectral approach, LRAF-Net, by 4.8% AP on LLVIP.The authors note that thermal-only detection performs well because most LLVIP images are low-light, yet TFDet-YOLOv5 exceeds prior approaches by a large margin.Source: D. Comparison on the LLVIP Dataset · p. 10, Table IV · p. 11

E. Extension to Multi-Class Object Detection Scenarios

F. Inference Time

  • TFDet has inference speed comparable to state-of-the-art counterpart DCMNet in the KAIST comparison.The timing comparison uses Faster R-CNN methods at 512 × 640 resolution.Source: Table VII · p. 12, F. Inference Time · p. 12

G. Ablation Study

The ablations associate lower miss rates with fewer high-confidence false positives as components are added to FFM. The conclusion attributes TFDet’s results to complementary-feature fusion, refinement, and contrast enhancement, and reports performance across four benchmarks.

G. Ablation Study

  • The ablation progressively adds FFM, FRM, Lseg, and Lneg corr; Fig. 13 highlights false-positive scores above 0.3 because these have a more negative impact on detection performance.The study reports MR and false-positive confidence-score counts.Source: G. Ablation Study · p. 12
  • Adding FRM, Lseg, and then all components to FFM lowers MR by 1.75%, 3.16%, and 4.1%, respectively, while reducing false positives by 564, 615, and 664 in the reported comparisons.The full-component comparison is against using FFM alone.Source: G. Ablation Study · p. 12

V. CONCLUSIONS

  • TFDet combines complementary features with FRM-based pedestrian/background distinction and a correlation-maximum loss to enhance feature contrast; visualization results show reduced false positives.The conclusion attributes the strategy’s discriminative features to these modules and loss.Source: §V · p. 12
  • The conclusion reports state-of-the-art performance on four benchmarks and a flexible, computationally efficient strategy applicable to one-stage and two-stage detectors.The authors identify multispectral object tracking as future work.Source: §V · p. 12

Related papers

Loading 2305.16580v4…