PaperlayerView exact revision on arXiv

Source-linked AI summary

Weakly Aligned Feature Fusion for Multimodal Object Detection

Lu Zhang, Zhiyong Liu, Xiangyu Zhu, Zhan Song, Xu Yang, Zhen Lei, Hong Qiao

arXiv:2204.09848v1cs.CV

TL;DR

Weak alignment shifts objects across modalities, challenging multimodal feature fusion and detector training. The paper proposes AR-CNN, combining region-feature alignment, RoI jitter, and confidence-aware fusion, alongside KAIST-Paired labeling. It reports effectiveness and robustness across RGB-T and RGB-D detection settings.

  • Problem

    Most multimodal methods assume aligned image pairs, although weak alignment shifts object positions and challenges feature fusion and CNN training.

    Source: §I · pp. 1–2

  • Method

    AR-CNN aligns cross-modal region features, uses RoI jitter to vary shift patterns, and reweights features to emphasize the more reliable modality.

    Source: §I · p. 2, 1) KAIST-Paired Annotation · p. 7

  • Results

    The method is reported effective and robust across 2-D/3-D, RGB-T, and RGB-D datasets, with state-of-the-art results on KAIST and CVC-14 and best robustness on NYUv2 and SL-RGBD.

    Source: §I · p. 2, §VII · p. 13

  • Takeaways & Limitations

    The reported scope spans RGB-T and RGB-D detection, including both 2-D and 3-D tasks.

    Source: §I · p. 2

Abstract

from arXiv · show

To achieve accurate and robust object detection in the real-world scenario, various forms of images are incorporated, such as color, thermal, and depth. However, multimodal data often suffer from the position shift problem, i.e., the image pair is not strictly aligned, making one object has different positions in different modalities. For the deep learning method, this problem makes it difficult to fuse multimodal features and puzzles the convolutional neural network (CNN) training. In this article, we propose a general multimodal detector named aligned region CNN (AR-CNN) to tackle the position shift problem. First, a region feature (RF) alignment module with adjacent similarity constraint is designed to consistently predict the position shift between two modalities and adaptively align the cross-modal RFs. Second, we propose a novel region of interest (RoI) jitter strategy to improve the robustness to unexpected shift patterns. Third, we present a new multimodal feature fusion method that selects the more reliable feature and suppresses the less useful one via feature reweighting. In addition, by locating bounding boxes in both modalities and building their relationships, we provide novel multimodal labeling named KAIST-Paired. Extensive experiments on 2-D and 3-D object detection, RGB-T, and RGB-D datasets demonstrate the effectiveness and robustness of our method.

I. INTRODUCTION

The introduction identifies weak alignment as a challenge for multimodal detection and presents an end-to-end framework to address it. It also introduces modality-specific labeling and reports results across RGB-T and RGB-D detection settings.

I. INTRODUCTION

  • Weak alignment shifts objects between modalities, mismatching features and making shared bounding boxes a source of biased supervision.The introduction links weak alignment to sensor properties, imperfect calibration, and operational disturbance.Source: §I · pp. 1–2
  • Region Feature Alignment predicts shifts and aligns region features end-to-end, while RoI jitter augments sensed-modality regions to address varied shift patterns.The paper also reports improving RFA shift prediction with an adjacent similarity constraint.Source: §I · p. 2
  • For RGB-T detection, the framework adds confidence-aware feature reweighting and KAIST-Paired labels with bounding boxes and relationships for both modalities.The paper also collects a drone-based RGB-T dataset with additional object categories and extends the framework to RGB-D 2-D and 3-D detection.Source: §I · p. 2
  • The method is reported to improve robustness on KAIST, CVC-14, NYUv2, and SL-RGBD, with state-of-the-art results on KAIST and CVC-14.The reported scope includes RGB-T and RGB-D object detection, including 2-D and 3-D tasks.Source: §I · p. 2
  • The paper extends earlier work with an experimental upper bound for defining weak alignment and additional experiments on effectiveness and generalization.It also extends the approach to RGB-Thermal and RGB-Depth detection, including 2-D and 3-D tasks.Source: §I · p. 2

II. RELATED WORK

Related work describes multimodal detection methods and their use of RGB-T and RGB-D data, then identifies weak alignment as a problem not addressed by most approaches. It contrasts calibration and image registration with the paper’s focus on position shifts in weakly aligned pairs.

II. RELATED WORK

  • Multimodal image fusion is described as improving performance and robustness over RGB-based methods, with applications including surveillance, autonomous driving, and robot grasping.The section notes infrared and depth as additional modalities.Source: §II · p. 2

A. Multispectral (RGB-T) Pedestrian Detection

  • RGB-T pedestrian methods include multispectral datasets and CNN fusion approaches, but most assume full alignment, limiting use of weakly aligned datasets such as CVC-14.Prior approaches include two-stream detection, RPN feature fusion, cross-modality learning, and attention-based feature recalibration.Source: A. Multispectral (RGB-T) Pedestrian… · pp. 2–3

B. RGB-D Object Detection

  • RGB-D work ranges from HOG/HOD fusion and CNN-based 2-D detection to methods that infer 3-D boxes, whereas this paper considers position shifts in RGB-D pairs.The section names Amodal3Det and 3D-SSD among prior 3-D approaches.Source: B. RGB-D Object Detection · p. 3

C. Weakly Aligned Image Pair

  • Weakly aligned pairs can be addressed through calibration or image registration, but recalibration is difficult and image-level registration can be time-consuming and challenging across modalities.The figure examples show that shared boxes may not precisely locate objects in same-position crops; the section also notes differing shift patterns and unpaired objects.Source: C. Weakly Aligned Image Pair · p. 3, Figure 2 · p. 3

III. MOTIVATION

The section characterizes weak alignment in RGB-T and RGB-D data, then tests its detection impact. It describes spatial and annotation mismatches and finds KAIST MR sensitive to simulated color-image shifts.

A. The Position Shift Problem

B. How the Position Shift Impacts?

  • At (−6, 6), KAIST MR rose from 15.2 to 25.1 (about 65% relative degradation), while (1, −1) reached 14.7 MR.The experiment manually simulated 169 shift patterns on KAIST; the surface plot maps color-image shifts to MR, with lower values better.Source: B. How the Position Shift Impacts? · p. 4, Figure 5 · p. 4

IV. OUR METHOD

AR-CNN integrates region-level alignment into multimodal detection, using paired proposals and aligned regional features in its detection pipeline. Its RFA module predicts shifts region by region and uses adjacent similarity to stabilize training.

IV. OUR METHOD

  • AR-CNN aggregates both modalities to generate proposals, predicts regional shifts, re-pools aligned sensed features, and then fuses and classifies the regions.The reference modality stays fixed while alignment is performed on the sensed modality.Source: A. Region Feature Alignment · p. 5

A. Region Feature Alignment

  • RFA enlarges each proposal for context, pools each modality into region features, concatenates them, and uses two fully connected layers to predict x- and y-shifts.The sensed region is re-pooled at its new coordinates to obtain aligned features.Source: A. Region Feature Alignment · p. 5, Figure 7 · p. 6
  • KAIST-Paired bounding boxes provide ground-truth shift targets, while the multi-task loss combines classification, shift, and adjacent-similarity terms.The reference modality is fixed during training, and feature alignment is performed on the sensed modality.Source: A. Region Feature Alignment · pp. 5–6
  • The adjacent similarity constraint assigns the same shift target to an RoI and a sampled neighboring region, encouraging similar predictions where natural-image shifts are spatially smooth.The paper describes this constraint as stabilizing RFA training.Source: A. Region Feature Alignment · p. 5

B. RoI Jitter Strategy

RoI jitter augments sensed-modality proposals during training to expose RFA to shift patterns beyond those represented in the training data.

B. RoI Jitter Strategy

  • RoI jitter randomly perturbs sensed-modality proposals and enriches the RFA shift targets to address unexpected practical shift patterns.The method is intended to bridge the difference between offline training patterns and online testing.Source: B. RoI Jitter Strategy · p. 6
  • Jitter targets are drawn from a Gaussian distribution, with σ denoting the jitter extent, and are used to move the sensed RoI.The RoI is jittered using the inverse process of Equation 1.Source: B. RoI Jitter Strategy · p. 6
  • Minibatch positive and negative samples are selected by IoU using the reference RoI because random jitter is applied to the sensed RoI.Source: B. RoI Jitter Strategy · p. 6

1) KAIST-Paired Annotation:

KAIST-Paired records pedestrian locations and relationships across color and thermal images, including pedestrians visible in only one modality. Confidence-aware fusion reweights modality features according to their reliability.

1) KAIST-Paired Annotation:

  • KAIST-Paired locates pedestrians in both modalities, assigns unique indexes to represent cross-modal relationships, and labels pedestrians appearing in only one modality as unpaired.Source: 1) KAIST-Paired Annotation · p. 6
  • Over half of pedestrians in the original KAIST dataset have position shifts, mostly ranging from 0 to 10 pixels.Source: 1) KAIST-Paired Annotation · p. 7

2) Confidence-Aware Fusion:

  • Confidence-aware fusion reweights features to emphasize the more reliable modality and suppress the less useful one as color and thermal information quality varies across conditions.The method uses a classification branch for each modality to calculate its confidence weight.Source: 1) KAIST-Paired Annotation · p. 7
  • For unpaired objects, a disagreement weight suppresses the sensed feature when its prediction contradicts the reference modality.Source: 1) KAIST-Paired Annotation · p. 7, 2) Confidence-Aware Fusion · p. 7

D. 3D RGB-D Object Detection

The RGB-D 3D detector starts from 2D proposals, initializes corresponding 3D boxes, and regresses their parameters.

D. 3D RGB-D Object Detection

  • RGB and depth jointly generate 2D proposals through an adapted Region Proposal Network.Source: D. 3D RGB-D Object Detection · p. 7
  • Each 3D box is represented by a seven-entry vector for its camera-frame centroid, dimensions, and gravity-axis orientation.The angle θ is measured around the gravity direction.Source: D. 3D RGB-D Object Detection · p. 7
  • The median depth within each proposal initializes z; the proposal center and camera intrinsics determine the initial x and y coordinates.Source: D. 3D RGB-D Object Detection · p. 7
  • A seven-target regression loss learns offsets from initialized boxes to target 3D boxes.The targets include offsets for position, dimensions, and orientation.Source: D. 3D RGB-D Object Detection · p. 8

V. EXPERIMENTS ON CASE 1: RGB-T OBJECT DETECTION

The RGB-T experiments evaluate pedestrian detection on KAIST and CVC-14, then test generalization on a four-category drone dataset. AR-CNN reports favorable comparisons across these datasets.

A. Dataset

  • The experiments cover KAIST and CVC-14 pedestrian detection, plus a drone-based dataset with car, truck, bus, and cyclist categories.The drone set contains 831 image pairs and 6,763 pairs of dense annotations.Source: §V · p. 8, A. Dataset · p. 8

B. Implementation Details

  • The study uses ImageNet-pretrained VGG-16, RoI jitter parameters σ0=σ1=0.05 by default, and MR/MRC/MRT or mAP according to dataset and modality.Source: B. Implementation Details · p. 8

C. Comparison Experiments

  • AR-CNN reports 9.79, 8.07, and 9.03 MR on KAIST reasonable day, night, and all-day subsets, and 8.52 vs. 11.28 MRC and 8.01 vs. 12.51 MRT.The paper reports state-of-the-art performance on the reasonable subsets and significant advantages with KAIST-Paired evaluation.Source: C. Comparison Experiments · p. 8
  • On CVC-14, AR-CNN achieves the best performance among compared methods, including 18.1 vs. 30.8 MR on the night subset.The authors relate the larger night-subset advantage to thermal modality and handling position shift.Source: C. Comparison Experiments · p. 8
  • On the drone-based dataset, AR-CNN achieves the best reported result, 36.4 mAP, among the available competitors.Source: C. Comparison Experiments · p. 8

D. Robustness to Position Shift

The section defines a weak-alignment bound from performance degradation and evaluates robustness across position shifts and reference-modality choices.

D. Robustness to Position Shift

  • The weak-alignment bound is set where shifting causes 50% performance degradation, using MR for pedestrian detection and mAP for object detection.The bound is calculated using the Halfway Fusion model.Source: D. Robustness to Position Shift · p. 8
  • With thermal as reference, AR-CNN achieves the best mean and smallest standard deviation across four shift-direction metrics, while also improving performance at the origin.The reported metrics cover ten shift modes between the origin and the weakly aligned bound for each direction.Source: Table III · p. 9, D. Robustness to Position Shift · p. 10
  • With color as reference, AR-CNN still has the best performance and smallest standard deviation, though this configuration performs below the thermal-reference setting.Source: Table IV · p. 10, D. Robustness to Position Shift · p. 10

E. Ablation Study

The ablations distinguish the effects of region alignment, RoI jitter, and confidence-aware fusion; the section also describes failure patterns and RGB-D evaluation scope.

E. Ablation Study

  • RFA reduces miss rate and shift-condition variability; for S45°, it reduces σ from 8.92 to 2.08, while aligned sensed proposals illustrate its region-wise adjustment.The authors state that this adjustment enables modality-aligned feature fusion for classification and localization.Source: E. Ablation Study · p. 10, Figure 11 · p. 11
  • RoI jitter improves shift robustness: for S0°, µ drops from 12.20 to 11.74 and σ from 2.92 to 1.29.The authors report a larger contribution to σ than to miss rate at the origin, and stable performance across tested hyperparameters.Source: E. Ablation Study · p. 10
  • Confidence-aware fusion reduces origin MRT by 1.61 while slightly suppressing σ, with its main contribution attributed to weighting reliable features.Source: E. Ablation Study · p. 10
  • Far-scale tiny pedestrians are easier to miss after pooling leaves insufficient features; clutter and clustered objects can also degrade detection through disturbed or incomplete features.The authors identify improving occluded-person classification and localization as future work.Source: E. Ablation Study · pp. 10–11

VI. EXPERIMENTS ON CASE 2: RGB-D OBJECT DETECTION

  • The RGB-D experiments cover 2D detection on SL-RGBD and 3D detection on NYUv2; the NYUv2 results include detection performance and position-shift robustness.The SL-RGBD figure shows RGB and depth pairs with differing degrees of position shift.Source: §VI · p. 11, Table VI · p. 12, Figure 13 · p. 12

A. Dataset

The section evaluates RGB-D detection on NYUv2 and SL-RGBD, reporting comparative performance and robustness. The conclusion describes AR-CNN results across RGB-T and RGB-D datasets.

A. Dataset

  • NYUv2 contains 1,449 RGB-D pairs with 19 indoor object classes; SL-RGBD contains 214 pairs and 1,297 object annotations across seven object classes.SL-RGBD includes varied position-shift patterns.Source: A. Dataset · p. 12

B. Implementation Details

  • Training uses ImageNet-pretrained models finetuned separately on NYUv2 and SL-RGBD, with the RGB-T pedestrian detection training schedule.Evaluation uses mAP, with IoU thresholds greater than 0.5 for 2D and 0.25 for 3D detection.Source: B. Implementation Details · p. 12, A. Dataset · p. 12

C. Performance and Robutstness

  • 66.3 mAP is AR-CNN’s SL-RGBD 2D detection result, compared with 58.7 mAP when incorporating depth and 52.9 mAP otherwise.The passage identifies Table VII as the source of the reported results.Source: C. Performance and Robutstness · p. 12, Table VII · p. 13
  • 33.5 v.s. 30.2 mean performance and 3.8 v.s. 6.7 standard variance in S0◦ are reported for AR-CNN versus the comparison on NYUv2 3D detection.The passage characterizes AR-CNN as having comparative performance and the best robustness.Source: C. Performance and Robutstness · p. 12
  • At shift pattern (50, −50), AR-CNN scores 32.1 mAP versus 23.3 mAP for Halfway Fusion.The example illustrates performance under position shift.Source: C. Performance and Robutstness · p. 13

VII. CONCLUSION

  • AR-CNN achieves the best performance and robustness on CVC-14 and KAIST, and comparative performance with better robustness on NYUv2 and SL-RGBD.The conclusion presents the method as a generic solution for multimodal object detection with weakly aligned image pairs.Source: §VII · p. 13

Related papers

Loading 2204.09848v1…