Source-linked AI summary
Multimodal Object Detection via Probabilistic Ensembling
Yi-Ting Chen, Jinghao Shi, Zelin Ye, Christoph Mertz, Deva Ramanan, Shu Kong
TL;DR
The paper asks how to fuse RGB and thermal detections for multimodal object detection, especially when data are limited, modalities are unaligned, or detectors miss objects. It introduces ProbEn, a non-learned Bayesian detector-ensembling method that marginalizes missing modalities, and finds improvements over prior work on aligned and unaligned benchmarks, including a reported relative gain above 13%.
Problem
Multimodal detection must combine RGB and thermal information despite limited paired data, alignment challenges, and modality-specific missing detections.
Method
ProbEn is a non-learned late-fusion method derived from Bayes' rule under conditional independence, using probabilistic marginalization to handle missing modalities.
Results
>13% relative improvement over prior work is reported on both aligned and unaligned multimodal benchmarks.
Takeaways & Limitations
ProbEn is proposed as a general detector-ensembling technique and a baseline for future multimodal detection research.
Takeaways & Limitations
ProbEn is derived under a conditional-independence assumption, although the paper reports effectiveness when that assumption does not strictly hold.
Abstract
from arXiv · showhide
Object detection with multimodal inputs can improve many safety-critical systems such as autonomous vehicles (AVs). Motivated by AVs that operate in both day and night, we study multimodal object detection with RGB and thermal cameras, since the latter provides much stronger object signatures under poor illumination. We explore strategies for fusing information from different modalities. Our key contribution is a probabilistic ensembling technique, ProbEn, a simple non-learned method that fuses together detections from multi-modalities. We derive ProbEn from Bayes' rule and first principles that assume conditional independence across modalities. Through probabilistic marginalization, ProbEn elegantly handles missing modalities when detectors do not fire on the same object. Importantly, ProbEn also notably improves multimodal detection even when the conditional independence assumption does not hold, e.g., fusing outputs from other fusion methods (both off-the-shelf and trained in-house). We validate ProbEn on two benchmarks containing both aligned (KAIST) and unaligned (FLIR) multimodal images, showing that ProbEn outperforms prior work by more than 13% in relative performance!
1 Introduction
The paper studies RGB–thermal multimodal detection for safety-critical settings and explores detector ensembling as a very-late fusion strategy. ProbEn is a non-learned probabilistic method designed to combine complementary evidence and handle missing detections across modalities.
- RGB–thermal detection is motivated by autonomous vehicles operating across day and night, with thermal cameras providing stronger object signatures under poor illumination.
- Multimodal detection faces limited paired data, alignment challenges, and detectors that may fail to fire on the same object.
- The paper explores very-late fusion by ensembling independent detectors trained separately on RGB and thermal inputs.
- ProbEn derives score fusion from Bayes' rule under conditional independence and uses probabilistic marginalization to handle missing modalities.
- Detector ensembling is presented as a meaningful multimodal fusion contribution because the precise technique matters and prior score-averaging or max-voting methods are less effective, especially with missing modalities.
2 Related Work
Prior work largely studies multimodal fusion through network architectures and feature merging, while very-late detector fusion has been largely absent from multimodal detection. The paper positions detector ensembling as an effective late-fusion alternative for both aligned and unaligned RGB–thermal data.
- Single-modal detectors and NMS are widely studied in RGB detection, but very-late fusion of multimodal detections had not previously been used to the authors' knowledge.
- Most multimodal detection methods focus on early, mid-, or translation-based fusion, with architectural design centered on merging RGB and thermal features.
- The paper evaluates multimodal detection under both aligned and heavily unaligned RGB–thermal scenarios, where the performance of existing methods is unclear.
3 Fusion Strategies for Multimodal Detection
The section compares multimodal detection fusion strategies and derives ProbEn as a Bayesian late-fusion method based on conditional independence. ProbEn combines detector outputs using class priors, handles missing modalities, and extends to bounding-box fusion.
- Fusion baselines: Mid-fusion is generally the most effective baseline, including on unaligned data annotated with only one modality.
- Late-fusion baselines: Naive pooling combines detections without resolving overlaps, while NMS retains the highest-scoring detection and suppresses overlapping lower-scoring boxes.
- Late-fusion baselines: Average Fusion averages confidence scores for overlapping detections, but this lowers the score relative to the maximum and performs worse than NMS.
- Probabilistic Ensembling: ProbEn assumes measurements are conditionally independent given the object label and derives multimodal posterior fusion from Bayes' rule.
- Probabilistic Ensembling: The resulting strategy trains independent single-modal classifiers, multiplies their posteriors, divides by the class prior, and normalizes the result.
- Probabilistic Ensembling: In logit form, ProbEn sums modality logits, corrects for repeated class priors, and normalizes with a softmax; probabilistic marginalization handles missing modalities.
- Bounding Box Fusion: ProbEn also extends the probabilistic fusion derivation from class posteriors to overlapping bounding-box coordinates using a continuous box variable and a uniform prior.
4 Experiments
Experiments evaluate multimodal fusion on the aligned KAIST and heavily unaligned FLIR benchmarks using non-learned late fusion and learned fusion baselines. ProbEn consistently improves detection, including when fusing outputs from models that are not conditionally independent.
- Experimental setup: Experiments evaluate fusion methods on KAIST and FLIR, using LAMR for pedestrian detection and AP with IoU>0.5 for FLIR.KAIST provides aligned RGB-thermal pairs, while FLIR contains heavily unaligned modalities.
- KAIST: KAIST ablations show that MidFusion outperforms EarlyFusion, while naive pooled detections perform poorly because repeated detections become false positives.NMS is effective at removing overlapping detections, but average score fusion is ineffective because it decreases the score relative to NMS.
- KAIST: ProbEn improves KAIST fusion by probabilistically integrating RGB and Thermal detections, with ProbEn3 performing best after adding MidFusion.The method can also probabilistically fuse overlapping box coordinates using predicted uncertainty.
- Qualitative results: Qualitative examples show ProbEn reducing missed detections relative to MidFusion on KAIST and relative to Thermal-only detection on FLIR.The KAIST examples depict true positives, false negatives, and false positives using colored bounding boxes.
- KAIST: 5.14 LAMR is achieved by ProbEn++ versus 6.48 for prior art on KAIST when using off-the-shelf mid-fusion detectors.ProbEn+ using three in-house models achieves 7.95 LAMR, while ProbEn remains effective despite violated conditional independence.
- FLIR: On FLIR, EarlyFusion and MidFusion improve over Thermal, late fusion further boosts performance, and ProbEn performs best among the explored methods.MidFusion handles RGB-thermal misalignment better than EarlyFusion, while uncertainty-based box fusion outperforms heuristic alternatives.
5 Discussion and Conclusions
ProbEn is a non-learned probabilistic fusion method that improves multimodal detection across aligned and unaligned RGB-thermal benchmarks. Its design exploits strong single-modal detectors and accommodates missing detections.
- ProbEn significantly outperforms prior multimodal detection approaches on both aligned and unaligned RGB-thermal benchmarks.The reported improvement is greater than 13% in relative performance.
- ProbEn leverages highly tuned single-modal detectors trained on large-scale single-modal datasets.
- ProbEn handles missing detections when particular modalities fail to fire on the same object.This addresses a common challenge in multimodal detection, especially when RGB detectors fail in nighttime conditions.
- NMS also performs remarkably well because it exploits the same key insights identified in the fusion analysis.
Appendix
The appendix extends the paper with analyses, calibration studies, fusion investigations, derivations, model comparisons, and qualitative demonstrations of ProbEn.
- The appendix analyzes ProbEn against other late-fusion methods and studies score calibration.
- Additional studies examine weight-score fusion and class priors in ProbEn.
- The appendix provides a detailed derivation of probabilistic box fusion.
- Further experiments study fusing more and better models, alongside qualitative results and a video demonstration.
6 Probabilistic Fusion for Logits
The logit analysis shows that ProbEn corresponds to summing modality-specific logits under a uniform class prior, while summation supports confidence growth as modalities agree. Experiments further show that ProbEn outperforms alternative late-fusion methods, including for non-independent detectors.
- Logit formulation: Under a uniform class prior, ProbEn is equivalent to summing per-modality logits before applying the softmax.This also offers a numerically stable implementation when cached logits are available.
- Summing versus averaging: Summing logits allows fused detections to become more confident as more modalities agree, unlike averaging logits.This distinction matters for detection because scores are ranked and thresholded during NMS and global filtering.
- Logit formulation: In single-class detection, posterior behavior can be analyzed through each modality’s relative logit score.
- Empirical comparisons: ProbEn consistently outperforms other late-fusion methods across different underlying detectors on KAIST reasonable-test.
- Empirical comparisons: Fusing non-independent detectors can outperform fusing independent detectors in the reported experiments.The comparison includes combinations such as A+B+D versus A+B.
- False-positive handling: ProbEn lowers scores for overlapping detections when at least one modality assigns a low score, helping suppress false positives.In the cited example, NMS retains a higher-scoring false-positive statue detection while ProbEn lowers its fused score.
7 Score Calibration for Fusion
ProbEn assumes calibrated class posteriors, motivating temperature-based score calibration. Calibration has little effect for the two in-house detectors but becomes important when fusing them with GAFF, where both temperature and shift matter.
- Calibration motivation: ProbEn assumes detectors return true class posteriors, although deep networks can be over-confident even when wrong.
- Calibration motivation: Temperature scaling divides logits by T, with T > 0, to soften overconfident predictions.
- Two-detector calibration: Temperature tuning yields only marginal improvement over standard ProbEn for the two single-modal RGB and thermal detectors.For this setting, T=1 corresponds to ProbEn, and only one modality needs calibration.
- GAFF calibration: Score calibration is particularly important when combining GAFF with the in-house RGB and thermal detectors.
- GAFF calibration: For GAFF fusion, calibration requires both a temperature variable and a logit shift variable, with the shift substantially affecting fusion results.
8 Further Study of Weighted Score Fusion
The authors compare learned weighted score fusion with non-learned alternatives for late fusion. ProbEn and AvgLogits use predefined weights, while temperature scaling tunes parameters using single-modal validation data.
- Learned late fusion combines single-modal logits using class-specific weights w_1[k] and w_2[k].
- ProbEn, AvgLogits, and Temperature Scaling can be viewed as special cases of learned weighted-logit fusion.
- ProbEn and AvgLogits require no learning because they use predefined weights.
- Temperature scaling requires single-modal validation data to tune each temperature parameter, but not multimodal training.
- These methods are useful when modalities are unaligned or when more single-modal than multimodal training data exists.
9 Further Study of Class Prior in ProbEn
The authors test whether replacing uniform class priors with FLIR-derived priors improves ProbEn. Uniform priors perform better, while tuning individual class priors yields only marginal gains.
- FLIR contains imbalanced counts for person, bicycle, and car annotations.The reported counts are person (21,744), bicycle (3,806), and car (39,372).
- Uniform class priors outperform computed class-frequency priors for ProbEn on FLIR.The comparison uses three fused models—Thermal, Early, and Mid—and AP at IoU>0.5.
- Tuning one class prior at a time produces only marginal improvements over uniform priors.
10 A Detailed Derivation of Probabilistic Box Fusion
The derivation models each detected bounding box probabilistically and combines modality-specific posteriors. It assumes Gaussian box posteriors, modality-predicted coordinates, and a uniform prior over image-plane locations.
- Probabilistic box fusion combines multiple bounding boxes through their posterior distributions.
- Each detection is represented by a continuous random variable z parameterized by centroid, width, and height.
- Single-modal detections are modeled with Gaussian posteriors p(z|x_i) having a single variance σ2_i.
- The modality-specific means μ_i are the predicted box coordinates.
- A uniform prior p(z) assumes box coordinates can lie anywhere in the image plane.
11 A Study of Fusing More and Better Models
The authors evaluate ProbEn across more detector combinations on KAIST and FLIR, including combinations with stronger or non-independent detections. ProbEn consistently performs best, while qualitative results also favor it.
- ProbEn consistently performs best across the evaluated combinations of underlying detectors on KAIST and FLIR.
- ProbEn can improve fusion even when the underlying detectors are not conditionally independent, such as Thermal and MidFusion.
- The FLIR demonstration compares Thermal with ProbEn fused from Thermal, Early-fusion, and Mid-fusion outputs.The dataset provides unaligned RGB and thermal frames and annotates only thermal frames.
- ProbEn always outperforms NMS when applied to the same ensemble, including strong detections.
- Qualitative examples on KAIST and FLIR show ProbEn performing better than the compared methods.