Source-linked AI summary
Cross-Modal Collaborative Representation Learning and a Large-Scale RGBT Benchmark for Crowd Counting
Lingbo Liu, Jiaqi Chen, Hefeng Wu, Guanbin Li, Chenglong Li, Liang Lin
TL;DR
RGB-only crowd counting lacks sufficient information for recognizing people in unconstrained scenes, motivating multimodal RGB-thermal learning. The paper introduces the RGBT-CC benchmark and an IADM-based collaborative framework, which improves multimodal crowd counting and generalizes to RGBD counting.
Problem
RGB-only crowd-counting methods may fail under poor illumination and confuse human-shaped or heating objects with pedestrians, despite the need for rich information.
Method
The paper combines modality-specific and modality-shared branches with an Information Aggregation-Distribution Module for dual propagation of complementary RGB and thermal information.
Results
Extensive experiments show the framework is effective on RGBT-CC and universal across multimodal crowd counting, including RGBD counting.
Takeaways & Limitations
The benchmark and framework support multimodal crowd counting by combining complementary optical and thermal information.
Abstract
from arXiv · showhide
Crowd counting is a fundamental yet challenging task, which desires rich information to generate pixel-wise crowd density maps. However, most previous methods only used the limited information of RGB images and cannot well discover potential pedestrians in unconstrained scenarios. In this work, we find that incorporating optical and thermal information can greatly help to recognize pedestrians. To promote future researches in this field, we introduce a large-scale RGBT Crowd Counting (RGBT-CC) benchmark, which contains 2,030 pairs of RGB-thermal images with 138,389 annotated people. Furthermore, to facilitate the multimodal crowd counting, we propose a cross-modal collaborative representation learning framework, which consists of multiple modality-specific branches, a modality-shared branch, and an Information Aggregation-Distribution Module (IADM) to capture the complementary information of different modalities fully. Specifically, our IADM incorporates two collaborative information transfers to dynamically enhance the modality-shared and modality-specific representations with a dual information propagation mechanism. Extensive experiments conducted on the RGBT-CC benchmark demonstrate the effectiveness of our framework for RGBT crowd counting. Moreover, the proposed approach is universal for multimodal crowd counting and is also capable to achieve superior performance on the ShanghaiTechRGBD dataset. Finally, our source code and benchmark are released at {\url{http://lingboliu.com/RGBT_Crowd_Counting.html}}.
1. Introduction
The paper identifies complementary weaknesses in RGB and thermal sensing for unconstrained crowd counting, then introduces an RGBT benchmark and collaborative framework to exploit both modalities.
- RGB-only methods may miss pedestrians in poor illumination and confuse human-shaped clutter with people.
- Thermal images help distinguish pedestrians from clutter, while optical information can eliminate heating objects highlighted in thermal images.
- The framework combines modality-specific branches, a modality-shared branch, and IADM to capture complementary multimodal information.
- IADM uses dual information propagation to enhance modality-shared and modality-specific representations, while the framework also applies to RGB-Depth counting.
- 2,030 pairs of RGB-thermal images with 138,389 annotated pedestrians form the proposed RGBT-CC benchmark.
2. Related Works
Prior crowd-counting benchmarks and methods have limitations in diversity, illumination robustness, or modality coverage, motivating RGBT data and multimodal representation learning.
- Earlier crowd-counting datasets often had limited diversity or density coverage, motivating larger and more varied benchmarks.
- Depth data can be insensitive to illumination but may be coarse outdoors because of limited 0∼20 meter depth ranges.
- RGBT sensing addresses illumination robustness and perception distance, motivating the first RGBT crowd-counting dataset.
- Crowd-counting approaches commonly predict counts directly or generate density maps whose pixel values are accumulated.
- Multimodal representation learning includes early, late, and two-stream fusion strategies for combining cross-modal features.
- The proposed direction aims to take full advantage of modal complementarities in crowd counting.
3. RGBT Crowd Counting Benchmark
The RGBT-CC benchmark collects paired RGB-thermal crowd images across varied scenarios, while the proposed framework uses parallel modality-specific and shared representations with IADM.
- The benchmark uses optical-thermal cameras to collect RGB-thermal images across malls, streets, playgrounds, train stations, and metro stations.
- RGB images have 2,048×1,536 resolution and a wider field of view, whereas thermal images have 640×480 resolution.
- The framework contains three parallel backbones: two modality-specific branches and one modality-shared branch.
- IADM dynamically transfers specific-shared information to collaboratively enhance modality-specific and modality-shared representations.
4. Method
The framework uses modality-specific and modality-shared branches connected by IADM, which extracts contextual information and transfers complementary content in both directions for mutual enhancement.
- Framework Overview: Three parallel backbones learn RGB-specific, thermal-specific, and modality-shared representations for multimodal crowd counting.The modality-specific branches preserve information from each modality, while the shared branch learns common representations.
- Framework Overview: IADM is embedded after multiple convolutional layers to transfer complementary information hierarchically among modality-specific and modality-shared features.The module is inserted after Conv1_2, Conv2_2, Conv3_3, and Conv4_3 for progressive representation learning.
- Prediction: The final enhanced modality-shared feature is fed to a 1*1 convolutional layer to predict the crowd density map.The dual propagation mechanism mutually enhances shared and modality-specific representations before prediction.
- Contextual Information Extraction: Contextual information is extracted with pyramid pooling and 1*1 convolution, enlarging receptive fields and reducing RGB-thermal misalignment effects.Max-pooling provides broader context and translation invariance, while concatenation and 1*1 convolution produce the contextual representation.
- Information Aggregation Transfer: Information Aggregation Transfer dynamically filters modality-specific contextual information before propagating it to enhance the modality-shared feature.Residual information and gating functions are used to adaptively select useful complementary content rather than directly absorbing all information.
- Information Distribution Transfer: Information Distribution Transfer sends contextual information from the enhanced shared feature back to each modality-specific feature for feedback refinement.The enhanced modality-specific features are then passed into subsequent layers for further multimodal representation learning.
5. Experiments
Experiments evaluate the framework on RGBT-CC and ShanghaiTechRGBD using multiple backbones, multimodal settings, and ablations. Across these evaluations, IADM improves backbone performance, captures RGB-thermal complementarity, and transfers effectively to RGB-depth counting.
- Implementation Details & Evaluation Metrics: The experiments use PyTorch implementations with CSRNet, MCNN, SANet, and BL backbones, feeding paired 640×480 RGBT images for training.Backbone channels are reduced to 70% or 60% of their original sizes for fair comparisons; Adam optimization and geometry-adaptive Gaussian density maps are used.
- Implementation Details & Evaluation Metrics: GAME measures regional counting error while RMSE measures overall counting error; GAME(0) is equivalent to MAE.At level l, images are divided into 4^l nonoverlapping regions and errors are aggregated across testing samples.
- Ablation Studies: 17.3% relative RMSE improvement is achieved by CSRNet+IADM over thermal-based CSRNet on RGBT-CC.RGB-only CSRNet obtains GAME(0) 33.94 and RMSE 69.59, while thermal-only CSRNet reaches 21.64 and 37.38; combining both modalities performs better.
- Ablation Studies: CSRNet+IADM consistently outperforms unimodal CSRNet in both bright and dark scenes by using thermal cues for pedestrians and optical cues to reject thermal false positives.The accompanying visualization reports more accurate density maps and estimated counts for the proposed multimodal model.
- Comparison with State-of-the-Art Methods: On RGBT-CC, all IADM instances improve their corresponding backbones, while CSRNet+IADM and BL+IADM outperform UCNet, HDFNet, and BBSNet across all evaluation metrics.MCNN+IADM and SANet+IADM each show 18.9% relative RMSE improvement over their early-fusion counterparts.
- Apply to RGBD Crowd Counting: On ShanghaiTechRGBD, CSRNet+IADM reaches GAME(0) 4.38 and RMSE 7.05, establishing the reported state of the art while outperforming its backbone and advanced comparison models.The RGB-depth experiment supports the framework’s reported universality beyond RGBT counting.
6. Conclusion
The work introduces a large-scale RGBT crowd-counting benchmark and a cross-modal collaborative framework that captures complementary information across modalities. Experiments on two real-world benchmarks support the method’s effectiveness and universality for multimodal crowd counting.
- The RGBT crowd-counting benchmark contains 2,030 RGB-thermal image pairs and 138,389 annotated people.
- The proposed framework uses an Information Aggregation-Distribution Module to capture complementary information from different modalities.
- Experiments on two real-world benchmarks demonstrate the method’s effectiveness and universality for multimodal RGBT and RGBD crowd counting.