Source-linked AI summary
PST900: RGB-Thermal Calibration, Dataset and Segmentation Network
Shreyas S. Shivakumar, Neil Rodrigues, Alex Zhou, Ian D. Miller, Vijay Kumar, Camillo J. Taylor
TL;DR
The paper addresses limited calibrated RGB-thermal segmentation resources and the challenge of using thermal imagery in difficult environments. It proposes passive calibration, the PST900 dataset, and a reusable dual-stream CNN, reporting state-of-the-art performance on PST900 and competitive performance on MFNet.
Problem
RGB-thermal segmentation methods need large annotated thermal datasets, but calibrated per-pixel thermal data are difficult to find.
Method
The paper combines passive no-heated-element calibration, the PST900 RGB-thermal dataset, and a dual-stream CNN with an independently reusable RGB stream.
Results
The proposed network achieves state-of-the-art performance on PST900 and is competitive with existing methods on MFNet.
Takeaways & Limitations
RGB and thermal fusion can support real-time semantic segmentation on embedded GPUs, while late fusion is favored over naive fusion approaches.
Abstract
from arXiv · showhide
In this work we propose long wave infrared (LWIR) imagery as a viable supporting modality for semantic segmentation using learning-based techniques. We first address the problem of RGB-thermal camera calibration by proposing a passive calibration target and procedure that is both portable and easy to use. Second, we present PST900, a dataset of 894 synchronized and calibrated RGB and Thermal image pairs with per pixel human annotations across four distinct classes from the DARPA Subterranean Challenge. Lastly, we propose a CNN architecture for fast semantic segmentation that combines both RGB and Thermal imagery in a way that leverages RGB imagery independently. We compare our method against the state-of-the-art and show that our method outperforms them in our dataset.
I. INTRODUCTION
The paper motivates RGB-thermal semantic segmentation for challenging, poorly illuminated environments and introduces a calibrated dataset and reusable dual-stream CNN to support accurate, real-time perception.
- Motivation: Thermal imagery can complement RGB segmentation in underground tunnels, mines, and caves where visibility and illumination are limited.LWIR is not dependent on visible-spectrum illumination, which RGB cameras rely heavily upon.
- Motivation: Thermal fusion also improves segmentation of objects without distinctive thermal signatures, including handdrills.
- Research gap: Existing RGB-T methods require substantial annotated thermal data, but such datasets are difficult to find.The authors present what they believe is the second calibrated RGB-thermal dataset with per-pixel annotations across four classes.
- Contributions: The proposed dual-stream CNN fuses RGB and thermal information while keeping the RGB stream independently reusable for real-time embedded-GPU inference.
- Contributions: The work contributes a calibration method using no heated elements, enabling faster and portable field calibration, and evaluates the method on PST900 and MFNet.
- Contributions: PST900 provides approximately 900 annotated RGB and LWIR images in raw 16-bit and AGC 8-bit formats, plus 3416 additional annotated RGB images.
II. RELATED WORK
Related work spans cross-modal prediction, stereo disparity, and datasets that exploit interactions between RGB and thermal imagery.
- Cross-modal learning: Cross-modal learning has been used to predict RGB from thermal imagery or thermal imagery from RGB imagery.
- Stereo disparity: Related stereo-disparity methods match features across RGB and thermal modalities and have produced datasets such as LITIV and St. Charles.
A. Semantic Segmentation with Thermal Images
Prior work addresses thermal semantic segmentation, calibration, and multimodal fusion, while the paper emphasizes calibration constraints in subterranean settings and the risks of naive fusion.
- Thermal segmentation: Thermal semantic-segmentation research includes RGB-D-T SLAM and YOLO-based pipelines, where experiments found substantial benefit from thermal information.
- Thermal segmentation: MFNet uses dual RGB and thermal encoders and outperforms naive fourth-channel fusion, which can underperform RGB alone when images are misaligned.
- Thermal segmentation: RTFNet fuses RGB and thermal feature blocks by element-wise summation and uses Upception blocks to manage spatial resolution and channel count.
- Thermal segmentation: The paper compares against MFNet and RTFNet because they are identified as the most relevant prior methods.
- Calibration: Existing active calibration targets require heating and can demand substantial effort, large heat sources, and careful timing because heated elements cool quickly.
- Calibration: Passive calibration using reflected cold-sky imagery improves portability but remains unsuitable when subterranean environments do not provide that sky reflection.
- Calibration: Figure 2 illustrates detected corners in undistorted RGB and thermal images and shows contrast from aluminum checkers against a backboard without heated elements.
III. METHOD
The calibration and segmentation method uses thermal reflectivity for passive RGB-thermal calibration and derives an undistorted-camera mapping for thermal-to-RGB registration.
- Calibration: The proposed calibration target uses thermally reflective sand-blasted aluminum squares on black acrylic, exploiting reflections of the person calibrating the system.
- Thermal-RGB Alignment: RGB and thermal intrinsic calibration produces camera matrices and distortion coefficients, which are used to obtain undistorted camera matrices.
2) Thermal-RGB Alignment:
The alignment procedure maps undistorted RGB pixels into 3D using stereo depth, then reprojects them into the thermal camera frame using calibrated camera parameters. Parallax and many-to-one mappings are handled by selecting the closest 3D point, with interpolation available for resulting holes.
- Calibration visualization: Figure 3 illustrates projection between thermal images at 320 × 256 resolution and RGB images at 1280 × 720 resolution when intrinsics and extrinsics are known.
- Projection procedure: The 3D points are reprojected onto the thermal image to obtain a mapping between RGB and thermal coordinates.
- Projection procedure: RGB pixels are projected into 3D using stereo depth and the inverse RGB camera matrix before re-projection into the thermal frame.The RGB and thermal cameras use calibrated intrinsics, while the system transformation uses calibrated rotation and translation matrices.
- Correspondence handling: The inverse mapping handles parallax and many-to-one correspondences by choosing the closest 3D point during reprojection.Aligned thermal imagery may contain holes, and the dataset includes a simple interpolation script for hole filling.
B. Penn Subterranean Thermal 900 Dataset (PST900):
PST900 provides aligned RGB-thermal imagery and per-pixel annotations for four subterranean-artifact classes, supplemented by a larger RGB-only corpus. The data were collected across varied underground, indoor, and outdoor environments and labels were verified by human annotators.
- Dataset: PST900 contains 894 aligned RGB and Thermal image pairs with per-pixel human annotations for four visible-artifact classes.The classes are fire-extinguisher, backpack, hand-drill, and survivor, represented by a thermal mannequin or human.
- Data collection: The dataset includes RGB imagery from a Stereolabs ZED Mini and aligned thermal imagery from a FLIR Boson 320 camera.The wider thermal field of view was selected to increase sensor overlap.
- Annotation: Human annotators created per-pixel labels, and the authors verified each RGB, thermal, and label set for accuracy.Incorrect or missed artifact annotations were returned for relabeling and reverification.
C. Segmentation
The segmentation system uses a sequential dual-stream design: an independently trainable RGB network produces a confidence volume, and a thermal fusion stream refines predictions. The design targets accuracy, reusability, and real-time embedded inference.
- Design goals: The sequential architecture is designed to support RGB-stream reuse, a rapid coarse prediction through early exit, and real-time inference on embedded hardware.The paper reports accuracy improvements and competitive performance on the MFNet dataset.
- RGB stream: The independent RGB stream is based on ResNet-18 with a UNet-like encoder-decoder and skip connections, and it is trained using RGB images alone.Training uses a weighted negative log-likelihood loss and selects the model with the highest mIoU.
- Feature transfer: After RGB training, the final softmax is removed so the resulting per-pixel confidence volume can feed the fusion stream with thermal imagery.
- Fusion stream: The fusion stream concatenates the RGB confidence volume, thermal imagery, and color input before processing them with an ERFNet-based encoder-decoder.The RGB stream is frozen while the full architecture is trained with the same loss function.
IV. RESULTS AND ANALYSIS
The evaluation compares the proposed method with established RGB-T networks and naive fusion variants using mIoU and embedded-device inference latency. Comparisons use PST900 RGB-T data while excluding the additional RGB-only data.
- Compared methods: The study compares against MFNet, RTFNet, and naive RGB-T versions of ERFNet, MAVNet, Fast-SCNN, and UNet.Naive fusion adds thermal imagery as a fourth input channel to the RGB image.
- Evaluation metrics: Performance is measured using mean Intersection over Union across all classes and inference latency in milliseconds on an NVIDIA AGX Xavier embedded GPU.Training was performed in PyTorch on an NVIDIA DGX-1.
- Evaluation protocol: For fair comparison, experiments use PST900 RGB-T data and exclude the additional RGB-only data.
A. MFNet Dataset
On MFNet, RTFNet performs best overall, while the proposed method ranks second and remains substantially faster than RTFNet-152.
- RTFNet achieves the best MFNet performance, with the proposed method ranking second.
- 48.42% mIoU is achieved by the proposed network on MFNet.
- The proposed method is roughly 4 times faster than RTFNet-152 on this dataset.
B. PST900 Dataset
On PST900, the proposed method leads the evaluated networks, while the dataset exposes challenges for naïve RGB-Thermal fusion when RGB already provides strong information.
- 68.36% mIoU is achieved by the proposed method on PST900, compared with 57.61% for RTFNet-152.
- The proposed method runs at approximately 42 ms latency on embedded GPU hardware and is significantly faster than RTFNet-152.
- PST900 may challenge thermal-fusion networks because RGB alone contains substantial information and objects can appear above or below ambient temperature.These conditions can make RGB-Thermal correlations difficult to learn and can invert thermal appearance.
- Late fusion is reported as beneficial because naïve fusion causes the RGB stream to perform poorly.
V. CONCLUSIONS
The paper combines calibration, a paired RGB-Thermal dataset, and a dual-stream segmentation network for challenging environments. Its method is reported as accurate, real-time on embedded GPUs, and competitive across PST900 and MFNet, while naïve fusion can underperform.
- On PST900, qualitative examples show thermal imagery helping under RGB motion blur, while RTFNet-152 misidentifies a visible Fire Extinguisher.
- The work presents a portable, easy-to-use RGB-Thermal camera calibration technique and PST900 with 894 aligned annotated image pairs.
- The proposed dual-stream CNN achieves state-of-the-art performance on PST900 and operates in real time on embedded GPUs.
- The method is competitive with existing methods on MFNet and supports late fusion over naïve thermal-channel addition.
- Naïve RGB-Thermal fusion can reduce performance for some networks, especially when objects have stronger RGB than thermal cues.