PaperlayerView exact revision on arXiv

Source-linked AI summary

TriSAR-3D and PaintFusion: A Novel Trimodal LiDAR-RGB-Thermal Dataset and Method for 3D Person Detection in Search and Rescue

Markus Kramer, Jan Wachenfeld, Oskar von Stryk

arXiv:2610.22509v1cs.CVcs.RO

TL;DR

Rescue robotics needs 3D person detection beyond the upright, near-road poses common in automotive datasets. The paper introduces TriSAR-3D and PaintFusion to study trimodal detection and pose diversity; PaintFusion achieves 65.4 AP@50, with 51.6 ms per-frame latency.

  • Problem

    Automotive 3D detection datasets mostly contain upright, near-road pedestrians, leaving performance in rescue settings with varied poses and environments unclear.

    Source: §I · p. 1

  • Method

    The paper introduces TriSAR-3D, a synchronized LiDAR-RGB-thermal dataset, and PaintFusion, a trimodal detector combining lightweight segmentation, MLP-based point scoring, and a PointPillars encoder with a CenterPoint head.

    Source: §VI · p. 8

  • Results

    65.4 AP@50: PaintFusion achieves the best overall score among evaluated methods and runs at 51.6 ms per frame.

    Source: B. Overall Detection Performance · p. 7

  • Takeaways & Limitations

    Removing lying instances from training collapses lying-pose AP across architectures, highlighting the importance of training-set composition.

    Source: §VI · p. 8

    Recordings use a fixed sensor rig and omit ego-motion-induced blur and environmental conditions such as fog, dust, and smoke.

    Source: E. Limitations · p. 8

Abstract

from arXiv · show

Reliable 3D person detection is important for rescue robotics to give first responders situational awareness. Yet most multimodal 3D detection research targets automotive scenarios in which pedestrians mostly appear in a narrow range of upright, near-road poses. It is therefore unclear how much of the reported performance transfers to rescue settings, as a person may be lying, partially occluded, or in unstructured indoor and outdoor environments. To investigate this gap, we introduce TriSAR-3D, a multimodal person detection dataset recorded with a high-resolution 128-channel LiDAR, an RGB camera, and a thermal camera. It captures varied poses in indoor as well as outdoor locations at different times of day to study their influence on 3D object detector performance. Furthermore, we propose a novel trimodal 3D detection pipeline, PaintFusion, that fuses PointPainting-based evidence using a simple MLP. We show that the derived method reaches 65.4 AP@50 and thereby outperforms strong LiDAR-only (CenterPoint, 53.4), RGB+LiDAR (BEVFusion, 60.5), and adapted trimodal (Tri-BEVFusion, 60.7) baselines on the dataset. At the same time, it is substantially faster than other fusion-based methods such as BEVFusion, with a per-frame latency of 51.6 ms compared to 101 ms. Additionally, we test how the performance of different 3D object detectors generalizes to unseen pose categories. Removing lying instances from training data collapses the lying-pose AP@50 to near zero across all evaluated detectors. The results indicate that dataset diversity exerts a large effect on 3D object detection performance. The dataset and code are available at https://github.com/MarkusKramer1/3D-Trimodal-People-Detection.

I. INTRODUCTION

Rescue robotics needs 3D person detection across varied environments and poses, while much existing work is evaluated on automotive data dominated by upright, near-road pedestrians. The paper introduces TriSAR-3D and PaintFusion to study multimodal sensing and training-set diversity in this setting.

I. INTRODUCTION

  • Automotive 3D detection datasets mostly show upright pedestrians near roads, unlike rescue settings that include unusual poses and darkness.Rescue missions may occur indoors or outdoors, from daylight to complete darkness, and 3D detection localizes a person in the scene.Source: §I · p. 1
  • The paper asks how thermal sensing affects LiDAR–RGB detection and how detectors perform on poses rare or absent from training.It investigates both questions using a new trimodal dataset and a controlled comparison.Source: §I · p. 1
  • TriSAR-3D is a person-detection dataset recorded with a 128-channel LiDAR, RGB, and thermal camera.The paper studies modality choice and training-set diversity using the dataset.Source: §I · pp. 1–2
  • PaintFusion is reported as the most accurate evaluated method and has about half the latency of other fusion baselines.The supplied introduction does not specify the method's architecture or quantitative scores.Source: §I · pp. 1–2

II. RELATED WORK

Existing trimodal human datasets vary in task focus, sensors, annotations, and scene coverage, and none directly targets search-and-rescue robotics. TriSAR-3D is distinguished by combining high-resolution LiDAR, pose labels, 3D boxes, and indoor and outdoor scenes.

II. RELATED WORK

  • Public trimodal datasets with 2D/3D human annotations remain rare and differ in task focus, sensor setup, and scene diversity.The review compares available datasets and multimodal 3D detection methods.Source: A. Datasets · p. 2, §II · p. 2

A. Datasets

  • Only TriSAR-3D combines 3D boxes, synchronized RGB and thermal instance masks, and pose tags for each person.R-LiViT has more 3D boxes overall but lacks pose labels and indoor coverage; TriStar and Mid-3K lack 3D annotations.Source: A. Datasets · p. 2
  • TriSAR-3D uses a 128-channel LiDAR and combines indoor and outdoor scenes, whereas prior datasets typically cover one or the other.The paper describes its higher LiDAR resolution as better suited to detecting humans at longer range.Source: A. Datasets · p. 2
  • TriStar annotations concentrate within 0–10 m, TriSAR-3D spans farther, and SeeingThroughFog mainly focuses on long-distance samples.TriStar includes all poses, but has few lying samples, making proper evaluation more difficult.Source: A. Datasets · p. 2

B. Multimodal Detection

Multimodal 3D detection combines LiDAR geometry with camera evidence to address sparse points and limited illumination. The problem is to predict oriented 3D person boxes from calibrated LiDAR, RGB, and thermal observations.

B. Multimodal Detection

  • LiDAR-camera fusion adds semantic cues to sparse point clouds; BEVFusion and TransFusion require large training sets.BEVFusion lifts both modalities into shared BEV space, whereas TransFusion learns associations between 3D queries and image features.Source: B. Multimodal Detection · pp. 2–3
  • Early fusion paints image evidence onto LiDAR points, while BEVFusion and TransFusion instead combine camera and LiDAR features in a shared BEV space or through learned association.PointPainting paints semantic scores, PointAugmenting paints CNN features, and MVP adds virtual points for sparse far-range objects.Source: B. Multimodal Detection · p. 3
  • T360Fusion combines LiDAR with RGB and thermal panoramas in shared BEV space, but accumulates frames over time and is computationally heavy.Source: B. Multimodal Detection · p. 3

III. PROBLEM STATEMENT

  • The task uses calibrated LiDAR, RGB, and thermal observations to predict oriented person boxes represented by center, size, and yaw.Predictions are matched to ground truth by 3D intersection-over-union, and performance is reported as average precision.Source: §III · p. 3

IV. METHOD

The method section covers a trimodal sensor-box, sensor calibration, and an early-fusion 3D detection network.

IV. METHOD

  • The method section describes the trimodal sensor-box, calibration of LiDAR, RGB, and thermal sensors, and the proposed early-fusion 3D detection network.Source: §IV · p. 3

A. Sensor-box

The sensor-box combines LiDAR, RGB-D, and thermal cameras, synchronized for data acquisition. The calibration target is designed to be detectable in the RGB image, thermal image, and LiDAR point cloud.

A. Sensor-box

  • The sensor-box integrates an Ouster OS0 LiDAR, FLIR A70 thermal camera, and Intel RealSense D455 RGB-D camera, using only its RGB stream.The LiDAR has 128 channels and a 250 m maximum range; the RGB stream captures 640 × 480 images at 15 Hz, and thermal captures 640 × 480 at 30 Hz.Source: A. Sensor-box · p. 3
  • Software timestamp alignment synchronizes sensors at 10 Hz, with estimated median RGB-LiDAR and thermal-LiDAR synchronization errors below 10 cm.The synchronization can induce minor motion artifacts in dynamic scenes.Source: A. Sensor-box · p. 3

B. Calibration Target

  • The extrinsic calibration target uses heated 3D-printed ArUco markers visible in RGB and thermal images, with holes that make it rotation-invariant and detectable in LiDAR.A PETG checkerboard on a heated plate is used for intrinsic calibration of both cameras.Source: B. Calibration Target · pp. 3–4, Figure 6 · p. 4

C. Calibration

Calibration is refined by jointly optimizing sensor and board poses against camera reprojection and LiDAR residuals. The reported errors are 1.56 px camera-camera, 3.58 px RGB-LiDAR, and 2.96 px thermal-LiDAR.

C. Calibration

  • Calibration refinement jointly optimizes RGB-camera, thermal-camera, and board poses while holding the LiDAR frame fixed.Camera residuals compare predicted and observed pixels; LiDAR residuals compare board corners with LiDAR-detected corners in the world frame, and the weighted squared residuals use Huber loss.Source: C. Calibration · p. 4
  • PaintFusion projects RGB and thermal segmentation features onto LiDAR points, scores each point, then uses PointPillars and a CenterPoint head to predict 3D boxes.Source: Figure 7 · p. 5
  • 1.56 px camera-camera, 3.58 px RGB-LiDAR, and 2.96 px thermal-LiDAR calibration errors are reported across five recordings.The target covered large parts of the image in the recordings.Source: C. Calibration · p. 5

D. Dataset Annotation

The annotation process produces an oriented 3D box, pose tag, and matched RGB and thermal instance masks for each person.

D. Dataset Annotation

  • Each person receives an oriented 3D bounding box, a pose tag, and an instance mask in both the RGB and thermal images.The 3D boxes are derived from the point cloud.Source: D. Dataset Annotation · p. 5
  • After background removal, HDBSCAN clusters are retained only when their projections overlap an image detection, then tracked with a Kalman filter.Source: D. Dataset Annotation · p. 5
  • Corrected 3D boxes are projected into both images and Hungarian-matched to detections; unmatched detections are discarded.The matched detection's instance mask becomes the 2D annotation.Source: D. Dataset Annotation · p. 5

E. Detection Network

The detection network paints fused RGB and thermal evidence onto LiDAR points, scores candidate points with an MLP, and generates 3D boxes from the decorated cloud.

E. Detection Network

  • The pipeline has three stages: RGB and thermal 2D instance segmentation, mask-evidence painting and point scoring, then 3D box proposal generation.Source: E. Detection Network · p. 5
  • Separate lightweight YOLO26n-seg models are fine-tuned on RGB and thermal masks, allowing the two camera streams to be processed in parallel.Source: E. Detection Network · p. 5
  • LiDAR points projecting inside a person mask become candidates, and RGB and thermal evidence is fused at each shared LiDAR index.This lets the pipeline use one camera's detection information when the other camera's detection fails.Source: E. Detection Network · p. 5
  • Each candidate uses 33 features spanning depth and geometry, mask layout, local seed support, intensity, overlap, detector confidence, and cross-camera agreement.Source: E. Detection Network · p. 5
  • A feed-forward MLP maps preprocessed candidate features to person probabilities and is trained with binary crossentropy against LiDAR-return-in-box labels.Source: E. Detection Network · p. 5
  • The MLP score is added as a point-cloud channel before PointPillars, a convolutional backbone, and a CenterPoint head generate oriented 3D box proposals.Source: E. Detection Network · p. 5

V. EVALUATION

The evaluation compares LiDAR-only and multimodal detector setups on TriSAR-3D, using a split designed to represent varied scenes while limiting temporal leakage.

V. EVALUATION

  • The evaluation compares the trimodal pipeline on TriSAR-3D against LiDAR-only and multimodal baselines.Source: §V · p. 5

A. Experimental Setup

  • Table II compares AP@50 across detection methods, sensor regimes, and pose tags.Source: Table II · p. 6
  • Frames are split 80%/10%/10% into training, validation, and test sets using 10–50-second subsequences, with a 1-second buffer between splits.Subsequence assignment aims to keep locations, poses, and lighting conditions represented across sets.Source: A. Experimental Setup · p. 6
  • Evaluation covers people with box centers 5–30 m away, excludes severely occluded instances under the stated point-count or image-occlusion criteria, and reports overall and pose-conditioned AP@50.Latency is averaged over the test set on an Intel Core i9-12900H and NVIDIA GeForce RTX 3080 Ti Laptop GPU.Source: A. Experimental Setup · p. 6
  • The sensor comparison includes LiDAR-only, RGB+LiDAR, Thermal+LiDAR, and trimodal setups, with thermal variants of PointPainting and BEVFusion-based pipelines.The models are trained on the new dataset under the same distance and occlusion limits.Source: A. Experimental Setup · p. 6
  • Figure 8 reports 95% bootstrap confidence intervals for overall AP@50 on the test split.Source: Figure 8 · p. 6

B. Overall Detection Performance

PaintFusion achieves the highest reported overall AP@50 among the evaluated methods, while the pose-diversity ablation shows a near-total collapse in lying-specific performance when lying instances are removed from training.

B. Overall Detection Performance

  • 53.4 overall AP@50 is achieved by CenterPoint, the strongest LiDAR-only detector; PointPillars reaches 47.1, PV-RCNN 38.6, and SECOND 32.6.Source: B. Overall Detection Performance · p. 6
  • 60.5 overall AP@50 is achieved by BEVFusion, a 7.1 AP improvement over the best LiDAR-only method.The passage attributes the gain to semantic cues that help when geometric evidence is weak.Source: B. Overall Detection Performance · p. 6
  • Thermal information instead of additional RGB yields a further 8.5 AP improvement from PointPainting, suggesting greater usefulness than RGB independent of pose.Source: B. Overall Detection Performance · pp. 6–7

C. Impact of Pose Diversity in Training

  • Removing lying instances from training causes lying-specific AP@50 to collapse almost entirely across architectures, while overall AP@50 drops only modestly for most methods.The table reports AP@50 for full training and training without lying instances on the full test split and by pose tag.Source: Table III · p. 7, C. Impact of Pose Diversity in Training · p. 7

B. Overall Detection Performance

  • 65.4 overall AP@50 and 49.7 sitting AP@50 are achieved by the trimodal method, which runs at 51.6 ms per frame versus 105.6 ms for Tri-PointPainting and 107.1 ms for Tri-BEVFusion.Tri-BEVFusion has the highest walking AP@50 at 71.0.Source: B. Overall Detection Performance · p. 7

D. Distance-Stratified Analysis

Across distance bins, PaintFusion improves on LiDAR-only and RGB+LiDAR baselines; the study also identifies limits in the recording setup and outlines dataset and method extensions.

D. Distance-Stratified Analysis

  • From 5–10 m to 25–30 m, mean LiDAR points per person fall from 135 to 9, while PaintFusion gains 17.3–23.2 AP over the LiDAR-only mean across distance bins.The RGB+LiDAR mean gains +0.7 to +19.6 AP; PaintFusion improves over that mean by up to 11.8 AP.Source: Figure 9 · p. 7, D. Distance-Stratified Analysis · p. 7

E. Limitations

  • The dataset uses a fixed sensor rig and currently excludes fog, dust, and smoke, leaving mobile-platform motion artifacts and these environmental conditions untested.Both are identified as future work.Source: E. Limitations · p. 8

VI. CONCLUSION AND OUTLOOK

  • TriSAR-3D provides synchronized 128-channel LiDAR, RGB, and thermal streams, indoor and outdoor recordings, and 3D boxes, 2D masks, and pose tags.The conclusion contrasts its higher LiDAR resolution and explicit pose labels with existing trimodal datasets.Source: §VI · p. 8
  • PaintFusion achieves the best overall AP@50 among evaluated methods at roughly half the latency of other multimodal baselines.The conclusion also reports that training-set composition matters more than detector choice for lying-pose AP.Source: §VI · p. 8
  • Planned extensions include dust, fog, and smoke recordings, a mobile sensor platform, and tracking across frames to stabilize cluttered-scene detections and improve recall for partially observable people.Source: §VI · p. 8

Related papers

Loading 2610.22509v1…