Stereo LWIR-RGB Automatic Labeling for Low-Light Camera-Based VRU Detection

• Low-light RGB detection is limited by inadequate illumination, and labeling overhead contributes to the shortage of large-scale low-light datasets.Source: §I · p. 2
• LWIR-RGB stereo image pairs let the pipeline label VRUs from LWIR detections and transfer filtered labels to RGB images.Source: §I · p. 2
• The proposed autolabeling pipeline improves VRU detection performance over baseline models and models fine tuned on subsets of collected RGB data.Source: §I · p. 2
• The paper presents an open-source dataset for nighttime VRU detection.Source: §I · p. 2

A. Pedestrian Detection

• Pedestrian detection progressed from hand-crafted features to CNNs and transformers, with R-CNN and DETR illustrating learned-feature and transformer approaches.Source: A. Pedestrian Detection · p. 2
• Task-specific YOLO and DETR adaptations target pedestrian detection, while SAM-family foundation models offer zero-shot labeling but may underperform trained detectors.YOLO-APD reached 77.7% mAP@50-95; RC-DETR reached 38.9% LAMR on CrowdHuman; on MinneApple, YOLO scored 72% F1 and SAM 59.8%.Source: A. Pedestrian Detection · p. 2

B. Detection in low-light conditions

• Low-light RGB research improves pedestrian detection through thermal-RGB distillation, image brightening, or learned image filtering.Source: B. Detection in low-light conditions · p. 2
• Thermal-RGB and other multisensor fusion can leverage sensors less affected by lighting, but requires purchasing and calibrating multiple sensors.Source: B. Detection in low-light conditions · p. 2

C. Cross-Modal Labeling and Automated Labeling

• Complementary modalities support automated label transfer and manual pedestrian annotation, including thermal-assisted labeling when RGB visibility is poor.Prior work transferred RGB road-segmentation labels to LiDAR, while LLVIP annotators used thermal images to label pedestrians.Source: C. Cross-Modal Labeling and Automated… · p. 3

D. Research Gap

• RGB-sourced autolabelers are poorly suited to low-light conditions, motivating a stereo-based alternative to the earlier complex IR-RGB data-gathering setup.Source: D. Research Gap · p. 3

A. Pipeline Architecture

• The pipeline labels LWIR images, filters detections by confidence and relative height, then transfers labels to the target RGB images.Source: A. Pipeline Architecture · p. 3

1) IR detection:

• Fine-tuned LWIR detection models generate labels at extremely low confidence, enabling subsequent label filtration.The models were fine-tuned using a dedicated publicly available dataset.Source: 1) IR detection · p. 3

2) Label Filtration:

• Filtration uses confidence and relative box-height thresholds: A_i keeps detections passing both, while R_i keeps confident detections below the height floor.Height indicates whether detections contain enough visual information and can be transferred reliably from LWIR to RGB.Source: 2) Label Filtration · p. 3
• Frames are POSITIVE only with localizable labels and no invalidating detections, BACKGROUND when no detection exceeds c_bg, and otherwise DISCARD.DISCARD also includes frames with detections above c_bg that fail the height-confidence criteria.Source: 2) Label Filtration · pp. 3–4

3) Label transfer:

• Label transfer aims to preserve pedestrian information while moving labels from the LWIR source images to the RGB target images.Source: 3) Label transfer · p. 4
• Stereo mapping imperfections are minor, so LWIR bounding-box coordinates map directly to RGB images with minimal shifting and no additional transfer overhead.Source: 3) Label transfer · p. 4

4) Training and evaluation of low-light RGB detection models:

• Final tests compare models trained on generated labels against off-the-shelf baselines and models fine-tuned on manually labeled validation data.Manually labeled validation and testing sets were prepared for model validation and testing.Source: 4) Training and evaluation of… · p. 4

1) Hardware and Mounting:

• The RGB camera uses an 84.8° horizontal field of view, wider than the thermal camera’s 75.0° field of view, to use the full thermal view.Both cameras were mounted side by side on the vehicle roof because windshield glass blocks LWIR waves.Source: 1) Hardware and Mounting · p. 4

2) Temporal Camera Synchronization:

• Because the LWIR camera lacked fire-on-command capability, images were captured freely and paired by PTP timestamps within a 10 ms sliding window.Each image was used at most once when a pair was available.Source: 2) Temporal Camera Synchronization · p. 4

3) Stereo Calibration:

• Stereo calibration used a floodlit paper chessboard as an active target, with 57 images yielding a 1.315 RMS projection error.The reported RGB and LWIR errors were 0.407 and 0.775, respectively.Source: 3) Stereo Calibration · p. 4
• A minor vertical calibration error persisted across recalibration attempts and was corrected by manually updating the translation matrix; best overall alignment occurred at z = 20 m.Alignment is precise only at one distance, with differences increasing farther from the convergence distance.Source: 3) Stereo Calibration · pp. 4–5

1) Data Collection:

• Data collection spanned two months, 60 driving hours, and roughly 56 km² across metropolitan and suburban areas in Espoo and Helsinki.Collection took place in late winter and early spring and aimed to represent diverse low-light VRUs.Source: 1) Data Collection · p. 5
• Recorded faces and license plates were anonymized with EgoBlur before the data was used.Source: 1) Data Collection · p. 5

2) RGB Model Evaluation Data:

• Two sequences from an area absent from training were sampled at 1 frame per second; validation has 441 images and 336 pedestrians, while testing has 1,400 images and 1,173 pedestrians.Source: 2) RGB Model Evaluation Data · p. 5

3) IR Model Training Data:

• FLIR-ADAS was selected for LWIR model training because it was produced by the camera manufacturer, minimizing the domain gap to the collected data.Training and test data were combined for fine-tuning, with validation performed on the validation set to maximize training samples.Source: 3) IR Model Training Data · p. 5, A. LWIR Model Performance · p. 5

A. LWIR Model Performance

• YOLOv11l performed best across all reported metrics on FLIR-ADAS, followed by RF-DETR, while SAM3 performed worst.The table evaluates the LWIR detection models on FLIR-ADAS.Source: A. LWIR Model Performance · p. 5, Table I · p. 5
• SAM3 was used as a prompted foundation model, while YOLOv11l and RF-DETR were fine-tuned on LWIR data.SAM3 received the prompt “person.”Source: A. LWIR Model Performance · p. 5

B. RGB Detection Model Evaluation

• The evaluation varied pedestrian-height thresholds while holding confidence at 0.25, then tested confidence values of 0.1, 0.25, and 0.5 using the best height settings.The upper empty-image threshold was fixed at 10% and the background confidence threshold at 0.02 throughout.Source: B. RGB Detection Model Evaluation · pp. 5–6
• 4% and 2% height thresholds yielded the best overall results in the height experiments, evaluated using mAP@50.Confidence was fixed at 0.25 for these experiments; Table II reports RGB detector mAP50 and remaining frame counts by height threshold.Source: B. RGB Detection Model Evaluation · p. 6, Table II · p. 6
• The comparison included COCO-pretrained YOLO and DETR models, prompted SAM3, and YOLO and DETR fine-tuned on 441 images containing 336 VRU instances.The fine-tuned subset was used to assess performance against models adapted with a small manually labeled set.Source: B. RGB Detection Model Evaluation · p. 6
• 41.7% on mAP@50 over every baseline, including its own baseline, and 38.2% over the best baseline, DETR, were achieved by YOLOv11l trained on LWIR-YOLO labels.It used confidence 0.25 and a 2% detection-height threshold, and beat all baselines on F1, mAP@50, mAP50-95, and LAMR.Source: B. RGB Detection Model Evaluation · p. 6
• Models trained on LWIR-DETR labels consistently exceeded all baselines on mAP@50 and LAMR, while none trained on SAM3 labels exceeded a baseline.The passage attributes SAM3’s poorer results to its general-purpose rather than dedicated, fine-tuned detection role.Source: B. RGB Detection Model Evaluation · pp. 6–7

V. DISCUSSION

• The stereo setup lowers hardware complexity, making large-scale dataset generation more practical despite labeling overhead and geographic dataset gaps.The authors identify labeling overhead as a dataset-generation limitation and note that public datasets can be geographically limited and mismatched by sensor domain.Source: §V · p. 7
• Fine-tuning on the small validation subset reduced baseline performance; distant pedestrians in this dataset make additional training samples especially important.The authors state that small datasets may not provide enough samples to reliably improve performance in difficult applications.Source: §V · p. 7
• Autolabeled models beat off-the-shelf and hand-subset fine-tuned models on F1, mAP@50, and LAMR; LWIR-YOLO and LWIR-DETR models both beat baselines.Table IV compares autolabeled models with the 441-frame manual reference using the same filtering setting.Source: §V · p. 7, Table IV · p. 9
• Stereo label transfer is limited by sensor gaps, fixed convergence distance mismatches, and temporal misalignment; despite improvements over baseline in other metrics, mAP@50-95 remains low, with one exception in the best overall model.The LWIR camera lacked “fire on command” capabilities; better timing approaches may further improve the method. The trade-off remains beneficial for detecting VRUs because detection matters more than pixel-perfect accuracy, and remains an open future-research improvement point.Source: §V · p. 10
• Per-detection online calibration is proposed to match modality-invariant pedestrian features across LWIR and RGB, assuming bounding boxes locate pedestrians well.The authors describe this as a possible improvement direction.Source: §V · p. 10
• Performance remains bounded by LWIR pedestrian detection; future directions include combining modalities or improving LWIR detectors directly.The authors also identify the autolabeler's pedestrian detection ability as a limitation affecting the generated labels.Source: §V · pp. 10–11

VI. CONCLUSION

• The end-to-end evaluation used a stereo camera setup instead of the previous beam splitter and a dataset collected in Finland’s capital region.The evaluated pipeline covered stereo calibration, data collection, and labeling.Source: §VI · p. 11
• Generated-label models beat both off-the-shelf models and the same models fine-tuned on hand-labeled subsets; results held for both tested LWIR autolabelers.The authors distinguish this scaled performance result from matching human labels on each individual image.Source: §VI · p. 11
• Label-transfer accuracy affects localization and is visible in the mAP@50-95 performance drop.The authors identify task-tailored transfer, including matching pedestrian features across modalities per detection, as a promising direction.Source: §VI · p. 11