PaperlayerView exact revision on arXiv

Source-linked AI summary

Real-Time Scene-Adaptive Tone Mapping for High-Dynamic Range Object Detection

Gongzhe Li, Linwei Qiu, Peibei Cao, Fengying Xie, Xiangyang Ji, Qilin Sun

arXiv:2608.30400v1cs.CV

TL;DR

LDR-trained embedded detectors struggle with high-bit-depth HDR RAW inputs because extreme dynamic ranges disrupt feature extraction and conflict with LDR sRGB requirements. The paper introduces neural photometric calibration and a scaling-invariant tone mapper optimized jointly with detection, achieving improved HDR detection against comparison methods and real-time 4K processing. Its current pipeline still requires cascading with detectors for joint HDR RAW processing.

  • Problem

    LDR-trained detection networks and embedded inference engines are poorly matched to high-bit-depth HDR RAW inputs, whose extreme dynamic ranges degrade feature extraction.

  • Method

    The paper combines neural HDR photometric calibration with a lightweight scaling-invariant tone mapper in an end-to-end HDR object-detection pipeline.

  • Results

    The method outperforms handcrafted tone mapping and AI-ISP baselines in automotive HDR scenes while enabling real-time processing of 4K HDR RAW data.

  • Takeaways & Limitations

    The framework supports efficient HDR object detection on embedded platforms and low-cost adaptation from LDR sRGB to HDR RAW through limited finetuning.

  • Takeaways & Limitations

    The method still requires cascading with detectors for joint processing of HDR RAW input.

Abstract

from arXiv · show

High-dynamic-range (HDR) images, with their rich tone and detail reproduction, hold significant potential to enhance computer vision systems, particularly in autonomous driving. However, most neural networks for embedded systems are trained on low-dynamic-range (LDR) inputs and suffer substantial performance degradation when handling high-bit-depth HDR images due to the challenges posed by extreme dynamic ranges. In this paper, we propose a novel tone mapping method that not only bridges the gap between HDR RAW inputs and the LDR sRGB requirements of detection networks but also achieves end-to-end optimization with downstream tasks. Instead of relying on the traditional image signal processing (ISP) pipeline, we introduce neural photometric calibration to regularize dynamic ranges and a scaling-invariant local tone mapping model to preserve image details. In addition, our architecture also supports performance transfer finetuning, enabling efficient adaptation from the LDR sRGB images to the HDR RAW images with minimal cost. The proposed method outperforms traditional tone mapping algorithms and advanced AI-ISP methods in challenging automotive HDR scenes. Moreover, our pipeline achieves real-time processing of 4K high-bit-depth HDR inputs on NVIDIA Jetson platforms.

1 Introduction

HDR RAW inputs challenge LDR-trained detection systems because extreme dynamic ranges degrade feature extraction and do not fit embedded inference pipelines. The paper proposes a lightweight, detection-optimized tone mapping framework with photometric calibration and scaling-invariant processing for real-time HDR detection.

  • Motivation: HDR scenes can reach approximately 280 dB, while sensors such as SONY IMX490 provide HDR RAW streams up to 140 dB and require real-time handling of changing illumination.The paper highlights transitions such as entering or exiting tunnels as representative lighting challenges.
  • Motivation: Directly applying LDR-trained detection networks to high-bit-depth HDR imagery causes significant performance degradation through extreme luminance contrasts and feature-extraction collapse.Traditional HDR ISP pipelines instead convert HDR RAW images into LDR sRGB equivalents by compressing dynamic range.
  • Motivation: Embedded platforms and inference engines optimized for LDR sRGB inputs struggle to represent HDR RAW dynamic ranges with hardware-friendly operators.The resulting input mismatch motivates a bridging solution that can run on embedded systems.
  • Proposed approach: The proposed framework bridges HDR RAW inputs and LDR sRGB detector requirements using lightweight tone mapping optimized specifically for HDR object detection.It is designed for efficient computation and low latency on edge platforms.
  • Proposed approach: Neural HDR photometric calibration dynamically regularizes extreme dynamic ranges through radiance-space unification.The method is intended to normalize diverse radiance distributions and improve cross-scene generalization and detection performance.
  • Results: A scaling-invariant local tone mapping model enhances detection performance while reducing runtime, and experiments demonstrate real-time processing of 4K HDR RAW videos.The end-to-end pipeline balances detection performance and computational efficiency.

2 Related Works

HDR imaging expands luminance capture but creates challenges for machine vision, especially with high-bit-depth RAW data and embedded deployment. Prior approaches include multi-exposure, single-shot, handcrafted, and AI-ISP methods, while this paper jointly optimizes tone mapping with detection.

  • High-Dynamic Range Imaging: Multi-exposure fusion recovers dynamic range from multiple exposures but is unsuitable for machine vision because of exposure latency and motion artifacts.Single-shot HDR alternatives avoid these artifacts through exposure control, spatially varying pixel exposure, or lighting modulation.
  • High-Dynamic Range Imaging: The 24-bit HDR RAW dataset captured with the IMX490 reaches 140 dB, substantially exceeding LoD and PASCAL RAW at 14-bit and RhoVision at 12-bit.This extreme dynamic range presents a considerable challenge to current computer vision systems.
  • Post-Captured Tone Mapping Algorithms: Traditional tone mapping and ISP systems compress HDR content for human perception, often including unnecessary steps and incurring computational expense.These systems are not designed primarily around downstream machine-vision requirements.
  • Post-Captured Tone Mapping Algorithms: The paper compares daytime and night HDR RAW scenes visually across different methods, reporting superior performance for its method.The caption directs readers to inspect image details by zooming in.
  • Embedded Vision: Practical automotive and robotic vision systems require optimized inference on resource-constrained embedded platforms such as NVIDIA Jetson and ARM Core.Lightweight architectures and quantization are used to support efficient edge inference.
  • Embedded Vision: Most deployed models use LDR sRGB training data, creating incompatibility with HDR RAW inputs and motivating jointly optimized tone mapping for embedded HDR perception.The paper frames its method as a bridge between HDR imaging and embedded perception.

3 Method

The method regularizes HDR dynamic ranges before detection, then applies a scaling-invariant local tone mapper optimized for detector-compatible detail preservation. Neural photometric calibration adapts to scene illumination, while end-to-end and transfer strategies support efficient HDR detection.

  • 3.1 Dynamic Range Regularization: HDR detection is analyzed using a Gaussian mixture model and a minimal CNN to relate pixel dynamic range to gradient propagation.The model represents HDR pixel histograms with Gaussian components and derives gradient behavior under MSE loss.
  • 3.1 Dynamic Range Regularization: Gradient magnitude is proportional to the square of a Gaussian component’s dynamic range, making HDR feature extraction unstable.The analysis attributes the instability to larger gradient fluctuations in HDR than LDR inputs.
  • 3.2 Neural Photometric Calibration: Neural photometric calibration learns scene-adaptive scale and bias parameters to project HDR inputs into a unified radiance space.The calibration approximates real photometric normalization when minimum and maximum scene radiance are difficult to measure directly.
  • 3.2 Neural Photometric Calibration: Log-scale prediction stabilizes calibration across day and night illumination, where sunlight radiance can be up to a million times more intense than nighttime radiance.The method predicts S := log K, resolves 10^S to the radiance scale, and upsamples the scale map.
  • 3.3 Scaling-Invariant Tone Mapper: The scaling-invariant tone mapper uses bias-free convolutional layers, ReLU, and batch normalization to preserve local relationships across dynamic ranges.The paper denotes the scaling-invariant mapper as TMSI and contrasts it with the scaling-variant mapper TM.
  • 3.4 End-to-End Optimization: The detector outputs bounding boxes, classes, and confidence scores from the tone-mapped calibrated HDR image, with training combining location and classification losses.The loss is Ltotal = Lobj. + Lclass.; transfer finetuning updates only the tone mapper and detection head while freezing remaining components.

4 Experiments

Experiments evaluate the method on HDR RAW object detection across datasets, detector architectures, scenes, efficiency metrics, and hardware. Results indicate improved detection performance, texture quality, transfer efficiency, and real-time 4K inference.

  • Experimental Setup: Evaluation uses 20,089 24-bit HDR RAW images from the RoD dataset and tests Faster R-CNN and YOLOv3 detectors.The study uses mixed scenes to assess effectiveness and generality, with implementations based on MMDetection.
  • Experimental Setup: Comparisons cover handcrafted ISP and tone mapping, end-to-end ISP, low-light enhancement, AI-ISPNet, and end-to-end tone mapping methods.Detection quality is assessed with mAR, AP50, and AP75, alongside parameters, FLOPs, and NVIDIA Jetson latency.
  • Efficiency Analysis: Downsampling-based methods reduce FLOPs but lose image details, whereas the proposed approach is reported to balance efficiency and performance.The comparison frames detail preservation as a trade-off against computational efficiency.
  • Detection Performance: 6.3% and 4.8% improvements over HDR ISP are reported with Sparse-RCNN and Deformable DETR, respectively, demonstrating generalization across downstream detectors.These results are presented for HDR RAW object detection in the advanced-detector ablation.
  • Method Analysis: The bias-free CNN enforces scaling-invariant tone mapping through TM(α · x) = α · TM(x) for x > 0.This design maps HDR images into the LDR image space while preserving behavior across positive input rescalings.
  • Texture Analysis: Image contrast and GLCM entropy improve, with object regions highlighted and texture information linked to improved detection performance.The analysis evaluates contrast and Gray-Level Co-occurrence Matrix entropy as image-texture metrics.
  • Real-Time Inference: The Lite variant processes 4K images at 22 ms, or 45 FPS, on an NVIDIA Jetson AGX Orin using 16-bit float precision.The evaluated resolution is 4096 × 2160, and a hardware prototype integrates HDR cameras with the NVIDIA Jetson platform for real-world evaluation.

5 Conclusion and Limitation

The proposed scene-adaptive tone mapping pipeline improves HDR object detection and supports real-time 4K HDR RAW processing, but joint processing still requires cascading with detectors.

  • Conclusion: The method combines neural photometric calibration, scaling-invariant tone mapping, end-to-end training, and low-cost performance transfer for HDR detection.It adapts from LDR sRGB to HDR RAW by fine-tuning only a few parameters.
  • Limitation: The current pipeline still requires cascading with detectors for joint processing of HDR RAW input.The authors identify direct HDR handling by a detector as future work.

Supplementary Material

The supplementary material documents the HDR ISP comparison pipeline, supporting proofs, additional experiments, ablations, vision comparisons, and real-world evaluation.

  • Supplementary Studies: The supplementary file introduces the included studies before listing their contents.It frames the material as a collection of supplementary investigations.
  • Supplementary Studies: Supplementary studies provide HDR ISP implementation details and proofs for dynamic-range approximation and scaling-invariant tone mapping.These materials are organized in Sections A–C.
  • Supplementary Studies: Additional comparison experiments, ablation studies, vision comparisons, and real-world evaluation are also included.The listed materials cover Sections D–F.

A HDR ISP Details

The HDR ISP comparison pipeline sequentially converts HDR RAW data into LDR sRGB through filtering, color processing, tone mapping, enhancement, and gamma correction.

  • HDR ISP Pipeline: The HDR ISP pipeline consists of sequential operations whose intermediate results show HDR data transformed into visually appealing LDR images.The key components are identified in Fig. 6 and visualized through marked modules.
  • Preprocessing: Black-level processing uses Ibl as the black-level quantity, while anti-aliasing applies a low-pass filter to prevent aliasing components from being sampled.The anti-aliasing operation uses a 5 × 5 kernel with non-zero corners and center.
  • Color Processing: Auto white balance linearly scales RGGB channels using Gray-World gains so scene grays correspond to image grays.The gain factors are derived from the Gray-World assumption.
  • Color Processing: Demosaicing converts the single-channel Bayer array into a full-resolution three-channel linear RGB image while preserving texture details.The implementation combines techniques from the Malvar algorithm.
  • Tone Mapping: Local tone mapping brightens darker areas while keeping brighter content unsaturated through exposure-fusion-style weighting.Synthetic exposure images are combined using weights calculated from image content.
  • Tone Mapping: Global tone mapping enhances overall luminance with an S-shaped contrast curve followed by the linear-to-nonlinear sRGB transfer function.The pipeline then applies gamma correction based on ITU-R BT. 709.
  • Image Enhancement: Edge enhancement accentuates finer image structures to improve image clarity and visual appeal.The EEH module is part of the series of ISP operations.

B The Proof of the Approximation to Dynamic Range

This section develops a dynamic-range approximation for Gaussian components and extends the analysis to a Gaussian mixture model. The derivation links dynamic range to component statistics and dominant terms.

  • Single Gaussian Component: For a Gaussian component with mean µ and standard deviation σ, the dynamic range is defined relative to its mean.The section introduces the component-level quantity before relating it to µ and σ.
  • Single Gaussian Component: The derivation solves for µ in terms of dynamic range d and standard deviation σ.This expresses the component mean using the defined dynamic-range quantity.
  • Inequality Analysis: As overlap between µ and σ increases, d grows and amplifies E(x^2) + E(x).The inequality analysis uses the AM-GM inequality to establish this relationship.

C Detailed Proof of Scaling-Invariant Tone Mapping

This section proves that the bias-free Conv-BN-ReLU tone mapper is scaling-invariant and functionally equivalent to a local tone mapping operator. The proof uses convolution linearity, adaptive normalization, and ReLU cascading.

  • Network Construction: The tone mapper TMSI is constructed from L bias-free Conv-BN-ReLU layers to establish functional equivalence with local tone mapping.The architecture removes bias terms before proving the desired property.
  • Scaling-Invariant Proof: Convolution acts linearly under input scaling, allowing the factor α to propagate through the network composition.The derivation explicitly identifies convolution linearity in the scaling argument.
  • Scaling-Invariant Proof: The proof concludes TMSI(α · x) = α · TMSI(x), establishing scaling invariance for the tone mapper.This equality is obtained after propagating α through the convolution, normalization, and activation layers.
  • Local Tone Mapping: Local normalization and ReLU cascades adapt gains using µi and σi, mimicking tone curves within local neighborhoods.These local operations compress highlights and shadows while preserving tones, yielding local tone-mapping behavior.

D More Ablation Studies

The ablation and visual studies examine robustness to radiance-range settings and qualitative detection behavior across HDR scenes. The reported examples emphasize fewer false detections and recovery of small objects.

  • Ablation Studies: Neural photometric calibration is not sensitive to the selected radiance range, indicating robustness across varying lighting conditions.The radiance range is treated as a hyperparameter and evaluated through an ablation study.
  • Visual Comparison: Figures 8 and 9 compare methods on HDR RAW day and night scenes using detection confidence scores and class predictions.The visual comparisons include separate rows for day and night scenarios.
  • Visual Comparison: In a tunnel scene, the proposed method reduces false detections, while in another HDR scene it detects small objects missed by other methods.The visual results use detections with confidence scores greater than 0.3 from different RoD dataset scenarios.

F Real World Evaluation.

The real-world evaluation collects HDR RAW driving videos and tests the complete detection pipeline on an NVIDIA Jetson AGX Orin. The setup uses YOLOv3 and evaluates 2K inputs that are resized to 4K for processing.

  • HDR Video Validation: HDR RAW video sequences from autonomous-driving scenes are collected to validate the proposed method.The validation uses YOLOv3 as the base detector.
  • HDR Video Validation: The complete pipeline is evaluated on an NVIDIA Jetson AGX Orin using 16-bit float processing.The evaluation targets an embedded platform representative of the paper’s deployment setting.
  • HDR Video Validation: The evaluation uses 2K inputs at 2048 × 1080 resolution and resizes the HDR RAW input to 4K at 4096 × 2160 for processing.A supplementary video demo showcases the detection results, with sequences and annotations planned for release after dataset completion.
Loading 2608.30400v1…