Source-linked AI summary
High Dynamic Range Video Reconstruction from Single-Exposure Raw Sequences
Tao Zhang, Peixian Su, Xingyu Gao, Yunhao Zou, Yu Lu, Zunjie Zhu, Bolun Zheng, Ying Fu, Chenggang Yan
TL;DR
Single-exposure Raw-video HDR is difficult because conventional sensors clip highlights and lose shadow detail, and existing sRGB processing discards radiometric information. RawHDRV combines Bayer-channel processing, temporal exposure complementarity, and a per-frame-annotated mobile Raw/HDR dataset; the authors report best performance across six evaluation metrics.
Problem
Single-exposure HDR must recover clipped highlights and noisy shadows without differently exposed observations, while sRGB processing loses radiometric information in extreme regions.
Source: §I · pp. 1–2Method
RawHDRV processes Bayer channels with parallel alignment and exposure-aware fusion, then uses mask-guided restoration and a mobile Raw/HDR dataset with per-frame annotations.
Source: §I · p. 2Results
RawHDRV achieves the best performance across six evaluation metrics, surpassing the second-best method by 2.54 dB in PSNR-L, 2.39 dB in PSNR-µ, and 1.36 in HDR-VDP-2.
Source: B. Evaluation Results on Our Dataset · p. 8Takeaways & Limitations
The per-frame-annotated mobile Raw/HDR dataset provides a benchmark for training and evaluating single-exposure Raw-video HDR reconstruction.
Source: §I · p. 2
Abstract
from arXiv · showhide
Due to the limited dynamic range of conventional image sensors, captured low dynamic range (LDR) video often suffers from highlight clipping and shadow detail loss, making high-quality high dynamic range (HDR) reconstruction from single-exposure sequences highly challenging without alternating exposures or extra hardware. Alternating-exposure HDR methods sacrifice frame rate and struggle with motion alignment, making them impractical for real-world capture. To address this, we propose RawHDRV, an end-to-end framework for single-exposure Raw video HDR reconstruction, that fundamentally exploits the linear response and channel-specific characteristics of Bayer data. Specifically, it features a channel-decomposition temporal alignment and fusion strategy that processes Bayer channels separately to exploit their distinct exposure characteristics, together with exposure-aware weighted fusion. It further incorporates an exposure complementarity mask-guided restoration module that leverages inter-frame exposure redundancy to adaptively fuse reliable information and suppress saturation artifacts, and introduces a mask-guided color loss that combines normalized error constraints with gradient smoothing to enhance highlight recovery. Furthermore, we construct a large-scale mobile Raw-HDR video dataset with per-frame HDR annotations. Experiments show that our method achieves the state-of-the-art results in all metrics, demonstrating superior spatial quality and temporal stability under extreme exposure conditions. The code is available at https://github.com/supeixian/RawHDRV.
I. INTRODUCTION
Conventional sensors lose highlight and shadow detail, while alternating-exposure video HDR can reduce temporal resolution and produce motion-related artifacts. The paper proposes single-exposure Raw-video HDR reconstruction, supported by a new annotated dataset and reported state-of-the-art performance.
I. INTRODUCTION
- Alternating-exposure video HDR reduces effective temporal resolution and can produce motion blur, ghosting, and color fringing in dynamic scenes –.The paper contrasts this with single-exposure capture, which preserves a 1:1 mapping to the sensor capture rate.Source: §I · p. 1, Table I · p. 2
- Single-exposure Raw-video HDR remains challenging because persistent over- or under-exposure limits temporal recovery, while sRGB processing discards radiometric information.Raw measurements preserve richer scene information through their linear response and higher bit depth, but large-scale Raw-video datasets with per-frame HDR ground truth have been lacking.Source: §I · p. 2
- RawHDRV combines Bayer-channel parallel alignment and exposure-aware fusion with mask-guided complementary restoration to recover detail and suppress saturation artifacts.Its four-channel processing accounts for distinct noise and saturation behavior, while the restoration scheme weights inter-frame regions and the training loss targets highlight recovery.Source: §I · p. 2
- The proposed method is reported to achieve state-of-the-art performance.The paper’s Fig. 1 contrasts alternating-exposure color fringing with the single-exposure approach’s more faithful HDR reconstruction in challenging RawHDRV scenes.Source: Figure 1 · p. 1, §I · p. 2
- The authors establish a mobile Raw/HDR video dataset with per-frame annotations as a benchmark for single-exposure Raw-video HDR training and evaluation.They describe it as the first benchmark for this task.Source: §I · p. 2
II. RELATED WORK
The related work reviews image-based HDR reconstruction, including multi-exposure fusion, inverse tone mapping, hardware-assisted capture, and Raw-domain methods.
II. RELATED WORK
- The section frames its review around image-based HDR reconstruction and Raw-domain processing alongside related datasets.The supplied discussion chiefly details HDR image reconstruction approaches and sensor-data options.Source: §II · p. 2, A. HDR Image Reconstruction · pp. 2–3
A. HDR Image Reconstruction
- Multi-exposure fusion combines differently exposed LDR images to expand luminance range, while reverse tone mapping reconstructs HDR from one LDR image, –.Deep-learning approaches include end-to-end CNNs and methods that synthesize pseudo multi-exposure stacks for fusion.Source: A. HDR Image Reconstruction · p. 2
- Specialized hardware can capture additional dynamic-range or motion cues, while high-bit-depth Raw data preserves linear sensor responses for single-image HDR reconstruction, –.The cited Raw-domain work demonstrates the potential of unprocessed sensor data.Source: A. HDR Image Reconstruction · pp. 2–3
B. HDR Video Reconstruction
Alternating-exposure video HDR faces temporal-resolution and alignment challenges, while single-exposure Raw video reconstruction must recover clipped highlights and noisy shadows from temporal and channel-level complementarity. The section motivates RawHDRV’s channel-aware alignment and mask-guided restoration against a backdrop of scarce annotated Raw-video data.
B. HDR Video Reconstruction
- Alternating-exposure HDR video methods fuse differently exposed frames but face registration difficulty and ghosting under large inter-frame brightness differences,,,.Single-exposure capture instead uses temporal redundancy while maintaining consistent exposure across consecutive frames.Source: B. HDR Video Reconstruction · p. 3
C. Raw Processing and Video HDR Dataset
- Raw sensor data preserves richer radiometric information through its linear response and higher bit depth, but single-exposure Raw-video datasets with per-frame HDR ground truth remain scarce.The lack of such data limits supervised learning and fair benchmarking for the task.Source: C. Raw Processing and Video HDR Dataset · p. 3
A. Problem Formulation and Motivation
- Single-exposure Raw sequences remain difficult because clipping and shadow noise persist where exposure diversity is absent and regions stay over- or under-exposed.Bayer channels offer different cues: green pixels have higher photon capture efficiency and signal-to-noise ratio, while red and blue retain more highlight information; motion can also make a saturated pixel well-exposed in a neighboring frame.Source: A. Problem Formulation and Motivation · pp. 3–4
- RawHDRV uses channel-decomposition temporal alignment and fusion with exposure-complementarity mask-guided restoration to combine reliable information across channels and frames.The network overview places channel-specific alignment and exposure-aware fusion before restoration of overexposed details.Source: §III · p. 3, A. Problem Formulation and Motivation · p. 4, Figure 3 · p. 4
B. Model Architecture
The architecture separates Bayer channels for temporal alignment and exposure-aware fusion, uses complementary exposure masks to guide restoration, and reconstructs the center HDR frame with a multi-scale encoder-decoder.
B. Model Architecture
- Three consecutive Raw frames are aligned and fused, restored with exposure-complementarity masks, then reconstructed as a center HDR frame through a multi-scale encoder-decoder.The encoder-decoder uses three downsampling levels, Transformer Blocks, skip connections, refinement, and a residual connection.Source: B. Model Architecture · pp. 4–5
- The channel-decomposition framework uses an all-channel stream for global context, an RB stream for highlight detail, and a dual-G stream for dark-area denoising.The framework splits the RGGB input into three parallel streams.Source: B. Model Architecture · p. 4
- Optical flow warps adjacent Raw-domain features toward the center frame, while separate RB and G cross-frame attention mitigates residual alignment errors.Separate attention preserves channel-specific characteristics and local exposure differences.Source: B. Model Architecture · p. 4
- Differentiated channel weighting enhances RB features in highlights, boosts G features in shadows, and balances their contributions in normal regions.The weights are applied through element-wise multiplication.Source: B. Model Architecture · p. 4
- The restoration module partitions warped overexposure masks into regions and weights center and neighbor features to use valid neighbor detail and suppress overexposed interference.Complementary regions contain one overexposed frame and one normal frame; doubly overexposed areas are attenuated to prevent error propagation.Source: B. Model Architecture · pp. 4–5
- Bidirectional enhanced features are concatenated with weighted RB and G features, integrated by convolution, and refined through the multi-scale encoder-decoder.The decoder's skip connections preserve fine structural details.Source: B. Model Architecture · p. 5
C. Loss Function
The loss function combines brightness and highlight-aware supervision, including a mask-guided color loss that focuses on clipped regions and smooths their boundaries.
C. Loss Function
- Three complementary losses provide brightness consistency, reduce highlight dominance in the linear domain, and focus supervision on overexposed regions.L1 loss supports brightness consistency, log-L2 operates in logarithmic space, and the mask-guided color loss targets highlights.Source: C. Loss Function · p. 5
- With ϵ = 1/5000, the log-L2 loss balances dark and bright regions to improve mid-tone detail recovery.The loss operates in logarithmic space to mitigate highlight dominance in the linear domain.Source: C. Loss Function · p. 5
- The mask-guided color loss supervises overexposed regions with a mask-normalized L1 term and a gradient-smoothness term for spatial continuity.The masked error is averaged over the overexposed area, while the gradient term reduces artifacts at highlight boundaries.Source: C. Loss Function · p. 5
- The mask-guided loss improves PSNR-µ in overexposed regions by ∼1.1 dB and yields visually smoother, more natural results.The reported weights are λo = 2.0 and λs = 0.1.Source: C. Loss Function · p. 5
IV. THE PROPOSED RAWHDRV DATASET
RawHDRV pairs mobile Bayer Raw video inputs with per-frame HDR ground truth, generated from seven exposures captured per timestamp. The dataset spans varied lighting and motion, including substantial measured motion.
IV. THE PROPOSED RAWHDRV DATASET
- Each 10-bit Bayer Raw frame is captured at 3072 × 4096 resolution, with seven exposure values from EV −3 to +3 per timestamp and per-frame HDR ground truth.The HDR ground truth is generated by multi-exposure fusion; the standard EV0 exposure is the input and the ground truth is calibrated to its photometric baseline.Source: §IV · p. 5, Figure 4 · p. 6
- 400 sequences yield 2,455 aligned Raw-HDR pairs, with 244 daytime and 156 nighttime sequences.Sequences contain 5–8 consecutive frames.Source: §IV · p. 6
- 66.6% of patches show motion of at least 10 pixels, including 17.6% with motion of at least 30 pixels; 28.15% are near-static at under 2 pixels.Motion was assessed using patch-level optical flow; the dataset includes global, local, and full motion patterns across four illumination conditions.Source: §IV · pp. 5–6, Table III · p. 6, Figure 5 · p. 6
- RawHDRV is presented as the first large-scale Raw video dataset designed for single-exposure HDR reconstruction with per-frame HDR ground truth.The paper contrasts it with prior dataset work focused on sRGB multi-exposure HDR video or single-image Raw-to-HDR, and notes Yue et al.'s Raw video dataset still uses alternating exposures.Source: §IV · pp. 6–7
- The dataset also supports multi-exposure video HDR, multi-exposure image fusion, and single-image Raw HDR.For Table II, “dataset format” denotes the input format used during training in the corresponding paper.Source: §IV · pp. 6–7
V. EXPERIMENTS
The experiments comprise implementation and comparison details, quantitative and qualitative evaluation on RawHDRV, and component ablations.
V. EXPERIMENTS
- The experiments evaluate the proposed method quantitatively and qualitatively on RawHDRV.The section also sets out implementation details, comparison methods, and evaluation metrics.Source: §V · p. 7
- The evaluation is followed by ablation studies that analyze each component's contribution.Source: §V · p. 7
- Implementation details, comparison methods, and evaluation metrics form the experimental setup.Source: §V · p. 7
A. Experimental Settings
The evaluation compares retrained image- and video-restoration models under shared training settings and uses six metrics spanning HDR image quality and temporal stability.
A. Experimental Settings
- All compared methods are retrained on RawHDRV using the same settings; training uses three consecutive EV0 Raw frames and the middle frame as target.Inputs are normalized in Raw and cropped to 128 × 128 patches; training runs for 100 epochs with Adam, batch size 1, and an initial learning rate of 1 × 10^-5.Source: A. Experimental Settings · p. 7
- The comparison covers image-based HDR methods and video-restoration models, adapted to four-channel Bayer Raw input where needed.The evaluation uses six metrics: PSNR-L, SSIM-L, PSNR-µ, MS-SSIM, HDR-VDP-2, and HDR-VQM; HDR-VQM measures temporal stability, with lower scores indicating better consistency.Source: A. Experimental Settings · pp. 7–8
B. Evaluation Results on Our Dataset
On RawHDRV, the proposed method leads the reported quantitative comparison and recovers detail in over- and underexposed regions. The section also includes visual comparisons and component ablations.
B. Evaluation Results on Our Dataset
- Evaluation uses 350 training sequences and 50 test sequences, with identical splits, resolutions, and protocols across methods.The test set contains 30 daytime and 20 nighttime scenes.Source: B. Evaluation Results on Our Dataset · p. 8
- 2.54 dB in PSNR-L, 2.39 dB in PSNR-µ, and 1.36 in HDR-VDP-2 are the proposed method's advantages over the second-best method; it achieves the best performance across six metrics.The authors characterize the gains as spanning pixel fidelity and perceptual HDR realism.Source: B. Evaluation Results on Our Dataset · p. 8
- The proposed method recovers details in overexposed and underexposed regions, while other models show highlight color shifts and noisy shadows.The visual comparisons are tone-mapped.Source: B. Evaluation Results on Our Dataset · p. 8
- The visual comparison in Fig. 10 covers exposure values EV−2, EV0, and EV+2.Source: Figure 10 · p. 9
- Across three consecutive frames, the method reconstructs dynamic overexposed regions by fusing reliable neighboring-frame information, outperforming HDRUNet and RealViformer.Source: B. Evaluation Results on Our Dataset · p. 9
- The ablation study presents visual and tabular comparisons of the proposed components, including channel-decomposition parallel alignment and mask-guided restoration.Source: Table V · p. 9, Figure 9 · p. 9
C. Ablation Studies
Ablations on RawHDRV vary one module or loss at a time under consistent training and evaluation settings. Channel decomposition, softened mask-guided fusion, and the full loss each improve reconstruction in distinct ways.
Channel-Decomposition Parallel Alignment
- The full X+RB+G configuration achieves the best results across all metrics, with RB improving highlight recovery and G enhancing shadow detail.Using only the X branch yields the weakest performance, especially in extreme exposure regions.Source: C. Ablation Studies · p. 9
Loss Function
- The full loss with mask-guided color loss achieves the best metrics, recovering more highlight detail and reducing boundary artifacts.LL1 alone converges stably but recovers highlights poorly; adding Llog-L2 balances bright and dark regions in the log domain and smooths tone transitions.Source: Table VI · p. 9, Figure 11 · p. 9, C. Ablation Studies · p. 9
Exposure Complementarity Mask-Guided
- Gaussian-blurring the hard mask produces smoother weighting that suppresses saturation-boundary artifacts and achieves the best performance.The hard binary mask introduces color artifacts near saturation boundaries.Source: C. Ablation Studies · p. 9
D. Cross-Sensor Generalization
Zero-shot testing on Sony IMX921 reaches 37.42 dB PSNR-µ without fine-tuning, while retrained models adapt to under- and overexposed inputs. The conclusion reports consistent gains over existing single-image and video restoration methods on RawHDRV.
D. Cross-Sensor Generalization
- 37.42 dB PSNR-µ without fine-tuning, outperforming RealViformer by 1.94 dB on 10 additional Sony IMX921 sequences.Standardized preprocessing and the shared Bayer structure are used for this zero-shot test.Source: D. Cross-Sensor Generalization · p. 10
E. Extended Evaluation of RawHDRV
- Retrained models restore standard brightness and suppress noise for EV −2 inputs, while EV +2 inputs retain inevitable clipped-region information loss despite plausible texture inference and mitigated color artifacts.The evaluation maps non-standard exposure inputs to standard HDR ground truth.Source: Table VIII · p. 10, E. Extended Evaluation of RawHDRV · p. 10
VI. CONCLUSION
- On RawHDRV, the method consistently outperforms existing single-image and video restoration methods in highlight and shadow recovery.The conclusion describes separate Red-Blue and Green streams fused with exposure-aware differential weighting, alongside neighboring-frame restoration of saturated regions.Source: §VI · p. 10