Double-stream registration with pyramid fusion for HDR video with alternating exposures

1. INTRODUCTION

• Alternating-exposure LDR sequences offer a practical, cost-effective route to HDR video, but fusing their frames remains ill-posed.Consumer cameras capture a narrow dynamic range, with detail loss, saturation in overexposed regions, and noise in underexposed areas.Source: §1 · p. 1
• Methods that register frames to a medium-exposure reference cannot directly handle video where each frame must serve as reference, leading to temporal inconsistency and flickering.Specialized HDR hardware can capture HDR video directly but remains expensive and difficult to include in everyday devices.Source: §1 · p. 1
• Deep-learning HDR video methods have developed from attention and HDR-supervised flow to exposure completion, alignment-free exposure representations, and physically guided motion fields.The cited approaches include feature-space attention, HDR-supervised optical flow, exposure interpolation, exposure-independent features, and cross-exposure motion fields.Source: §1 · p. 1
• The proposed method registers three consecutive frames using direct optical flow and a complementary midpoint displacement, then fuses five radiance and LDR images with a pyramid.The midpoint displacement is intended to improve registration when the central frame has a long exposure; the approach assumes approximately linear per-pixel flow across consecutive frames.Source: §1 · p. 1

2.1. Classical methods for HDR Video Reconstruction

• HDR video methods commonly align neighboring frames to a reference and merge them, refining either registration or the merging strategy.The first method described combines global transformation and optical flow, then uses a weighted average accounting for exposure and flow errors.Source: §2.1 · p. 2
• Classical approaches improve cross-exposure registration or formulate reconstruction as energy minimization.The section directs readers to broader reviews of classical and deep-learning HDR imaging.Source: §2.1 · p. 2, §2 · p. 2

2.2. Deep learning methods for HDR Video Reconstruction

• Deep-learning methods progress from multi-scale flow and blending to deformable convolutions [19], saturated-region recovery, and HDR-domain flow training.Other approaches interpolate missing LDR frames before fusion, bypass explicit alignment, or refine generic flow to suppress ghosting.Source: §2.2 · p. 2

3. METHOD

• The target frame is reconstructed from the current frame and its previous and following frames; the approach can be extended to more exposures and frames.Input is a multi-exposure video sequence with two different exposures.Source: §3 · p. 2
• Radiance frames are obtained by inverting the camera response function or applying gamma correction, then dividing by exposure time for comparison and combination.The cited response-function inversion is attributed to Devebec and Malik, and the gamma-based approach to.Source: §3 · p. 2
• Two complementary strategies register each temporal neighbor to the central frame: direct flow between neighbor and center, and half the flow between the two outer frames.The latter supplies midpoint motion vectors from the outer-frame displacement.Source: §3 · p. 2
• A multiscale pyramid fuses the registered radiance and LDR images into output radiance, combining details at each scale with a U-Net architecture.The pyramid fusion uses a Laplacian-pyramid approach and Grid-Net, described as a generalization of U-Net.Source: §3 · p. 2
• Flow registration first calibrates the darker image photometrically, then computes flow with PWC-Net and forward-warps both LDR and radiance images.The method uses differentiable µ-law tone mapping and an L1 reconstruction loss.Source: §3 · p. 2

4.1. Experimental setup

• Training data comes from Vimeo-90K with simulated alternating exposures, while evaluation uses Vimeo-90K and Cinematic Video sequences.Training selects three consecutive frames and extracts 128×128 patches; evaluation simulates alternating exposures from three consecutive frames in the Cinematic Video HDR sequences.Source: §4.1 · p. 3

4.2. Comparsion with state-of-the-art

• The method achieves state-of-the-art performance across Vimeo metrics and the highest Stuttgart long-exposure PSNR, with the second-best result on Stuttgart short exposures.LAN-HDR and Deep-HDR use five consecutive frames, and Deep-HDR was trained on Stuttgart data; the short-exposure ground truth is noisy, making PSNR less reliable.Source: §4.2 · p. 3, Table 1 · p. 4
• The Stuttgart short-exposure ground truth contains noisy regions, so its PSNR values are less reliable.Figure 3 shows a short-exposure frame extract and corresponding ground truth.Source: §4.2 · p. 3, Figure 3 · p. 4
• Visual comparisons report correct mid-flow alignment even when the reference frame is saturated, including the woman’s face in Vimeo and carousel lights in Stuttgart.The figures show processed images and errors relative to ground truth.Source: §4.2 · pp. 3–4, Figure 2 · p. 4

• The conclusion presents a novel approach to HDR video from alternating-exposure sequences, with a scheme producing the final HDR output.Source: §5 · p. 4
• Dual-stream registration adaptively leverages the most reliable alignment information at each pixel and substantially improves performance over state-of-the-art HDR reconstruction methods.Source: §5 · p. 4
• Future work may explore more exposure levels, learning-based flow tailored to video HDR, realistic noise modeling, and large displacement.Source: §5 · p. 4