Source-linked AI summary
HaRP: High Dynamic Range Photosequencing through Dual Reversed Shutter Scanning
Xiang Ji, Guixu Lin, Jiancheng Zhao, Zhengwei Yin, Yinqiang Zheng
TL;DR
Rolling-shutter distortions affect dynamic scenes, and RSGR also limits capture speed and dynamic range. The paper pairs reversed RSGR views with a reconstruction framework and real-data capture system; experiments report improved detail recovery and reduced flicker, with at least 1.5 dB PSNR advantage over existing approaches across diverse setups.
Problem
Rolling-shutter effects degrade appearance and geometry in dynamic scenes, while RSGR sacrifices early scanning and compresses dynamic range, and existing HDR methods remain vulnerable to fast-motion distortions and ghosting.
Source: §1 · p. 1Method
The paper synchronizes complementary RSGR and inverted-RSGR views and reconstructs HDR sequences using row-adaptive alignment, hallucination, and a real paired-data capture system.
Source: §1 · p. 2, §3.2 · p. 4Results
At least 1.5 dB on PSNR, the method outperforms existing approaches across diverse setups, with improved detail recovery and reduced temporal flickering.
Source: §4.1 · p. 7, §5 · p. 9Takeaways & Limitations
Dual-reversed RSGR photosequencing offers a supported approach to expanding dynamic range and reconstructing detail under motion degradation.
Source: §5 · p. 9
Abstract
from arXiv · showhide
The adoption of CMOS sensors in mobile photography is frequently compromised by the rolling shutter (RS) effect, which introduces geometric distortions and motion artifacts. Particularly, recent rolling shutter with global reset (RSGR) mode, while mitigating some RS issues, also incurs major limitations, including reduced capture speed and compressed dynamic range. To address these problems, we propose a novel dual reversed scanning setup utilizing both RSGR and inverted RSGR views. This solution not only handles the inherent flaws of RSGR by synchronizing complementary exposures to balance the dynamic range across the frames but also introduces an effective method for HDR photosequencing under highly dynamic scenes. Our proposed network first accommodates row-wise complementarity and manages visual shifts by row-adaptive feature alignment. Subsequently, the hallucination module, built upon a correlation-guided mixattention block, integrates the mutually reinforced features to recover missing details. In addition, we construct a coaxial imaging system to collect a real-world dataset, enabling robust training and evaluation beyond numerical simulation. Experimental results demonstrate the twofold benefits of our solution in mitigating RSGR limitations and advancing HDR reconstruction techniques.
1. Introduction
Rolling-shutter distortions complicate imaging of dynamic scenes, while RSGR trades some rolling-shutter issues for reduced capture speed and compressed dynamic range. The paper proposes synchronized, reversed RSGR views and a supporting network and imaging system for HDR photosequencing.
1. Introduction
- RSGR sacrifices early scanning of the next frame and compresses dynamic range through row-varying exposures.Downward scanning can underexpose top rows and overexpose bottom rows.Source: §1 · p. 1
- Existing HDR methods based on alternating-exposure LDR sequences and motion compensation or feature alignment remain vulnerable to distortion and ghosting under fast motion.Source: §1 · p. 1
- The dual-reversed RSGR configuration uses complementary row-wise exposures to address RSGR’s frame-rate and dynamic-range limitations in dynamic scenes.The views scan in opposite directions; corresponding rows provide complementary exposures, with visual shifts under motion described as typically minor.Source: §1 · p. 2
- The network combines row-adaptive feature alignment for visual shifts with a correlation-guided mix-attention hallucination module for detail recovery.The alignment adapts to row-wise complementarity; the hallucination module generates details, especially in under- or over-exposed rows.Source: §1 · p. 2
- The framework also uses a tri-pivot temporal loss on luminance to emphasize geometric consistency and suppress color variations.Source: §1 · p. 2
- A specialized imaging system captures RSGR-iRSGR pairs alongside corresponding high-speed HDR videos, forming a real dataset of outdoor dynamic scenes.Source: §1 · p. 2
2. Related Work
Prior rolling-shutter work spans geometric correction, learned motion estimation, and reconstruction of latent sharp frames, while HDR work develops alignment and fusion methods for motion-varying LDR inputs. The paper positions dual-reversed RSGR as a simple-to-implement alternative for robustness in dynamic scenes.
2.1. Rolling Shutter Correction and Interpolation
- Earlier rolling-shutter correction methods rely on scene-structure or camera-motion assumptions, while later learned methods estimate motion or reconstruct latent frames but can remain fragile under large or complex motion.RS correction commonly shifts scanlines toward a virtual global-shutter frame using estimated flow or hallucinated context.Source: §2.1 · pp. 2–3
- Albl et al. (Albl et al., 2020) introduce a dual-reversed RS configuration that adds geometric constraints from sparse correspondences.Source: §2.1 · p. 3
2.2. High Dynamic Range Imaging
- Traditional HDR pipelines align motion-varying LDR frames to avoid ghosting, but rejection can discard useful information and optimization-based methods can be time-consuming and sub-optimal.Source: §2.2 · p. 3
- Deep HDR methods reduce misalignment and ghosting through flow-guided fusion, image translation, coarse-to-fine alignment, and Transformer or luminance-based approaches.Source: §2.2 · p. 3
- Unlike event-camera HDR systems described as costly or vulnerable to large motions, the paper presents dual-reversed RSGR as simple to implement and more robust in dynamic scenes.Source: §2.2 · p. 3
3. Methodology
HaRP combines a real-world dual-reversed RSGR dataset with a two-stage HDR photosequencing network. The network aligns complementary views at multiple row scales, then reconstructs details and promotes temporal consistency.
3.1. Optical System and RSGR-HDR Dataset
- The quad-axis imaging system captures aligned RSGR and inverted RSGR pairs alongside high-speed global-shutter frames for HDR ground truth.The RSGR-HDR dataset contains 90 urban-scene videos at 640×480, each with 70 paired degraded frames and 1,400 aligned high-speed HDR frames.Source: §3.1 · pp. 3–4
3.2. Network Architecture
- The network uses row-shaped attention at multiple scales to align reversed RSGR views, adapting to visual shifts under dynamic scenes.The row-adaptive approach addresses limitations of dense attention and fixed-scale row-wise alignment.Source: §3.2 · p. 4
- A dual-stream correlation-guided mix-attention block combines self-similar and cross-stream features to recover fine details in under- or over-exposed regions.The reconstruction block outputs a high-speed HDR sequence within the input exposure duration.Source: §3.2 · pp. 4–5
- The tri-pivot temporal loss compares luminance-channel difference maps at three temporal pivots with corresponding ground truth to promote structural and temporal consistency.The loss uses the Y channel of YCrCb to emphasize structures and details over color information.Source: §3.2 · pp. 5–6
- Figure 5 presents qualitative comparisons on RSGR-HDR and reports that the model outperforms existing approaches in detail.Source: Figure 5 · p. 6
4. Experiments
Experiments compare the method with prior approaches, ablate its components, test robustness to shifted views, and evaluate it on synthetic GOPRO-Dual data. The reported results show strong comparisons on RSGR-HDR and no significant drop under tested view shifts.
4.1. Comparison with SOTA methods
- At least 1.5 dB on PSNR: our method outperforms existing approaches across diverse setups and has the highest perceptual and fidelity scores while remaining competitively efficient.The comparisons use models retrained on RSGR-HDR; inference time is measured on an RTX 4090 at 640 × 480.Source: §4.1 · p. 7
4.2. Ablation Study
- Removing feature alignment or replacing row-adaptive alignment lowers performance across metrics; the proposed tri-pivot loss contributes substantially, while sRGB temporal-loss gains remain limited.The ablations also report that a Swin Transformer block fails to capture complementarity between RSGR-iRSGR views, and row-shape experiments evaluate different row-size and stride combinations.Source: Figure 6 · p. 7, §4.2 · p. 8
4.3. Visual Shifts and Convergent Exposures
- Under tested misalignments of up to 4, 6, or 8 pixels, performance shows no significant drop; motion-induced displacement also helps alleviate central exposure convergence.The comparisons use retrained models on misaligned RSGR-iRSGR pairs.Source: §4.3 · p. 8, Table 4 · p. 8, Figure 7 · p. 8
4.4. Comparison on Synthetic Data
- The method is also evaluated on synthetic GOPRO-Dual data, with quantitative and qualitative comparisons using the RSGR-HDR evaluation protocol and PSNR, SSIM, LPIPS, and TMQI.Source: §4.4 · p. 8, Table 5 · p. 8
5. Conclusion
The paper introduces HDR photosequencing from dual reversed RSGR inputs to address RSGR's frame-rate and dynamic-range limitations, and reports advantages on synthetic and real data.
5. Conclusion
- Dual reversed RSGR inputs define a new HDR photosequencing task aimed at RSGR's frame-rate and dynamic-range limitations.The framework exploits row-wise complementary exposures for feature alignment and dynamic-range expansion under large motions.Source: §5 · p. 9
- The system combines row-adaptive alignment, mix-attention hallucination, and tri-pivot structural supervision to improve detail recovery and reduce temporal flickering.Source: §5 · p. 9
- Experiments on synthetic and real data validate the setting and model, with reported advantages over existing solutions.Source: §5 · p. 9
A. Implementation Details
The implementation combines RSGR and inverted RSGR views with two global-shutter cameras in a coaxial system, aligning sensor viewpoints computationally and capturing complementary exposures for HDR.
A. Implementation Details
- The two GS cameras' inter-frame deadtime is described as extremely short and ignorable.Source: Table A.1 · p. 13
- The system uses RSGR, iRSGR, and two global-shutter cameras, with all four devices operating at 640×480 resolution.RSGR and iRSGR use 20 ms exposures and 20 ms deadtime at 25 fps; the GS cameras use 2 ms exposures at 500 fps with no deadtime.Source: A. Implementation Details · p. 13
A.1. Construction of Imaging system
- Three homographies under a closed-loop constraint computationally synchronize the four sensor viewpoints after physical alignment proved challenging.Source: §A.1 · p. 13
- The global-shutter camera reduces noise through 9µm × 9µm effective pixels formed by combining four subpixels and active cooling to 0°C.Source: §A.1 · p. 13
- A 12.5 mm lens and 1:1 relay yield 26.1° × 19.5° fields of view, covering 4.6 m × 3.4 m at 10 m.The high-grade lens is selected to minimize lens-distortion artifacts that may interact with rolling-shutter effects.Source: §A.1 · p. 13
- A neutral-density filter attenuates one GS camera's input so two GS frames record different illumination levels without changing exposure duration.The complementary exposures are merged into an HDR image.Source: §A.1 · p. 13
A.2.1. REAL DATASET
RSGR-HDR contains aligned rolling-shutter views and high-speed sharp HDR sequences from 90 urban scenes, with each paired input corresponding to ten HDR ground-truth frames.
A.2.1. REAL DATASET
- RSGR-HDR contains 90 urban scenes, each with 70 RSGR–iRSGR input pairs and 1,400 aligned high-speed sharp frames.Source: §A.2.1 · p. 13
- Each RSGR–iRSGR pair spans a 40 ms cycle, while synchronized GS cameras capture 20 pairs at 500 fps, yielding ten HDR frames within the exposure window.The remaining ten GS pairs fall within the 20 ms deadtime; each RSGR–iRSGR pair corresponds to ten HDR ground-truth frames.Source: §A.2.1 · p. 13
- The mutually inverted scan directions provide exposure compensation and supplemental details and cues for HDR reconstruction.Source: §A.2.1 · p. 13
- The dataset is split into 54 training, 12 validation, and 24 testing sequences.Source: §A.2.1 · p. 13
A.2.2. SYNTHETIC DATASET
GOPRO-Dual provides supplementary synthetic validation using aligned, oppositely scanned RSGR views; training uses nine-frame HDR clips, and comparisons include reconstructed HDR frames and sequences.
A.2.2. SYNTHETIC DATASET
- GOPRO-Dual is supplementary independent validation, while the paper's main training and evaluation use the real RSGR-HDR dataset.The authors note that synthetic data cannot fully capture real-world scene complexity and diversity.Source: §A.2.2 · pp. 13–14
- The synthetic data derive from 33 GOPRO videos and increase their frame rate by 64× using video interpolation.Each source video has 1,200 frames at 240 fps and 1280 × 720 resolution.Source: §A.2.2 · p. 14
- The synthetic method generates frame-level-aligned RSGR and iRSGR views by weighted-summing latent sharp-frame patches, with scans running in opposite directions.Centrally cropped GOPRO frames are 512 × 512; the samples illustrate normalized timestamps or ground-truth frames at the stated instants.Source: §A.2.2 · p. 14, Figure A.3 · p. 14, Figure A.2 · p. 14
- Nine frames are uniformly sampled from the 512 sharp frames corresponding to each RSGR or iRSGR frame for supervised learning.Source: §A.2.2 · p. 15
- Comparisons include event-assisted EFI, EvUnroll, and EBFI and alternating-exposure LAN, with event streams simulated and alternating-exposure inputs generated from HDR frames.The GOPRO-Dual table covers single-frame HDR reconstruction and HDR photosequencing of lengths 3, 5, and 9, evaluated by mean PSNR, SSIM, LPIPS, and TMQI.Source: §A.2.2 · p. 15, Table A.2 · p. 15
A.2.3. TRAINING DETAILS
- Training uses nine-frame HDR clips, Adam with an initial learning rate of 10^-4, cosine annealing, and 800 epochs; the paper reports qualitative detail-recovery and dynamic-range gains on GOPRO-Dual.Training crops are 256 × 256, with vertical and horizontal flips; the qualitative comparison is shown for GOPRO-Dual.Source: §A.2.3 · p. 15, Figure A.5 · p. 16
B.1. Experimental Results on Synthetic Data
The section reports comparisons between the proposed model and selected state-of-the-art methods on synthetic data, with quantitative and qualitative results provided.
B.1. Experimental Results on Synthetic Data
- The model is compared with selected state-of-the-art methods on synthetic data.The complete quantitative and qualitative results are reported in Table A.2 and Figure A.5, respectively.Source: §B.1 · p. 16
B.2. Third-party Evaluation
Third-party evaluations report strong generalization on two testsets, while video comparisons assess reconstructions across multiple consecutive HDR frames.
B.2. Third-party Evaluation
- On Adobe-Dual and LiU-HDRv-Dual, the method achieves the best overall performance and consistently reconstructs fine temporal details with a wide dynamic range.The testsets assess generalization to varied scenes, motions, synthesis parameters, and illumination conditions, including conventional HDR scenes with challenging underexposed and overexposed regions.Source: §B.2 · p. 16, Table B.4 · p. 17, Figure B.7 · p. 17
- The reported advantage is attributed to dual reversed inputs’ row-wise complementarity and the proposed row-adaptive alignment and correlation-guided hallucination modules.The passage contrasts BiT with BiTm and BiTm with Ours.Source: §B.2 · pp. 16–17
B.3. Video reconstruction results
- Across consecutive latent frames, EFI misses waving-hand motion, while PMBNet, EvUnroll, and DeMFI introduce artifacts; the proposed results are clearer and closer to ground truth than BiTm.The passage reports a consistent visual observation on real RSGR-HDR data.Source: §B.3 · p. 17
B.4. Additional Qualitative Results
The section provides additional qualitative comparisons on RSGR-HDR and GOPRO-Dual.
B.4. Additional Qualitative Results
- The paper presents additional qualitative comparisons on RSGR-HDR and GOPRO-Dual in Figures B.10–B.16; captions identify Figures B.10–B.13 as RSGR-HDR and Figure B.14 as GOPRO-Dual.Source: §B.4 · p. 17, Figure B.10 · p. 19, Figure B.11 · p. 19
- Figures B.8 and B.9 show five of nine reconstructed intermediate frames at temporal positions t = [0, 2/8, 4/8, 6/8, 1] on GOPRO-Dual and RSGR-HDR, respectively.The figures compare outputs generated by different models.Source: Figure B.8 · p. 18, Figure B.9 · p. 18
- Figures B.14–B.16 present additional qualitative comparisons on GOPRO-Dual; separately, Figures B.10–B.12 show qualitative comparisons on RSGR-HDR, with B.10 corresponding to Figure 5 in the main manuscript.Source: Figure B.10 · p. 19, Figure B.11 · p. 19, Figure B.12 · p. 20