PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-exposure HDR Imaging: A Review of Pixel-level and Feature-level Reconstruction Methods

Qian Tao, Wei Wang, Chaobing Zheng, Zhengguo Li

arXiv:2608.28674v1cs.CV

TL;DR

Multi-exposure imaging broadens captured luminance but remains vulnerable to ghosting, exposure inconsistency, and unreliable evaluation in dynamic scenes. This survey reviews MEF and ghost removal across pixel-space and feature-space methods, datasets, and metrics. It concludes that ghost removal remains open and that single-shot HDR is a promising complementary direction.

  • Problem

    Dynamic scenes create cross-exposure motion, occlusion, ghosting, and exposure-related brightness and color artifacts, while existing metrics may miss locally visible defects.

  • Method

    The survey categorizes MEF and ghost-removal literature by objectives, output forms, pixel or feature fusion domains, alignment strategies, datasets, and evaluation metrics.

  • Results

    The review finds that alignment-based methods increasingly model correspondence and reliability, while alignment-free methods select reliable feature-space evidence for deghosting.

  • Takeaways & Limitations

    Ghost removal remains an open problem for multi-shot HDR imaging, making single-shot HDR a highly demanded complementary research direction.

  • Takeaways & Limitations

    Existing benchmarks and metrics remain insufficient: datasets may lack realistic motion or camera conditions, and no single metric comprehensively evaluates multi-exposure HDR imaging.

Abstract

from arXiv · show

Multi-exposure is an efficient way to capture real-world high-dynamic-range (HDR) scenes. However, HDR imaging suffers from severe ghosting artifacts in dynamic scenes due to the temporal gap between sequential exposures. In this article, we categorize the literature on two important topics on HDR imaging: multi-exposure fusion (MEF) and ghost removal. Conventional filter-based and data-driven methods are studied in pixel space and feature space. For popular deep learning-based approaches, we provide a granular taxonomy based on their alignment and fusion domains: pixel-space methods, which typically employ explicit motion compensation such as optical flow or spatial transformers, and feature-space methods, which leverage implicit alignment through deformable convolutions, attention mechanisms, or latent representation merging. Representative works are compared across different supervision settings, and key design principles are summarized. In addition, this survey summarizes commonly used datasets and evaluation metrics, discussing their applicability under diverse output forms. Finally, major bottlenecks and promising directions for future research are outlined.

1. Introduction

Multi-exposure imaging combines differently exposed LDR images to cover a wider luminance range, but dynamic scenes introduce ghosting, structural breaks, and exposure-related color and brightness artifacts. This survey organizes MEF and ghost removal across traditional and deep learning methods, including pixel-space and feature-space approaches.

  • Motivation: Multi-exposure capture combines complementary LDR images without hardware modification to represent a broader luminance range than a single exposure.It is widely used in mobile photography and industrial vision.
  • Challenges: Underexposure and overexposure can cause detail loss, brightness reversal, color shifts, and unnatural contrast even when images are aligned.These effects arise from saturation, nonlinear camera response, and local contrast variation.
  • Challenges: Camera shake, scene motion, and occlusion create displacement across exposures, so direct fusion can produce ghosting and structural breaks.Ghost removal therefore targets stable structures in dynamic regions.
  • Output formulations: The literature includes both display-ready LDR fusion and HDR reconstruction followed by tone mapping, reflecting different output formulations.Both formulations use the same multi-exposure inputs.
  • Survey organization: The survey separates methods by two objectives: aligned-image fusion for luminance and color consistency, and ghost removal for motion-induced structural inconsistency.Works are categorized by their primary objective, even when auxiliary components address the other issue.
  • Method taxonomy: Pixel-space methods evolved from hand-crafted weighting and pyramid blending toward structure-aware, edge-preserving, and reliability-aware fusion.Feature-space methods instead progress from CNN feature fusion toward attention, Transformer interaction, and generative restoration.

2. Differently Exposed LDR Images by Multiple Shots

Multiple shots capture differently exposed LDR images by varying exposure time or sensor gain, with raw measurements transformed through a camera response function into sRGB images. These images can be merged either after HDR reconstruction or directly through exposure fusion.

  • Capture model: A raw image records scene radiant power through exposure time and sensor gain, with black-point offset, saturation, and signal- and gain-dependent noise included in the capture model.The saturation level is the maximum sensor value that can be recorded.
  • Image formation: The corresponding sRGB image is obtained by applying a camera response function to each color channel of the raw measurement.The channel mapping is written as Z_i,c(p) = f_c(I_i(p)).
  • Image quality: The pixel SNR formulation accounts for saturation and separates read-out noise from analog-to-digital conversion noise.The saturation indicator [I_i(p) < I_max] determines whether the pixel is saturated.
  • Acquisition settings: Exposure sequences are commonly captured by changing exposure time while fixing sensor gain, or by changing gain while fixing exposure time; the former is more popular.The differently exposed sRGB images are intended to cover the scene’s dynamic range.
  • Fusion pipelines: One pipeline estimates camera response functions, converts inputs to HDR images, merges them by weighted frame averaging, and tone-maps the result for display.Another pipeline directly fuses the input images into an information-enriched 8-bit image.

3. Exposure Fusion for Multi-Shot HDR Imaging

MEF methods seek to preserve scene depth and fine details while avoiding halo and brightness-order-reversal artifacts. The survey traces progression from pixel-space weighting and multi-scale fusion to feature-space representation learning, selective aggregation, and physically or consistency-driven alternatives.

  • MEF algorithms fuse differently exposed images to preserve scene depth and fine details without halo or BOR artifacts.
  • Pixel-Space MEF: Pixel-space methods estimate weights and combine exposures through spatial, intensity, or multi-scale fusion strategies.Hand-crafted methods use cues such as contrast, saturation, and exposure; later methods strengthen weighting and reconstruction for extreme exposures.
  • Pixel-Space MEF: Pixel-space methods improve detail preservation, color fidelity, and robustness by refining exposure selection and reconstruction, but remain vulnerable to BOR artifacts.
  • Feature-Space MEF: Feature-space methods map exposures into learned representations before fusion, using correction, normalization, or exposure-aware encoding to reduce luminance and color inconsistency.These approaches commonly use multi-branch encoder–decoder architectures and supervised objectives involving pixel-level, structural, or perceptual losses.
  • Feature-Space MEF: Attention and multi-scale feature methods selectively aggregate reliable exposure information across spatial scales, combining local texture recovery with global contextual modeling.Examples incorporate exposure interpolation, self-attention, CNN–Transformer components, and cross-scale interaction to improve luminance consistency and preserve details.
  • Other Learning-Based Methods: Other learning-based methods explore no-reference supervision, physical or consistency constraints, and preference-based strategies beyond standard CNN, attention, or Transformer architectures.
  • Other Learning-Based Methods: A newer direction seeks outputs closer to the real HDR scene, learning and preserving highlight and shadow details absent from the input exposures.

4. Ghost Removal for Multi-Shot HDR Imaging

Ghost removal addresses motion and occlusion across multi-exposure images, where direct fusion produces ghosting and structural distortions. The survey compares reliability-based, registration-based, and feature-space approaches, emphasizing trade-offs between correspondence accuracy, robustness, fidelity, and efficiency.

  • Moving objects and imperfect alignment violate the assumption behind conventional MEF, producing ghosting artifacts in fused images.
  • Pixel-Based Deghosting: Pixel-based deghosting estimates pixel or patch reliability and down-weights inconsistent regions, often filling them from a reference exposure.
  • Pixel-Based Deghosting: Robust modeling can separate low-rank background structure from sparse motion changes without explicit registration or optical flow.
  • Registration-Based Deghosting: Registration-based methods align non-reference frames to a reference using transforms or optical flow before fusing the aligned stack.Gradient, structure, or log-intensity representations can reduce the influence of exposure differences during matching.
  • Feature-Space Deghosting: Feature-space deghosting categorizes methods as alignment-based, alignment-free, or generative and hybrid approaches, with supervised, self-supervised, and consistency-driven training settings.
  • Alignment-Based Methods: Alignment-based feature methods increasingly model correspondence and reliability together, enabling more stable fusion under large exposure differences and complex motion.
  • Alignment-Free Methods: Alignment-free methods select reliable evidence through attention, confidence modeling, non-local interaction, memory, or Transformers rather than warping inputs.They are generally more attractive under severe saturation, occlusion, or large exposure differences, but depend on correctly identifying unreliable regions.
  • Critical Comparison and Trade-Offs: Architecture and supervision choices impose trade-offs: CNNs are efficient locally, Transformers model global relationships at higher cost, and generative methods may produce visually convincing but physically unfaithful details.Paired HDR supervision is difficult for dynamic scenes, while weaker supervision is easier to extend but may not match true HDR quality.

5.1. Datasets

The survey organizes MEF datasets by their role in training and evaluation, distinguishing static benchmarks from dynamic datasets for motion and deghosting. It also identifies substantial gaps between current benchmarks and difficult real-world capture conditions.

  • Multi-exposure sequences provide both benchmarks for method comparison and data for training and evaluating deep learning models.Table 2 summarizes dataset names, venues, data types, splits, and sequence or image counts.
  • Static and Dynamic Datasets: Static datasets support detail enhancement, luminance consistency, and tone-mapped quality assessment, whereas dynamic datasets support motion compensation, saturation recovery, HDR reconstruction, and deghosting evaluation.
  • Dataset Limitations: Static datasets cannot evaluate motion-induced ghosting, while many dynamic benchmarks contain limited motion patterns, moderate exposure gaps, or simple occlusions.
  • Dataset Limitations: Synthetic datasets such as NTIRE_2021 aid supervised training but may have a domain gap from real camera noise, ISP processing, and handheld capture.
  • Dataset Limitations: Current benchmarks lack sufficient sequences combining large exposure ratios, severe exposure errors, complex object motion, and cross-exposure occlusion.This limits how reliably benchmark scores reflect robust real-world HDR imaging performance.

5.2. Evaluation Metrics

Multi-exposure fusion quality is assessed with subjective judgments and objective metrics tailored to HDR reconstruction or fusion without HDR ground truth. Full-reference metrics measure pixel, structural, or perceptual similarity, while source-reference metrics evaluate structure preservation from the input sequence.

  • Evaluation overview: Subjective evaluation considers exposure naturalness, detail clarity, color fidelity, and artifact presence but has limited reproducibility.Results depend on evaluator experience, display conditions, and task preferences.
  • Full-reference metrics: PSNR measures pixel-level error against a reference using mean squared error, with R denoting peak dynamic range and R = 1 for normalized images.PSNR-L operates in the linear domain, whereas PSNR-µ applies µ-law tone mapping before comparison.
  • Full-reference metrics: PSNR-L and PSNR-µ are preferred over raw PSNR because pixel-wise error correlates weakly with perceived structure, luminance, and color distortions in MEF.Higher PSNR values indicate smaller reconstruction errors, but this interpretation does not fully capture perceptual quality.
  • Full-reference metrics: SSIM evaluates local structural consistency by jointly considering luminance, contrast, and structural components between prediction and reference images.SSIM-L is computed in linear radiance, while SSIM-µ applies µ-law mapping to both images before evaluation.
  • Fusion-specific metrics: MEF-SSIM evaluates fused-image structure against a desired structure estimated from the input exposure sequence rather than an HDR ground-truth reference.The images are divided into local blocks, and larger values indicate better structure preservation.

5.2.5. MI

Mutual information measures how much information the fused image shares with the source exposure sequence. It uses local or global probability relationships between the fused image and source images, with larger values indicating richer retained source information.

  • Definition: Mutual information measures the amount of shared information between the fused image and the source images.It reflects how much information from the input exposure sequence is preserved in the fusion result.
  • Interpretation: A larger MI value usually indicates richer information retained from the source exposure sequence.The accumulated MI over all source images measures transferred source information.
  • Formulation: MI is computed from the joint and marginal probability distributions of the fused image and each source image.The relevant distributions are pF,Si(f, s), pF(f), and pSi(s).
  • Related measures: Standard deviation describes the spread of gray levels and is often used as a contrast indicator, while entropy measures information content from the fused image’s gray-level distribution.Larger SD and EN values usually indicate higher contrast and richer information, respectively.

5.2.8. QAB/F

QAB/F is a no-reference metric for source-edge preservation, but metric suitability depends on the output form and task. The survey emphasizes that statistical scores should be combined with visual and local assessment because no single metric captures HDR quality comprehensively.

  • QAB/F: QAB/F evaluates how well a fused image preserves edge information from the source images without requiring a reference image.Its components represent edge-strength preservation, edge-orientation preservation, and gradient-related weighting.
  • QAB/F: A higher QAB/F generally indicates that more source edge details are retained in the fused result.The metric aggregates preservation across source images and image locations.
  • Metric suitability: HDR reconstruction with ground-truth radiance commonly uses full-reference metrics, whereas fusion without HDR ground truth more often uses source-reference or no-reference metrics.The metric choice follows the output form and availability of reliable references.
  • Limitations: No single metric comprehensively evaluates HDR reconstruction quality, exposure correctness, color fidelity, and ghost removal simultaneously.Metric results should therefore be interpreted together with perceptual assessment whenever possible.
  • Limitations: High metric scores may coexist with noise, halos, over-sharpening, color shifts, or unnatural contrast that reduce visual quality.These distortions can improve some statistical indicators despite producing less pleasing results.
  • Limitations: Dynamic-scene deghosting requires visual comparison and local inspection because global scores may underrepresent double edges, motion residues, and structural duplications.Relevant inspection regions include motion, saturation, and occlusion areas.

5.3. Quantitative Comparison

Quantitative evaluations cover MEF on SICE and MEFB and deghosting on Kalantari, Hu, and Tel, showing stronger recent learning-based performance across these benchmarks.

  • MEF algorithms are evaluated on SICE and MEFB using PSNR, SSIM, CC, AG, VIF, MEF-SSIM, TMQI, and MSSSIM.
  • SICE dataset: Recent deep learning MEF methods improve structural-detail preservation and global luminance consistency on SICE.Table 5 reports clear differences between early CNN models and newer approaches.
  • MEFB dataset: Recent MEF approaches achieve more balanced performance across MEFB metrics, with gains in perceptual quality and information preservation.The survey attributes these improvements to more effective feature fusion and exposure modeling.
  • Deghosting algorithms are evaluated on Kalantari, Hu, and Tel using PSNR-µ, SSIM-µ, PSNR-L, SSIM-L, and HDR-VDP-2.
  • Deghosting datasets: Learning-based deghosting methods outperform traditional approaches and perform more consistently across most metrics, especially in dynamic scenes.The reported pattern reflects improved handling of motion and exposure variations.

5.4. Subjective Comparison

Subjective comparisons show that feature-space deep models improve brightness consistency and dynamic-region robustness, while persistent artifacts and real-world generalization gaps remain.

  • The subjective evaluation uses representative methods, common input conditions, pretrained models, and consistent MATLAB tonemapping for fair visual comparison.
  • MEF comparison: Traditional pixel-space fusion can preserve highlight and shadow details but suffers from halos and BOR artifacts, especially with two LER images.
  • MEF comparison: Feature-space MEF methods can address BOR artifacts and halos, but may flatten global contrast and introduce over-enhancement or color deviation.AGAL, HoLoCo, and Retinex-MEF are cited as representative examples.
  • Real-world limitations: Current MEF methods may approximate training-data LDR distributions rather than recover reliable radiance across complex real-world dynamic ranges.The survey identifies data-driven priors as limiting true physical restoration.
  • Deghosting comparison: Feature-space deghosting methods using attention or Transformers show higher robustness in dynamic regions than early CNN-based models.AHDRNet, HDR-Transformer, and SCTNet preserve structural integrity under complex non-rigid motion.
  • Deghosting comparison: Existing deghosting methods can still blur or produce local inconsistencies in extremely challenging regions where correspondence is difficult.The cited example involves curved strings in unseen data.

6. New Perspectives from Multi-Shot to Single-Shot

The survey identifies difficult transitions from multi-shot to single-shot HDR, including occlusion, exposure-ratio, BOR, and low-exposure trade-offs that motivate new imaging pipelines.

  • Existing ghost-removal algorithms fail when large heterogeneous exposure regions are occluded by moving objects or combined with non-rigid motion.
  • Single-shot HDR: Future single-shot HDR systems require neural spatially varying auto-exposure, raw-image processing, and a new image signal processor.The passage also notes possible rowing or blocking and BOR artifacts in independently exposed rows or blocks.
  • Single-shot HDR: Single-shot fusion of two LER images can retain BOR artifacts and fail to preserve real-world highlight and shadow details.
  • Low-exposure imaging: Low-exposure imaging reduces motion blur through short exposure time but introduces a trade-off among sharpness, noise, shadow recovery, and highlight preservation.
  • Low-exposure imaging: Future low-exposure methods should jointly address noise suppression, detail recovery, and exposure compensation, preferably in the raw domain.

7. Conclusions

The paper reviews MEF and ghost removal, concludes that ghost removal remains open for multi-shot HDR, and points to single-shot HDR as a needed direction.

  • The review covers two HDR imaging topics: multi-exposure fusion and ghost removal.
  • The authors identify ghost removal as an open problem for multi-shot HDR imaging.
  • The conclusion highlights single-shot HDR imaging as highly demanded and identifies related works as directions for future research.
Loading 2608.28674v1…