Source-linked AI summary
HDR Video Generation via Latent Alignment with Logarithmic Encoding
Naomi Ken Korem, Mohamed Oumoumad, Harel Cain, Matan Ben Yosef, Urska Jelercic, Ofir Bibi, Yaron Inger, Or Patashnik, Daniel Cohen-Or
TL;DR
HDR generation is difficult because HDR radiance distributions do not match the bounded, perceptually compressed data used to train generative models. The paper aligns HDR with pretrained video-model latents using LogC3 and camera-mimicking degradations, then adapts the model lightly to generate high-quality, temporally coherent HDR video. Results and ablations support the approach across temporal stability and representation alignment, while the authors identify real-world HDR data and one-pass text-to-HDR generation as future extensions.
Problem
HDR’s scene-linear, wide-range radiance distribution is mismatched with the bounded, perceptually compressed data used to train generative models, while dedicated HDR representations add complexity and data requirements.
Method
LumiVid aligns HDR with a pretrained model’s latent distribution using fixed camera-inspired LogC3 encoding and trains LoRA adapters with camera-mimicking SDR degradations that encourage prior-based recovery.
Results
LumiVid generates high-quality, temporally coherent HDR video with minimal adaptation; it achieves a JOD of 7.86 and an F2F-PSNR of 45.63 dB.
Takeaways & Limitations
HDR video generation can leverage pretrained video diffusion priors without redesigning the generative architecture when representation alignment and targeted degradations are used.
Takeaways & Limitations
The authors identify scaling training data with real-world HDR captures and moving from two-stage text-to-SDR-then-SDR-to-HDR generation to one-pass text-to-HDR as future directions.
Abstract
from arXiv · showhide
High dynamic range (HDR) imagery offers a rich and faithful representation of scene radiance, but remains challenging for generative models due to its mismatch with the bounded, perceptually compressed data on which these models are trained. A natural solution is to learn new representations for HDR, which introduces additional complexity and data requirements. In this work, we show that HDR generation can be achieved in a much simpler way by leveraging the strong visual priors already captured by pretrained generative models. We observe that a logarithmic encoding widely used in cinematic pipelines maps HDR imagery into a distribution that is naturally aligned with the latent space of these models, enabling direct adaptation via lightweight fine-tuning without retraining an encoder. To recover details that are not directly observable in the input, we further introduce a training strategy based on camera-mimicking degradations that encourages the model to infer missing high dynamic range content from its learned priors. Combining these insights, we demonstrate high-quality HDR video generation using a pretrained video model with minimal adaptation, achieving strong results across diverse scenes and challenging lighting conditions. Our results indicate that HDR, despite representing a fundamentally different image formation regime, can be handled effectively without redesigning generative models, provided that the representation is chosen to align with their learned priors.
1 Introduction
HDR generation is difficult because scene-linear radiance is poorly matched to the bounded distributions used by generative models. LumiVid addresses this mismatch with logarithmic encoding and camera-mimicking degradations while adapting a pretrained model lightly.
- Motivation: HDR imagery spans wide intensity ranges but has heavy-tailed, unbalanced linear-space distributions unlike the bounded data used to train modern generative models.This mismatch affects faithful representation of highlights, shadows, and subtle illumination effects.
- Motivation: Training a dedicated HDR encoder would add extra stages, architectural components, and data requirements to a pretrained generative pipeline.The authors therefore frame the challenge as representation compatibility rather than increasing model capacity.
- Representation alignment: A fixed logarithmic encoding used in cinematic pipelines aligns HDR imagery with pretrained models’ latent distributions, enabling lightweight fine-tuning without learning a new encoder.The transformation is presented as a way to bridge HDR and the pretrained model’s learned data distribution.
- Training strategy: Camera-mimicking degradations corrupt extreme luminance regions so the model must infer missing highlight and shadow details from learned priors rather than copy input pixels.The strategy uses contrast clipping, compression artifacts, and selective blurring.
- Outcome: Combining log-encoded HDR inputs with minimal adaptation yields high-quality HDR video while preserving dynamic range and using pretrained visual priors.The authors argue that appropriate representation and training signals can avoid fundamentally new architectures.
2 Related Work
Prior HDR methods commonly learn task-specific SDR-to-HDR mappings, while LumiVid instead aligns HDR with pretrained video-model distributions and extends this strategy to native video generation. Its training and inference pipelines use frozen pretrained components with lightweight LoRA adaptation.
- Image and video reconstruction: Early HDR reconstruction work used multiple exposures, tone mapping, and gradient-domain compression to reduce dynamic range while preserving local details.Later approaches evolved toward CNN-based reconstruction and generative detail recovery for saturated regions.
- Image and video reconstruction: HDR video methods have addressed temporal consistency through optical flow, coarse-to-fine alignment, event cameras, illumination robustness, and larger datasets.These approaches focus on reconstruction and temporal alignment under particular sensing or imaging settings.
- Distribution alignment: Task-specific SDR-to-HDR methods operate under specific imaging assumptions and generally remain constrained by information present in the input.LumiVid instead treats HDR generation as alignment between HDR data and pretrained diffusion distributions.
- Generative HDR methods: Diffusion-based HDR research includes unsupervised synthesis, HDR reconstruction, exposure-aware bracket modeling, and pretrained-model adaptation through gain-map decomposition.These methods represent several generative directions for synthesizing or recovering missing radiance.
- Comparison with concurrent work: Compared with X2HDR, LumiVid uses camera-oriented LogC3 compression, camera-mimicking degradations, limited-data video training, and native video generation.The paper contrasts this with X2HDR’s display-oriented encoding and frame-by-frame application, which exhibits temporal instability.
- Video diffusion models: Modern video diffusion systems use compressed video VAEs, Diffusion Transformers, and LoRA for efficient adaptation, but are primarily trained on SDR imagery.LumiVid extends this model family to HDR without modifying the backbone by adding an HDR-aware compression layer.
- LumiVid pipeline: LumiVid training compresses HDR frames with LogC3, pairs target latents with degraded SDR reference latents, and trains only LoRA adapters through flow matching.At inference, the pretrained VAE and DiT remain frozen while the adapted model generates HDR video from SDR input.
3 Method
LumiVid adapts a pretrained SDR video diffusion model for HDR generation by aligning log-encoded HDR with the model’s latent domain and training it to infer missing radiance. The pipeline combines frozen-VAE encoding, SDR conditioning, camera-mimicking degradations, and inverse LogC3 decoding for scene-linear HDR output.
- Overview: LumiVid generates temporally coherent, scene-linear HDR video from SDR references without modifying the pretrained model architecture or retraining its VAE.The framework uses a pretrained SDR video diffusion model and produces unbounded HDR radiance.
- Training Pipeline and Inference: The training pipeline conditions a DiT through AVControl on SDR references while keeping the VAE frozen.The DiT denoises VAE latents to produce video content, making representation compatibility essential.
- Latent Manifold Alignment via Logarithmic Mapping: LogC3 and PQ maintain bounded VAE roundtrip error across the luminance range, whereas ACES and HLG diverge above diffuse white.The analysis evaluates compression, VAE encoding and decoding, and decompression back to scene-linear HDR.
- Latent Manifold Alignment via Logarithmic Mapping: LogC3 maps unbounded HDR radiance into the VAE’s expected range, aligning pixel and latent distributions with the pretrained SDR manifold.The transformation is selected by minimizing pixel-space and latent-space KL divergence.
- Training Pipeline and Inference: Camera-mimicking degradations corrupt extreme luminance regions so the model must synthesize missing highlights and shadows from learned visual priors.Augmentations include MP4 compression, contrast scaling, selective blur, and joint exposure shifts.
- Training Pipeline and Inference: At inference, decoded latents are inverse-transformed with LogC3 and saved as float16 EXR files for re-exposure and post-production grading.The output preserves scene-linear, unbounded radiance data.
4 Experiments
LumiVid is evaluated for reconstruction fidelity, out-of-distribution generalization, temporal stability, and the effects of representation alignment and camera-mimicking augmentation. It outperforms the evaluated baselines across both benchmarks while producing temporally coherent HDR video.
- 4.1 Benchmarks: LumiVid outperforms all baselines on both the ARRI video and UPIQ image benchmarks.The evaluation compares reconstruction quality on professional video and calibrated HDR images.
- 4.2 HDR Reconstruction Quality: 30.05 dB PU21-PSNR and JOD 8.22 on UPIQ show strong performance on calibrated photometric HDR images.The result indicates generalization from scene-linear video training to static HDR images.
- 4.3 Temporal Stability: 7.86 JOD and 45.63 dB F2F-PSNR on ARRI demonstrate high perceptual quality and frame-to-frame fidelity.All 49 frames are produced jointly by the diffusion backbone, supporting native temporal coherence.
- 4.4 Ablation Studies: LogC3 achieves the lowest divergence from SDR pixels and the best JOD score among the evaluated HDR transformations.PQ, another smooth logarithmic curve, performs comparably on most metrics.
- 4.4 Ablation Studies: The full camera-mimicking augmentation pipeline improves PU21-PSNR by 0.35 dB and JOD from 7.62 to 7.86.The ablation attributes the improvement to corrupting inputs so the model synthesizes radiance from learned priors.
5 Conclusion
LumiVid generates temporally coherent HDR video from a single SDR input by aligning HDR content with a pretrained model and training it to reconstruct corrupted luminance regions. The method uses minimal adaptation and outperforms a dedicated SDR-to-HDR baseline on out-of-distribution ARRI footage, while future work targets broader data and simpler workflows.
- 5 Conclusion: LumiVid uses LogC3 alignment and highlight-shadow corruption to adapt a pretrained SDR video diffusion model for HDR generation.The method retains the frozen VAE and uses a LoRA adapter without modifying the backbone architecture.
- 5 Conclusion: +3.7 dB on out-of-distribution ARRI footage is achieved with a LoRA adapter adding less than 1% parameters.Training uses 10,000 steps on approximately 300 clips, with inference requiring 11 denoising steps in a two-stage pipeline.
- 5 Conclusion: Future work includes real-world HDR video data, single-pass text-to-HDR-video generation, and perceptual HDR training objectives.These directions are presented as extensions that could strengthen generalization, simplify workflows, or improve perceptual quality.