PaperlayerView exact revision on arXiv

Source-linked AI summary

Mono4DGS-HDR: High Dynamic Range 4D Gaussian Splatting from Alternating-exposure Monocular Videos

Jinfeng Liu, Lingtong Kong, Mi Zhou, Jinwen Chen, Dan Xu

arXiv:2510.18489v2cs.CV

TL;DR

Unposed monocular videos with alternating exposures pose an unaddressed setting for reconstructing renderable 4D HDR scenes. Mono4DGS-HDR learns video Gaussians before transforming and refining them in world space, with temporal luminance regularization. On a new benchmark, it outperforms adapted alternatives in rendering quality and speed.

  • Problem

    Prior dynamic HDR NVS work is limited to known camera poses and multi-camera scenes, leaving reconstruction from unposed monocular alternating-exposure videos unexplored.

    Source: §1 · p. 1

  • Method

    Mono4DGS-HDR first learns HDR video Gaussians in orthographic camera coordinates, then transforms them to world space and jointly refines them with camera poses, using temporal luminance regularization.

    Source: §1 · p. 2, §3 · p. 4, §3.3.1 · p. 5

  • Results

    Mono4DGS-HDR significantly outperforms adapted alternative solutions in rendering quality and speed on the new HDR video reconstruction benchmark.

    Source: §1 · p. 2, §5 · p. 10

  • Takeaways & Limitations

    The approach reconstructs renderable 4D HDR scenes from unposed monocular LDR videos captured with alternating exposures.

    Source: §1 · p. 2, §5 · p. 10

    For simplicity, Gaussian scaling, opacity, and color are assumed to be time-invariant.

    Source: §3.1 · p. 5

Abstract

from arXiv · show

We introduce Mono4DGS-HDR, the first system for reconstructing renderable 4D high dynamic range (HDR) scenes from unposed monocular low dynamic range (LDR) videos captured with alternating exposures. To tackle such a challenging problem, we present a unified framework with two-stage optimization approach based on Gaussian Splatting. The first stage learns a video HDR Gaussian representation in orthographic camera coordinate space, eliminating the need for camera poses and enabling robust initial HDR video reconstruction. The second stage transforms video Gaussians into world space and jointly refines the world Gaussians with camera poses. Furthermore, we propose a temporal luminance regularization strategy to enhance the temporal consistency of the HDR appearance. Since our task has not been studied before, we construct a new evaluation benchmark using publicly available datasets for HDR video reconstruction. Extensive experiments demonstrate that Mono4DGS-HDR significantly outperforms alternative solutions adapted from state-of-the-art methods in both rendering quality and speed.

1 INTRODUCTION

The paper addresses 4D HDR reconstruction from unposed monocular LDR videos with alternating exposures, a setting not previously explored. Mono4DGS-HDR uses staged Gaussian optimization and temporal luminance regularization, and is reported to outperform adapted alternatives in rendering quality and speed.

1 INTRODUCTION

  • Existing dynamic HDR NVS work uses known camera poses in a multi-camera setting, leaving unposed monocular alternating-exposure videos unexplored.Simply adapting unposed 4D reconstruction methods can yield suboptimal results because priors are noisy, brightness variation impedes photometric pose optimization, and HDR appearance can be temporally inconsistent.Source: §1 · pp. 1–2
  • The first stage learns HDR video Gaussians in orthographic camera coordinates without camera poses; the second transforms them to world space and jointly refines Gaussians and poses.The learned video representation supports pose optimization and initializes world Gaussian optimization; temporal luminance regularization targets HDR appearance consistency.Source: §1 · p. 2, Figure 2 · p. 3
  • Mono4DGS-HDR significantly outperforms adapted alternatives in rendering quality and speed on the new HDR video benchmark.Figure 1 compares reconstruction quality with HDR extensions of SplineGS, MoSca, and GFlow.Source: §1 · p. 2, Figure 1 · p. 2
  • The authors construct a new benchmark from publicly available datasets and report that their approach surpasses adapted alternatives in reconstruction quality and speed.The benchmark addresses a task the authors say had not previously been explored.Source: §1 · p. 2

2 RELATED WORK

Prior work treats HDR novel-view synthesis, dynamic reconstruction, and HDR video reconstruction as related but distinct problems. The paper positions its approach at their intersection: renderable 4D HDR scenes from unposed monocular videos with alternating exposures.

Dynamic Reconstruction and View Synthesis

  • Most dynamic reconstruction methods use synchronized multi-view video, while unposed monocular methods such as MoSca, SplineGS, and GFlow target standard videos with consistent brightness.The paper frames alternating exposures as a challenge for these methods.Source: §2 · p. 3

HDR Novel View Synthesis

  • Most HDR novel-view synthesis methods target static scenes; HDR-HexPlane handles dynamic scenes but assumes known poses and is validated with multiple cameras.The paper identifies unposed monocular video as the more practical setting addressed by its work.Source: §2 · pp. 3–4

HDR Video Reconstruction

  • HDR video reconstruction methods merge aligned alternating-exposure frames, but the described approaches cannot synthesize novel views or LDR images at new exposure levels.The paper contrasts these with self-supervised 4D scene reconstruction for rendering HDR and controllable-exposure LDR videos.Source: §2 · p. 4

3 METHODOLOGY

Mono4DGS-HDR addresses unposed monocular video with alternating-exposure LDR frames through two-stage Gaussian optimization. It first fits HDR video Gaussians without camera poses, then transforms them to world space for joint Gaussian and camera refinement.

3.2 PRIOR PRECOMPUTE

  • The input is an unposed monocular LDR video with alternating exposures; foundation-model depth, tracks, and same-exposure optical flow provide priors and initial camera parameters.For two exposures, odd-numbered frames are short-exposure and even-numbered frames are long-exposure.Source: §3 · p. 4, §3.2 · p. 5

3.1 PRELIMINARIES

  • HDR Gaussian Splatting represents irradiance instead of LDR color, while explicit Gaussian trajectories preserve 3DGS rendering speed; scaling, opacity, and color are time-invariant for simplicity.HDR irradiance is tone-mapped to LDR color at a given exposure time.Source: §3.1 · pp. 4–5

3.3 TWO-STAGE GAUSSIAN OPTIMIZATION

  • The framework first trains video Gaussians, transforms them to world space, then refines world Gaussians with camera parameters; noisy 2D priors make this staged strategy useful.The second-stage optimization is described as converging fast and stably with the video-Gaussian initialization.Source: §3.3 · p. 5, §3.3.2 · p. 6

3.3.1 VIDEO HDR GAUSSIANS

  • Stage one fits fully dynamic HDR Gaussians in orthographic camera coordinates, treating camera and object motion uniformly and avoiding camera parameters during initial HDR video reconstruction.The representation uses normalized projected pixel coordinates and depth, and provides an initialization for later world-Gaussian and camera refinement.Source: §3.3.1 · p. 5

3.3.2 VIDEO-TO-WORLD GAUSSIAN TRANSFORMATION

  • Because video Gaussians lack true world geometry, they are transformed using initial camera parameters; world scales are refit to preserve projected 2D covariance.The refit seeks consistent projected Gaussian shapes and sizes before and after transformation.Source: §3.3.2 · pp. 5–6
  • Dynamic-versus-static identification counts trajectory samples in dynamic masks, excluding occluded samples; a dynamic-region fraction above a preset threshold, such as 0.1, marks a Gaussian dynamic in world space.The masks are derived from epipolar error maps.Source: §3.3.2 · p. 5

3.3.3 OPTIMIZATION STRATEGY

  • Optimization combines LDR RGB, depth, flow or track, unit-exposure, motion-regularization, and reprojection losses, with the reprojection loss jointly refining camera poses and world Gaussians.The reprojection loss uses the recovered HDR training video because ordinary monocular photometric reprojection is unsuitable for alternating-exposure inputs.Source: §3.3.3 · pp. 6–7
  • Temporal luminance regularization flow-warps adjacent HDR renderings and compares luminance while excluding occluded pixels, promoting consistent appearance across exposure-dependent weak-supervision times.It is applied after Gaussian densification ends, using rendered flow maps for adjacent frames at different exposures.Source: §3.3.3 · p. 7

4 EXPERIMENTS

Experiments compare Mono4DGS-HDR with HDR novel-view-synthesis and unposed monocular reconstruction baselines, then ablate its components and visualize induced tracking. The method outperforms the baselines across tracks, while the ablations report benefits from video-Gaussian initialization, transformation design, HDR reprojection loss, and temporal luminance regularization.

4.1 EXPERIMENTAL SETTINGS

  • The benchmark covers public-dataset HDR videos; evaluation uses µ-law PSNR, SSIM, LPIPS, and HDR-TAE, with baselines trained for 15K iterations.The supplied benchmark description says scenes contain 50–100 frames and distinguishes synthetic and real-world exposure categories.Source: §4.1 · pp. 7–8, §4.2 · p. 8

4.2 PERFORMANCE COMPARISON

  • Mono4DGS-HDR outperforms all baselines across tracks and demonstrates state-of-the-art temporal stability, rendering quality, and speed.Qualitative comparisons report finer details and fewer artifacts than baselines.Source: §4.2 · pp. 8–9, Table 2 · p. 8

4.3 ABLATION STUDY

  • Removing video-Gaussian initialization drops PSNR by more than 1 dB, while those Gaussians provide position, rotation, scale, opacity, and color priors for world optimization.The ablation also reports noticeable visual degradation without this initialization.Source: §4.3 · p. 9, Table 3 · p. 9
  • Without occlusion handling, static content can be misclassified as dynamic, dropping PSNR by 0.3 dB; 2D covariance invariance supports reasonable initial world-Gaussian sizes.The reported occlusion-handling ablation also produces blur in background regions that were occluded.Source: §4.3 · p. 9, Table 3 · p. 9
  • Removing HDR photometric reprojection loss reduces performance; its dense supervision benefits camera-pose and scene-geometry refinement.The comparison is reported for the second optimization stage.Source: §4.3 · p. 9, Table 3 · p. 9

4.4 VISUALIZATION OF INDUCED 2D/3D TRACKING

  • Figure 7 visualizes induced 2D/3D tracking from dynamic Gaussian motion.The figure caption identifies tracking by dynamic Gaussian motion; the supplied section passages do not state a tracking-accuracy result.Source: Figure 7 · p. 10

4.3 ABLATION STUDY

  • Removing temporal luminance regularization greatly degrades HDR-TAE despite little PSNR change, with artifacts and noise reported in dynamic regions.The ablation attributes more temporally coherent appearance across frames to this regularization.Source: Table 3 · p. 9, §4.3 · p. 10
  • Spline control-point sampling intervals above 4 degrade performance; the study uses Ns = 4 as a trade-off with memory cost.Smaller intervals mean more control points and higher memory cost.Source: §4.3 · p. 10, Table 4 · p. 10

5 CONCLUSION

Mono4DGS-HDR addresses 4D HDR reconstruction from unposed monocular alternating-exposure LDR videos with a two-stage Gaussian optimization and temporal luminance regularization. On a new benchmark, it achieves superior rendering quality and speed over adapted methods.

5 CONCLUSION

  • The two-stage approach first learns video HDR Gaussians in orthographic camera coordinates, then transforms them to world space and refines them with camera poses.The conclusion presents this as its optimization approach for 4D HDR reconstruction from unposed monocular alternating-exposure LDR videos.Source: §5 · p. 10
  • Temporal luminance regularization is proposed to ensure temporal consistency of HDR appearance.Source: §5 · p. 10
  • Experiments on the new benchmark show superior rendering quality and speed over adapted methods.Source: §5 · p. 10

A.1.1 3D GAUSSIAN SPLATTING

3D Gaussian Splatting represents scenes with anisotropic Gaussians that are projected onto the image plane and rasterized with alpha blending.

A.1.1 3D GAUSSIAN SPLATTING

  • Each anisotropic 3D Gaussian is parameterized by position, rotation quaternion, scaling, opacity, and color.The passage gives these parameter domains as position and scaling in R^3, rotation in R^4, and opacity and color in [0, 1].Source: §A.1.1 · p. 14
  • A Gaussian’s spatial distribution uses covariance Σ = RSS⊤R⊤, derived from its rotation matrix and scaling matrix.The passage gives the Gaussian spatial distribution in terms of its position and covariance.Source: §A.1.1 · p. 14
  • Projected 2D Gaussians are sorted by a tile-based rasterizer and alpha-blended at queried pixels.The projection computes 2D covariance from the viewing transform and the Jacobian of an affine approximation.Source: §A.1.1 · p. 15

A.1.2 CUBIC HERMITE SPLINE INTERPOLATION

The paper models dynamic Gaussian motion with cubic Hermite splines defined by control points and interpolated over time.

A.1.2 CUBIC HERMITE SPLINE INTERPOLATION

  • Cubic Hermite splines model dynamic Gaussian motion using control points.Source: §A.1.2 · p. 15
  • For time t in [t_k, t_{k+1}], the spline computes a Gaussian’s position from the cubic Hermite interpolation.The passage introduces the position computation for the interval between adjacent control-point times.Source: §A.1.2 · p. 15
  • The interpolation uses Hermite basis functions and an approximated tangent at control point p_k.Source: §A.1.2 · p. 15

A.1.3 VIDEO GAUSSIAN REPRESENTATION

The video Gaussian representation places motion in orthographic camera coordinates, avoiding reliance on camera poses for this stage. Tracklets and depth priors initialize each Gaussian’s trajectory.

A.1.3 VIDEO GAUSSIAN REPRESENTATION

  • Video Gaussians use video width, height, and depth as the X, Y, and Z axes, with orthographic rasterization.The representation replaces the original projection Jacobian; the passage also notes image resolution w and h.Source: §A.1.3 · p. 15
  • Each 2D tracklet is lifted into camera coordinates using its per-frame depth, then sampled for spline-trajectory control points.Each tracklet corresponds to a video Gaussian; rotation quaternion polynomial coefficients initialize to [1, 0, 0, 0].Source: §A.1.3 · p. 16

A.1.4 SUPERVISION FROM 2D PRIORS

The 2D-prior supervision combines image reconstruction, depth, motion, and exposure-scale constraints to train the Gaussian representation.

A.1.4 SUPERVISION FROM 2D PRIORS

  • The LDR RGB loss combines DSSIM and L1 image-reconstruction losses, weighted by λd = 0.2.It compares rendered LDR images with the ground-truth images.Source: §A.1.4 · p. 16
  • The depth loss is the L1 difference between rendered depth and the prior depth map.Source: §A.1.4 · p. 16
  • The flow/track loss matches rendered motion maps to 2D priors under valid-flow or sparse-track masks, sampling either supervision at each iteration.Flow supervision uses same-exposure frame pairs at offsets set by the number of exposure levels; track supervision samples another video timestamp.Source: §A.1.4 · p. 16
  • The unit-exposure loss fixes learned HDR irradiance scale through the tone mapper, using C0 = 0.5 for real scenes and C0 = 0.73 for synthetic scenes.Source: §A.1.4 · p. 16

A.1.5 GAUSSIAN MOTION REGULARIZATION

This appendix details motion and reprojection regularization, benchmark protocols, implementation, further results, and limitations. Across the reported comparisons, the method is described as outperforming adapted alternatives, while several settings remain qualitative or expose specific failure modes.

A.1.5 GAUSSIAN MOTION REGULARIZATION

  • ARAP regularization preserves distances between each dynamic Gaussian and its k-nearest neighbors across times, with k = 5 in the experiments.Velocity and acceleration smoothness losses are also applied to encourage smooth Gaussian motion; the rotation term uses the Frobenius norm of the rotation logarithm.Source: §A.1.5 · p. 16

A.1.6 HDR PHOTOMETRIC REPROJECTION LOSS

  • HDR photometric reprojection warps HDR colors between nearby frames, masks dynamic pixels, and normalizes by summed HDR values to remove scale influence.It uses recovered first-stage HDR training frames with gradients stopped, sampling reference times from {t ± 1, t ± 2}.Source: §A.1.6 · p. 17

A.2.1 DATASETS

  • Syn-Exp-3 uses alternating frames for an equal train/test split and evaluates interpolated views and times; Real-Exp-3 tests observed exposures, while Real-Exp-2 trains and evaluates on the full video.Syn-Exp-3 has nine synthetic HDR videos; each real benchmark has eight scenes, with three and two exposure levels respectively.Source: §A.2.1 · pp. 17–18

A.2.2 EVALUATION METRICS

  • HDR-TAE measures per-pixel temporal consistency using warped adjacent frames, with lower values indicating better consistency.Optical flow and visibility masks come from RAFT; real scenes use rendered HDR frames and synthetic scenes use ground-truth HDR frames.Source: §A.2.2 · pp. 18–19

A.2.3 IMPLEMENTATION DETAILS

  • Training uses two optimization stages, with temporal luminance regularization applied after densification; loss curves for WavingHands and Cars show effective convergence despite reset-related loss spikes.The TLR loss appears only in the later training stage, after Gaussian densification ends.Source: §A.2.3 · pp. 19–20, Figure 11 · p. 19, §A.3.2 · p. 20

A.3.4 MORE ABLATION RESULTS

  • Randomized exposure schedules retain good performance, but differing exposures can impair flow estimates and worsen LPIPS; larger exposure-value perturbations also degrade performance and CRF fitting.Small perturbations such as [−0.25, 0.25] EV cause only a slight performance drop; the authors note that real-world exposure values can typically be read accurately from metadata.Source: Table 6 · p. 20, §A.3.4 · pp. 21–22, Table 7 · p. 22
  • Removing depth or flow/track losses substantially reduces reconstruction performance, while challenging scenes include low-light noise, motion blur, complex motion, and non-rigid deformation that the method does not model well.The authors identify reliance on 2D-prior quality, blur, and the ARAP constraint as limitations; no quantitative evaluation is available for significant view changes because ground truth is absent.Source: §A.3.3 · p. 21, §A.3.4 · p. 22, §A.4 · pp. 22–23

Related papers

Loading 2510.18489v2…