PaperlayerView exact revision on arXiv

Source-linked AI summary

WildRelight: A Real-World Benchmark and Physics-Guided Adaptation for Single-Image Relighting

Lezhong Wang, Mehmet Onurcan Kaya, Siavash Bigdeli, Jeppe Revall Frisvad

arXiv:2605.11696v1cs.CVcs.AIcs.GR

TL;DR

Real-world performance of single-image relighting models remains uncertain because existing evaluation relies mainly on synthetic data and lacks aligned benchmarks with ground-truth illumination. WildRelight introduces such a benchmark and a physics-guided DPS with Sampling-Aware TTA framework, achieving performance rivaling fully supervised baselines without synthetic pre-training.

  • Problem

    Single-image relighting lacks a real-world benchmark with ground-truth illumination, leaving models evaluated predominantly on synthetic data that misses real-world complexity.

  • Method

    WildRelight captures aligned outdoor images with co-located HDR environment maps and uses temporal light evolution to guide DPS and Sampling-Aware TTA.

  • Results

    The physics-guided framework achieves performance rivaling fully supervised baselines without synthetic pre-training.

  • Takeaways & Limitations

    WildRelight supports evaluating and adapting relighting models against aligned real-world illumination through a self-supervised temporal formulation.

  • Takeaways & Limitations

    The framework is presented as a structured case study rather than a fully optimized solution, leaving room for methodological improvement.

Abstract

from arXiv · show

Recent single-image relighting methods, powered by advanced generative models, have achieved impressive photorealism on synthetic benchmarks. However, their effectiveness in the complex visual landscape of the real world remains largely unverified. A critical gap exists, as current datasets are typically designed for multi-view reconstruction and fail to address the unique challenges of single-image relighting. To bridge this synthetic-to-real gap, we introduce WildRelight, the first in-the-wild dataset specifically created for evaluating single-image relighting models. WildRelight features a diverse collection of high-resolution outdoor scenes, captured under strictly aligned, temporally varying natural illuminations, each paired with a high-dynamic-range environment map. Using this data, we establish a rigorous benchmark revealing that state-of-the-art models trained on synthetic data suffer from severe domain shifts. The strictly aligned temporal structure of WildRelight enables a new paradigm for domain adaptation. We demonstrate this by introducing a physics-guided inference framework that leverages the captured natural light evolution as a self-supervised constraint. By integrating Diffusion Posterior Sampling (DPS) with temporal Sampling-Aware Test-Time Adaptation (TTA), we show that the dataset allows synthetic models to align with real-world statistics on-the-fly, transforming the intractable sim-to-real challenge into a tractable self-supervised task. The dataset and code will be made publicly available to foster robust, physically-grounded relighting research.

1 Introduction · 2 Related Work

WildRelight addresses the unresolved synthetic-to-real gap in single-image relighting by providing a strictly aligned, HDR-equipped benchmark for diverse outdoor scenes. It also motivates physics-guided, test-time adaptation and distinguishes the dataset from controlled, multi-view, and task-mismatched prior datasets.

  • 1 Introduction: Single-image relighting has advanced through deep generative models that decompose photographs into intrinsic components and rerender scenes under novel illumination.These methods have enabled increasingly photorealistic results and broad applications in computational photography, filmmaking, and augmented reality.
  • 1 Introduction: Synthetic training and evaluation leave a critical gap because they omit real-world light transport effects and non-ideal natural materials.Consequently, strong synthetic-benchmark performance may not transfer to practical real-world applications.
  • 1 Introduction: WildRelight is designed as a high-precision evaluation benchmark that prioritizes strict pixel alignment and radiometric accuracy over dataset scale.It captures diverse outdoor scenes across times ranging from golden hour to harsh midday sunlight.
  • 1 Introduction: The dataset’s strictly aligned natural-illumination evolution enables physics-guided inverse rendering with sampling-aware test-time adaptation as a self-supervised domain-adaptation task.The framework uses only available test-time observations to support instance-specific adaptation from synthetic to real data.
  • 2 Related Work: Controlled laboratory datasets obtain high-fidelity ground truth using calibrated lights, light stages, synchronized cameras, or robotics, but operate in highly constrained environments.Examples include OpenIllumination, OpenSubstance, and RelightMyNeRF.
  • 1 Introduction: WildRelight combines a fixed single-view camera, diverse in-the-wild outdoor scenes, and spatially aligned HDR environment maps for every image.Its strict viewpoint alignment enables direct, pixel-aligned evaluation of relighting methods.
  • 2 Related Work: Neural inverse-rendering methods and multi-view outdoor datasets support scene decomposition or relighting, but primarily target multi-view reconstruction and relightable 3D modeling.Examples include NeRF-based approaches, NeRF-OSR, OWL, and Stanford-ORB.
  • 2 Related Work: Existing fixed-view datasets use sparse point lights, darkroom capture, or image-processing setups, leaving them mismatched to real-world single-image outdoor relighting.This motivates WildRelight’s combination of single-view capture, natural outdoor illumination, and HDR ground truth.

3 Dataset Collection and Curation

WildRelight is collected as a real-world single-image relighting dataset using fixed-view outdoor scenes captured across naturally changing illumination. Its dual-camera acquisition, radiometric processing, and manual masking preserve correspondence between scene images, environmental lighting, and evaluation regions.

  • Dataset Scope: WildRelight contains 30 scenes, each recorded from a fixed camera position under 5 to 7 natural illumination conditions.Sampling across different times of day captures both gradual afternoon changes and rapid pre-sunset transitions.
  • Temporal Sampling: Natural-light acquisition requires hours of continuous monitoring because illumination follows the sun’s immutable trajectory rather than active indoor lighting schedules.Sampling occurs every 45 to 60 minutes from 11:00–17:00 and every 10 to 15 minutes before sunset, when changes are more dramatic.
  • Capture System: The dataset uses a Sony A7 for high-resolution scene capture and an Insta360 Pro 2 for simultaneous 360-degree environmental illumination maps.The cameras are co-located by aligning the Insta360 optical center with the Sony lens’s nodal point, representing the incident light field at the scene-camera viewpoint.
  • Calibration and HDR Processing: The processing pipeline uses synchronized, calibrated cameras and linear RAW data to produce radiometrically consistent HDR images.ColorChecker profiles unify sensor color rendition, while bracketed exposures are merged using weighted radiance estimates and stored as float64 linear EXR.
  • Dynamic Scene Handling: Hand-annotated binary masks identify dynamic regions such as wind-blown foliage and moving clouds without warping ground-truth pixels.Researchers can optionally exclude these regions during metric computation to separate relighting performance from artifacts caused by scene motion.

4 Methodology

WildRelight’s methodology combines physics-guided posterior sampling with sampling-aware test-time adaptation to exploit real-world illumination supervision. The framework enforces rendering consistency during inference and adapts lightweight scene-specific modules using temporally varying paired photo–environment-map data.

  • Framework Overview: The reference framework integrates physics-guided posterior sampling for inverse decomposition regularization with sampling-aware TTA to align forward relighting dynamics.It is presented as a structured case study rather than a fully optimized solution, leaving room for future improvements.
  • Physics-Guided Inference: Physics-guided inference uses Diffusion Posterior Sampling to enforce physical validity by rendering decoded intrinsic components under illumination with a differentiable Cook–Torrance renderer.The measurement loss compares the rendered image with the observed image Igt, while diffusion network parameters remain frozen and split-sum lighting improves stability and efficiency.
  • Physics-Guided Inference: The DDIM trajectory is refined using the rendering-loss gradient, updating x_{t-1} with guidance strength ζt at each diffusion step.The update is x_{t-1} ← x_{t-1} − ζt g_t, where g_t = ∇x_tLrender.
  • Temporal Self-Supervision: Temporal self-supervision follows a leave-one-out protocol, adapting on N −1 observed lighting conditions and evaluating on the held-out light using paired photo–environment-map data.The protocol does not require ground-truth G-buffers and addresses real-world domain shifts through WildRelight’s temporal illumination variations.
  • Sampling-Aware Adaptation: Sampling-aware optimization freezes the diffusion backbone, trains attention-layer LoRA modules, and backpropagates perceptual reconstruction loss through the final K steps of partial DDIM sampling.The objective is L = Lnoise + λpercLLPIPS(D(z0), Itarget), aligning denoising and rendering dynamics with scene-specific light transport such as complex shadows.

5 Experiments

Experiments quantify WildRelight’s synthetic-to-real gap, demonstrate its utility for supervised adaptation, and evaluate physics-guided DPS+TTA for inference-time adaptation. Across 30 scenes, the combined method reaches near-supervised performance while avoiding full retraining.

  • Experimental Design: Experiments benchmark zero-shot methods, supervised finetuning, and the proposed DPS and TTA strategies on WildRelight.Evaluation uses PSNR, SSIM, and LPIPS after global least-squares intensity alignment to address scale ambiguity.
  • Zero-Shot Benchmark: Synthetic-trained RGB↔X and DiffusionRenderer struggle zero-shot on WildRelight, achieving below 16 dB PSNR while failing on complex outdoor illumination.The results expose a synthetic-to-real gap involving high-frequency shadows and outdoor indirect illumination.
  • Supervised Adaptation: LoRA finetuning of DiffusionRenderer on WildRelight improves PSNR from 23.28 dB to 25.95 dB, demonstrating the dataset’s value for bridging the domain gap.The adapted model serves as a supervised reference and learns realistic outdoor lighting effects from real-world data.
  • Inference-Time Adaptation: The inference-time protocol evaluates instance-specific adaptation across all 30 scenes by holding one lighting as the target and using the remaining N −1 lightings as self-supervised signals.This leave-one-lighting-out setup simulates deployment without ground-truth supervision.
  • Ablation Study: DPS adds physical constraints for a +0.95 dB PSNR gain, while TTA alone adds +2.5 dB PSNR but worsens LPIPS from 0.390 to 0.392.Unconstrained TTA overfits pixel intensities, creating a photometric-perceptual trade-off and perceptual artifacts.
  • Ablation Study: The combined DPS and TTA pipeline achieves 25.04 dB PSNR and 0.345 LPIPS, approaching supervised finetuning at 25.95 dB PSNR without expensive retraining.DPS regularizes TTA to preserve physical plausibility and high-frequency details during scene-specific adaptation.

6 Conclusion

WildRelight bridges the synthetic-to-real gap with aligned in-the-wild images and co-located HDR environment maps, enabling real-world self-supervised domain adaptation. Physics-Guided Inverse Rendering and Sampling-Aware TTA achieve performance rivaling fully supervised baselines without synthetic pre-training.

  • Contributions: WildRelight bridges the synthetic-to-real gap using aligned, in-the-wild images paired with co-located HDR environment maps.The benchmark’s temporal structure supports reformulating domain adaptation as a real-world self-supervised task.
  • Contributions: Physics-Guided Inverse Rendering with DPS and Sampling-Aware TTA achieves performance rivaling fully supervised baselines without synthetic pre-training.The conclusion presents these methods as validating the temporal self-supervised adaptation formulation.
  • Future Directions: Future work includes modeling dynamic scene elements instead of masking them and improving the computational efficiency of instance-specific adaptation.The authors aim for models that are physically robust to the unconstrained complexity of the natural world.

Supplementary Materials · 1 Quantitative Validation of Illumination Alignment

The supplementary validation shows that scene images and environment maps are temporally aligned closely enough for outdoor relighting. Timestamp statistics, solar displacement analysis, and model-resolution calculations establish that even worst-case delays produce negligible illumination error.

  • 1.1 Temporal Synchronization Statistics: Timestamp metadata was extracted for all Sony A7 scene images and Insta360 environment maps to analyze their temporal synchronization.The time-delay distribution is summarized in Table 5.
  • 1.1 Temporal Synchronization Statistics: 1.7 minutes is the maximum capture delay, and it remains within the stated tolerance for accurate outdoor lighting estimation.The validation uses the delay statistics together with subsequent physical error analysis.
  • 1.2 Physical Error Analysis: Solar Angular Displacement: 0.00417° per second is the assumed solar angular motion derived from Earth’s approximately 15°-per-hour rotation.Solar displacement is computed as capture delay multiplied by 0.00417°/s.
  • 1.2 Physical Error Analysis: Solar Angular Displacement: 0.17° is the estimated solar displacement for the mean capture delay of 40.14 seconds.This is approximately one-fifth of the sun’s 0.5° angular diameter and is described as perceptually imperceptible in lighting effects.
  • 1.2 Physical Error Analysis: Solar Angular Displacement: 0.48° is the estimated solar displacement for the worst-case delay of 114 seconds, remaining below the sun’s approximately 0.5° angular diameter.The analysis therefore treats even the worst-case displacement as smaller than the light source’s angular size.
  • 1.3 Impact on Relighting Tasks: Most samples have capture delays below 20 seconds, while a long tail extending to approximately 100 seconds represents a small fraction of outliers.Figure 8 shows a heavily right-skewed distribution with a dominant peak in the first bin.
  • 1.3 Impact on Relighting Tasks: 1.4° is the approximate horizontal angular resolution at 256-pixel environment-map width, the scale commonly used by relighting models.The cited resolution is 360°/256 ≈ 1.4°.
  • 1.3 Impact on Relighting Tasks: 0.3 pixels is the maximum effective-resolution shift caused by the 0.48° worst-case solar displacement, confirming high-quality illumination alignment.The calculation treats the shift as sub-pixel at the resolution used by state-of-the-art relighting models and supports training and evaluation of inverse-rendering methods.

2 Details Setting of Baseline Benchmark

WildRelight benchmarks three single-image relighting methods on aligned real-world scenes using PSNR, SSIM, and LPIPS, exposing a synthetic-to-real domain gap. Finetuning DiffusionRenderer on the training split substantially improves performance on the unseen test set.

  • Dataset and Metrics: WildRelight is partitioned into 21 training scenes, 4 validation scenes, and 5 held-out test scenes, evaluated with PSNR, SSIM, and LPIPS.
  • Baseline Models and Evaluation: The benchmark evaluates RGB↔X, DiffusionRenderer, and Materialist using single-image decomposition into G-buffers followed by relighting with novel environment maps.Materialist instead refines predicted G-buffers through physical-renderer optimization while using the dataset’s ground-truth environment maps.
  • Finetuning Protocol: DiffusionRenderer finetuning freezes the VAE and environment encoder, updates the spatio-temporal UNet with rank-8 LoRA, and trains for 48 hours on one NVIDIA H100 GPU.Training uses an L2 loss, AdamW with learning rate 1 × 10^-4, and checkpoint selection on the four-scene validation set.
  • Zero-Shot Results: Synthetic-only RGB↔X and DiffusionRenderer perform poorly on real-world images, achieving low PSNR and SSIMw scores, whereas Materialist obtains better average metrics.The reported advantage for Materialist is attributed to providing the ground-truth environment map during optimization and using a physical renderer.
  • Domain Adaptation Results: 16.67 dB PSNR versus 11.04 dB after finetuning, and 0.5317 SSIM versus 0.4649, demonstrate large gains on the unseen test set.The finetuned model is DiffusionRenderer, adapted using WildRelight’s training split.

3 Evaluation Protocol and Scale Alignment.

Evaluation uses PSNR, SSIM, and LPIPS on the WildRelight test split, with global least-squares alignment addressing single-image relighting’s inherent illumination-scale ambiguity. Metrics are computed after alignment to preserve relative illumination structure and chromaticity while enabling reliable quality assessment.

  • Experiments use PSNR, SSIM, and LPIPS on the WildRelight test split.
  • Scale Alignment: Global least-squares alignment solves an optimal scalar α for each predicted image and corresponding ground truth to correct arbitrary global intensity scaling.The optimization minimizes ||Ipred · α − Igt||^2.
  • Scale Alignment: Metrics are computed on Ipred · α∗, compensating for global intensity discrepancies while preserving relative illumination structure and chromaticity.

4 Advantages of RAW-Based HDR Image

RAW-based HDR imaging preserves more scene information than JPG and enables direct HDR synthesis through RAW’s linear relationship with light intensity. Its higher bit depth supports recovery of shadow detail and retention of highlight information with greater color fidelity.

  • RAW Linearity: RAW sensor data has a fundamentally linear relationship with scene light intensity, enabling HDR synthesis through linear combinations of multiple exposures.This approach avoids calculating and inverting the Camera Response Function.
  • JPG Limitations: JPG’s 8-bit storage and compression discard substantial color and luminance information, especially under extreme lighting conditions.These limitations are illustrated in Figure 9.
  • Shadow Detail and Color Fidelity: RAW’s typically 12- or 14-bit depth captures extensive low-light detail that can be recovered by increasing exposure with minimal loss.JPG exposure boosting instead produces color distortion, banding, and noise because discarded information cannot be meaningfully recovered.
  • Overall Advantage: RAW-based HDR synthesis preserves shadow detail and highlight information better than JPG while supporting accurate restoration of original colors through exposure adjustment.The figure states that HDR photo details are preserved, allowing original colors to be accurately restored.

5 Justification of Data Acquisition Setup

The acquisition protocol uses a dual-camera setup because projected 360° imagery cannot reliably meet photorealistic relighting benchmark standards. Precise co-location is required to preserve pixel alignment and prevent parallax-induced inconsistencies between illumination and scene images.

  • Camera Alignment: Precise co-location of the Insta360 environment-map camera and Sony A7 scene camera is necessary for pixel-aligned evaluation.The protocol identifies spatial alignment as a critical acquisition requirement.
  • Limitations of a Single Panoramic Camera: Projected 360° captures fall short of benchmark fidelity because they reduce effective resolution, degrade image quality and dynamic range, and introduce optical artifacts.An 8K 360° image projected to a 40mm FOV has lower effective resolution than the rig’s 24MP+ full-frame Sony A7; smaller sensors add noise and chromatic aberration, while stitching adds resampling artifacts.
  • Limitations of a Single Panoramic Camera: The dedicated Sony A7 scene camera preserves higher-frequency detail needed to evaluate texture preservation and generation.The comparison is against an 8K 360° capture projected to a standard 40mm field of view.
  • Camera Alignment: A non-confocal setup with a 10cm baseline changes occlusion relationships and can create false shadows between the environment map and photograph.Foliage may occlude the sun in one camera’s view but not the other, causing shadows to appear in the illumination map but not the photograph, or vice versa.

6 Methodology for Determining the Nodal Point (No-Parallax Point)

The methodology identifies the lens’s no-parallax point by rotating a camera around an adjustable pivot and observing whether near and distant reference objects remain aligned. Correct alignment eliminates parallax-induced stitching errors in panoramic images.

  • Purpose and Principle: Parallax occurs when foreground objects appear displaced relative to the background during camera rotation, so the camera must rotate around the entrance pupil.The term “nodal point” is commonly used, but the technically precise locus is the center of the entrance pupil.
  • Experimental Setup: The camera is mounted on a stable tripod with a panoramic head that permits longitudinal adjustment, using a 40mm focal length and two vertically aligned objects at different distances.A lamppost and a more distant utility pole serve as reference points.
  • Iterative Alignment: The camera is panned approximately 30 degrees left and right while the relative positions of the two reference objects are inspected for displacement.Observed shifting indicates that the rotation axis is not coincident with the entrance pupil.
  • Iterative Alignment: When displacement is observed, the camera’s fore-aft position on the panoramic head is adjusted and the rotation test is repeated.The process continues until the reference objects remain in perfect alignment during panning.
  • Validation: At the no-parallax point, the camera’s rotation axis aligns with the lens entrance pupil at the selected focal length, preventing parallax-induced stitching errors.This condition is illustrated by the nearby utility pole continuing to occlude the poles behind it when the camera rotates.

7 Differentiable Cook–Torrance Renderer

The method uses a fully differentiable Cook–Torrance microfacet renderer with split-sum approximation to evaluate physics consistency from predicted G-buffers. It combines energy-conserving diffuse and Fresnel-weighted specular reflection from HDR environment maps, enabling gradient backpropagation during DPS.

  • Renderer formulation: The renderer models per-pixel basecolor, normal, roughness, and metallicity under HDR environment illumination using Cook–Torrance microfacet shading.Outgoing radiance is decomposed into diffuse and specular components.
  • Renderer formulation: Diffuse radiance uses energy-conserving Lambertian reflection, modulated by metallicity, Fresnel response, basecolor, and spherical-harmonics diffuse irradiance.The diffuse irradiance Ediff is obtained by projecting the environment map onto spherical harmonics.
  • Renderer formulation: Specular radiance uses the Schlick Fresnel term and prefiltered environment sampling along the reflection direction with roughness-dependent mipmapping.The Fresnel base term is F_0 = lerp(0.04, cb, m), and energy conservation weights diffuse and specular components according to F and m.
  • Differentiable optimization: All renderer operations are differentiable, allowing reconstruction gradients to backpropagate from rendered radiance to predicted G-buffers during Diffusion Posterior Sampling.The final radiance is clamped to non-negative values for HDR consistency.
Loading 2605.11696v1…