PaperlayerView exact revision on arXiv

Source-linked AI summary

HELIOS: From midnight to noon, continuous outdoor urban scene relighting

Hala Djeghim, Nathan Piasco, Luis Roldão, Moussab Bennehar, Dzmitry Tsishkou, Céline Loscos, Désiré Sidibé

arXiv:2609.00901v2cs.CV

TL;DR

Driving-scene relighting lacks paired multi-illumination data and existing synthetic-data approaches face challenging real-world nighttime conditions. HELIOS learns unpaired relighting from real-world data using albedo-conditioned cycle-consistent diffusion, distilled nighttime albedo, and GPS-derived solar-angle control. Evaluations report structurally consistent, realistic results that outperform state-of-the-art methods across night-to-day and day-to-night tasks.

  • Problem

    Driving datasets are captured at limited times, while paired multi-illumination supervision and synthetic data limit relighting in diverse real-world conditions.

  • Method

    HELIOS combines albedo-based conditioning in a cycle-consistent diffusion pipeline with cross-domain albedo distillation and continuous GPS-derived solar-angle control.

  • Results

    HELIOS produces structurally consistent and realistic night-to-day and day-to-night relighting results, outperforming state-of-the-art methods.

  • Takeaways & Limitations

    The method enables smooth, precise lighting manipulation across the day-night cycle while preserving scene structure in real-world driving images.

  • Takeaways & Limitations

    HELIOS remains challenged by extremely low-light nocturnal environments where accurate albedo-like images and relighting results are difficult to predict.

Abstract

from arXiv · show

Modifying the illumination of driving images is a fundamental challenge, as most datasets are captured at specific times of day. Existing methods rely on synthetic data or paired multi-illumination supervision, which limits their generalization to the diverse and challenging conditions of real-world scenarios. To address this, we propose HELIOS, a novel image relighting approach that relies on unlabeled real-world datasets without requiring any paired images for training. Our approach integrates albedo-based conditioning into a cycle-consistent diffusion pipeline to prevent identity collapse and ensure accurate domain translation. To handle low-visibility nighttime conditions, we introduce a robust albedo distillation strategy that transfers structural stability from the daytime domain. Additionally, we replace traditional text prompts with a fine-grained control mechanism based on GPS-derived solar angles, enabling smooth and continuous lighting manipulation across the day-night cycle. Through extensive evaluation and a user study, we demonstrate that HELIOS produces structurally consistent and realistic results in both night-to-day and day-to-night tasks, outperforming state-of-the-art methods.

1 Introduction

HELIOS targets real-world driving-scene relighting without paired multi-illumination supervision or synthetic training data. It combines albedo-aware cycle-consistent translation, nighttime albedo distillation, and GPS-based continuous solar-angle control.

  • Motivation: Limited capture times and uncontrolled outdoor conditions make realistic driving-scene relighting difficult while preserving structure and semantics.Existing approaches are constrained by synthetic-to-real nighttime gaps involving sensor noise, low signal-to-noise ratios, and complex illumination effects.
  • Approach: HELIOS trains a single-image relighting framework exclusively on real-world datasets without paired multi-illumination data.Its cycle-consistent diffusion pipeline performs unpaired domain translation while preserving scene content.
  • Approach: Albedo-based conditioning separates reflectance from illumination, preventing identity collapse and encouraging illumination transfer independent of the input image.The albedo estimation module conditions relighting on albedo maps rather than input images.
  • Approach: Albedo distillation addresses low-visibility nighttime conditions by transferring structural knowledge from a well-lit domain.The strategy is intended to provide reliable nighttime albedo prediction when details are partially invisible or shadowed.
  • Approach: GPS-derived solar angles replace coarse text prompts with fine-grained control for smooth and precise lighting interpolation across the day-night cycle.The continuous signal is computed from publicly available GPS metadata associated with driving images.

2 Related work

Prior relighting methods face ill-posed inverse rendering, synthetic-to-real gaps, and limited nighttime generalization. HELIOS addresses these constraints with cycle-consistent training and robust albedo distillation for real-world driving data.

  • Relighting via Inverse Rendering: Inverse rendering in large-scale outdoor driving is highly ill-posed because transient lighting and non-uniform materials complicate illumination-reflectance disentanglement.Existing urban solutions often depend on manually designed priors and high-quality daytime captures.
  • Learning-based Relighting: Learning-based relighting methods can generalize on standard datasets but struggle in driving environments without multi-illumination ground truth.Some methods rely on synthetic data or real-data augmentation, while nighttime performance may remain undocumented.
  • Learning-based Relighting: Pseudo-labels from DiffusionRenderer fail to produce accurate nighttime albedo maps, limiting related methods under nighttime conditions.UniRelight is described as high-fidelity in daytime settings, but its nighttime performance is undocumented.
  • HELIOS: HELIOS uses cycle-consistent training and robust albedo distillation to learn structurally accurate relighting directly from real-world driving datasets.The approach avoids the synthetic-data pathway emphasized in the related methods discussion.
  • Unpaired Image-to-Image Translation: CycleNet faces a trade-off between structural consistency and domain translation that can produce identity collapse in complex driving relighting.HELIOS modifies this framework with a robust albedo prior and continuous relighting control.

3 Background

CycleNet formulates unpaired translation as a cycle-consistent diffusion process: an image is translated between domains and reconstructed while conditioning separates structural content from target-domain control. HELIOS’s architecture instantiates this process with albedo and solar-altitude conditioning.

  • CycleNet: Unpaired image-to-image translation learns mappings between domains X and Y without aligned paired samples.CycleNet adapts cycle consistency to diffusion by requiring translation to domain Y and back to reconstruct the source.
  • CycleNet: The framework assumes an invariant structural condition cimg and uses ctext to control the target domain.A ControlNet architecture predicts the clean sample from a noisy latent at a specified timestep.
  • CycleNet: The reconstruction loss trains the network to recover the original clean sample from a noisy latent under the source-domain condition.The cited formulation minimizes squared error between x0 and the network prediction Fθ(xt, cx, x0).
  • HELIOS Architecture: HELIOS extends this architecture by conditioning unpaired relighting on albedo a and solar altitude θs, then mapping the translated representation back through cycle consistency.Daytime samples use θs > 0 for reconstruction, while nighttime translation uses θs < 0.
  • CycleNet: The cycle-consistency loss reconstructs the source after an intermediate translation to the target domain and re-noising of that translated representation.The intermediate representation is defined as ¯y = Fθ(xt, cy, x0), with yt formed by adding noise before source-domain reconstruction.
  • CycleNet: An invariance loss is additionally used to keep the target domain stable under repeated transfers.This complements reconstruction and cycle-consistency objectives in the diffusion translation pipeline.

4 Method

HELIOS formulates single-image driving-scene relighting as unpaired day-night translation using cycle-consistent diffusion trained on unpaired real-world data. Albedo conditioning prevents identity collapse, while GPS-derived solar altitude provides continuous lighting control.

  • Unpaired Relighting: HELIOS trains an unpaired image-to-image relighting model exclusively on real-world datasets, without synthetic supervision or paired captures.Cycle consistency supports domain translation while preserving scene content.
  • Unpaired Relighting: Cycle-consistency with input-image conditioning can force identity mapping, preventing meaningful lighting shifts despite preserving reconstruction.CycleNet predictions illustrate this failure during Day-to-Night translation.
  • Albedo Conditioning: The Albedo Estimator extracts lighting-invariant reflectance and replaces image conditioning, separating illumination transfer from the input image.The albedo remains constant across domain translations, providing structural conditioning without an explicit invariance loss.
  • Albedo Conditioning: The reconstruction and cycle objectives train diffusion predictions conditioned on the target domain, noisy image state, and extracted albedo.The supplied objective passages define reconstruction from x_t and cycle reconstruction from y_t.
  • Continuous Relighting: HELIOS replaces discrete text prompts with GPS-derived solar altitude θs, enabling fine-grained and smooth lighting transitions across day and night.θs is computed from latitude, solar declination, and local hour angle; positive values indicate daytime and negative values nighttime.

5 Robust Night-to-Day Translation via Albedo Distillation

HELIOS distills stable daytime albedo knowledge into an estimator trained across real daytime and generated nighttime illuminations. The resulting reflectance prior supports structurally consistent relighting in both directions, including low-visibility nighttime conditions.

  • Motivation: Existing intrinsic decomposition methods can produce incomplete nighttime albedo maps, causing structural loss when geometry is obscured by shadows.DiffusionRenderer is described as accurate for daytime albedo but prone to structural loss and color drifting at night.
  • Cross-Domain Albedo Distillation: HELIOS distills daytime decomposition knowledge into a domain-invariant estimator for nighttime and daytime albedo prediction.The strategy exploits greater intrinsic-decomposition stability in daytime images.
  • Cross-Domain Albedo Distillation: The initial Day-to-Night model generates three nighttime and two daytime variants per daytime image, while the static scene geometry makes the daytime albedo pseudo-ground truth for every variant.This synthetic illumination sampling supplies multi-illumination training examples.
  • Robust Albedo Estimator: The robust estimator is trained with latent-space denoising on real daytime and generated nighttime images to disentangle illumination from base color.Training across both illumination domains yields a structural prior invariant to daytime and nighttime conditions.
  • Unified Relighting: The final unified model combines reconstruction and cycle losses while conditioning on distilled albedo priors for high-fidelity Day-to-Night and Night-to-Day translation.The authors state that this conditioning preserves scene geometry while synthesizing global illumination.

6 Experiments

HELIOS is evaluated on real driving datasets across relighting directions, qualitative fidelity, continuous control, quantitative metrics, user preference, and albedo-distillation ablations. Across these evaluations, it preserves scene structure while producing realistic lighting and offers fine-grained solar-angle control.

  • 6.1 Experiments details: HELIOS is trained on 41k balanced daytime and nighttime images from nuScenes, Waymo, and Pandaset, with approximately 40k synthetic multi-illumination images for albedo-estimator training.The relighting model uses public real-world driving datasets, while the separate albedo estimator uses six generated illumination versions for each of 6,798 original images.
  • 6.2 Qualitative results: HELIOS maintains structural integrity while translating both global and local lighting effects across Night-to-Day and Day-to-Night tasks.CycleNet, InstructPix2Pix, DiffusionRenderer, and Qwen-Image each exhibit failures involving insufficient translation, missing local effects, synthetic-looking outputs, or structural inconsistency.
  • 6.2 Qualitative results: Continuous solar elevation conditioning produces realistic, fine-grained transitions across twilight, night, and deep-night illumination states.Compared with discrete text prompts, HELIOS avoids exaggerated localized lighting and preserves natural ambient illumination.
  • 6.3 Quantitative results: HELIOS consistently outperforms all methods on FID and CLIP metrics for both Night-to-Day and Day-to-Night translation.The reported balance combines reasonably low DINO scores with stronger distribution alignment and semantic relighting accuracy, whereas some baselines score well structurally by copying the source.
  • 6.3 Quantitative results: In a study with 18 participants and 48 pairwise comparisons, Qwen-Image's Day-to-Night win rate drops to 27.6% and its failure rate doubles.The study measures preference and rejection of non-realistic translations; Qwen-Image performs impressively for Night-to-Day but often hallucinates vehicles or loses structural consistency for Day-to-Night.
  • 6.3 Quantitative results: Cross-domain albedo distillation improves structural recovery from obscured nighttime inputs and produces sharper Day-to-Night relighting results.Standard albedo decomposition performs poorly for Night-to-Day in deep shadows, while distilled albedo maps recover geometry and retain more high-frequency detail.

7 Discussion

HELIOS supports color editing and relighting-based data augmentation, and its daytime relighting improves nighttime object detection. The method remains challenged by extremely low-light nocturnal environments.

  • 7 Discussion: HELIOS enables realistic color editing for data augmentation by modifying albedo color within a mask before relighting.The application changes masked scene colors and then uses HELIOS to produce the edited illumination.
  • 7 Discussion: HELIOS's applications extend beyond relighting to scene color manipulation and downstream perception enhancement.The paper presents both masked color editing and object-detection evaluation as applications.
  • 7 Discussion: HELIOS's robust albedo estimator maintains accurate structures and colors in dark areas for Night-to-Day and sharper details for Day-to-Night.The ablation compares the robust estimator with a standard albedo estimator and links sharper albedo detail to sharper relighting outputs.
  • 7 Discussion: 5.3% improvement in mAP@70 is obtained when night scenes from ACDC are relit to daytime for YOLOv11 detection.The result is reported as an image-enhancement application and is attributed to recovered details in night conditions.
  • 7 Discussion: Extremely low-light nocturnal environments remain a limitation because accurate albedo-like images and relighting results become challenging to predict.The stated boundary concerns the method's robustness under the most severely dark conditions.

8 Dataset Distribution and Illumination Analysis

The training datasets cover different solar-altitude distributions, from clustered recording times to continuous deep-night-to-noon coverage. Training across these diverse elevations supports smooth continuous relighting control.

  • 8 Dataset Distribution and Illumination Analysis: Pandaset solar altitudes are highly clustered at sharp peaks, indicating collection during a small number of fixed-time driving sessions.Its distribution provides limited temporal coverage compared with the broader ranges in the other datasets.
  • 8 Dataset Distribution and Illumination Analysis: Waymo has the most continuous distribution, spanning deep night at −40° to high noon at +70°.Its smoother density curve reflects capture across more sessions and a broader timeframe.
  • 8 Dataset Distribution and Illumination Analysis: Training on diverse sequences and sun elevations provides continuous relighting control for outdoor driving images.The paper describes the resulting control as smooth and continuous across illumination conditions.

9 User study details

HELIOS is compared with four state-of-the-art baselines on Day-to-Night and Night-to-Day translation. A randomized web-based user study evaluates realism and structural preservation through pairwise comparisons and win/fail rates.

  • The evaluation compares HELIOS with four state-of-the-art baselines across Day-to-Night and Night-to-Day translation tasks.
  • Eighteen participants each completed 48 randomized pairwise comparisons, with 24 rounds for each translation task.
  • References were randomly selected from 1000 images, while method pairings and left-right positioning were randomized to reduce comparison and side bias.
  • Participants selected the most realistic relit image that preserved the reference structure, with an option to mark both results as failed.
  • Win Rate measures preference for a method’s output, while Fail Rate measures unsuccessful translation caused by unrealistic artifacts or failure to follow the relighting instruction.

10 Additional results

Additional qualitative results show that HELIOS produces realistic relighting for both translation directions while preserving scene geometry and modulating illumination with target solar altitude. The results also expose baseline instability and a HELIOS failure boundary under extreme nighttime lighting.

  • Additional qualitative results: HELIOS consistently preserves scene geometry while modulating global illumination according to the target solar altitude.
  • Night-to-Day: HELIOS predicts realistic Night-to-Day relighting in qualitative comparisons.
  • Day-to-Night: HELIOS predicts realistic Day-to-Night relighting in qualitative comparisons.
  • Baseline failure: Qwen-Image exhibits modal instability and sometimes confuses relighting instructions with other tasks.
  • HELIOS limitation: Under extreme nighttime lighting, HELIOS fails to recover structural details; a wider extremely low-light training distribution for the albedo estimator is proposed as a remedy.
Loading 2609.00901v2…