PAM-ToD: Plug-and-Play Appearance Modeling for Cross-Time-of-Day 3D Gaussian Splatting

1 Introduction

• Practical relighting needs to reuse an existing 3DGS with sparse target-time images, while preserving 3D consistency and real-time rendering.Inverse-rendering methods typically build a dedicated relightable representation, and separating material from illumination remains ambiguous with limited lighting or viewpoint variation; image-space diffusion can compromise multi-view consistency and is slower than 3DGS rendering.Source: §1 · pp. 1–2
• Cross-Time Scene Transfer adapts a frozen, pre-trained road-scene 3DGS from a few posed target-time images.It avoids separately recovering materials and illumination by eliminating unchanged albedo under a simplified diffuse image-formation model.Source: §1 · p. 2
• PAM-ToD combines hierarchical multiplicative corrections with additive brightness and regularization, without modifying the base 3DGS architecture.Its additive emission term captures newly active lights, while global, spatial, and per-Gaussian corrections support broad and local changes under sparse supervision.Source: §1 · p. 2
• CARLA-ToD evaluates appearance transfer across three times of day with matched geometry, camera poses, and dynamic-object trajectories.The benchmark contribution reports improved appearance transfer while preserving real-time rendering.Source: §1 · p. 2

Inverse Rendering

• Inverse-rendering methods equip Gaussian primitives with material and illumination attributes, but material-light separation remains ambiguous under a typical single-illumination capture.Methods such as Relightable 3D Gaussians, GaussianShader, and GS-IR recover surface normals, material, and incident illumination; each builds its own relightable representation.Source: §2 · p. 2

Relighting for Outdoor Scenes

• Outdoor appearance models learn variation from image collections, while other approaches explicitly model sunlight, skylight, shadows, or driving-scene illumination.PAM-ToD instead derives source-to-target appearance corrections on a frozen 3DGS without separately estimating albedo and illumination.Source: §2 · p. 3

Diffusion-based Relighting

• Diffusion relighting edits images or videos, but without an underlying 3D scene it does not ensure cross-view consistency and takes many sampling steps.This contrasts with PAM-ToD's render-time plug-in, which uses multiplicative illumination and additive emission terms.Source: §2 · p. 3, Figure 1 · p. 3

3.1 Problem Formulation

• With Gaussian parameters and the sky model frozen, target-time anchor views supervise render-time adaptation using unchanged albedo to derive a source-color rescaling plus an additive correction.The additive correction may be positive or negative under the formulation.Source: §3 · p. 3, §3.1 · p. 4

3.2 Parameterization of the Transport Equation

• Log gain combines scene-wide, smooth spatial, and per-Gaussian corrections; total variation and semantic k-NN smoothing regularize the grids and residuals, while nonnegative emission adds brightness.The emission is factored into per-Gaussian brightness and a learnable chromaticity.Source: §3.2 · pp. 4–5
• A separate global affine transform adapts the frozen sky model, with its bias compensating for an absolute source-light deficit.The sky transform's learnable gain and bias are initialized to zero.Source: §3.2 · p. 5

3.3 Optimization

• A robust anchor loss smooths renderings and anchor images, then excludes the largest smoothed pixel errors from its L1 term.This is intended to reduce sensitivity to fine-scale inconsistencies in pseudo-images while retaining broader appearance structure.Source: §3.3 · p. 5
• The appearance-transport prior uses class-specific source-color luminance bins to assign pseudo-targets, guiding gain-corrected colors even for Gaussians unseen in anchors.The prior omits emission so that it constrains the gain alone.Source: §3.3 · p. 5
• For unseen Gaussians, a one-way prior pulls log gain toward the visible median for the same semantic class and light-distance bin.Poles and traffic lights serve as proxies for light sources.Source: §3.3 · pp. 5–6
• Chromaticity regularization limits residual hue shifts, suppresses spurious shifts in unseen regions, and steers uniformly textured classes toward their detached median hue change.Luminance is left free; structural regularization additionally matches total variation, semantic k-NN smoothing, and emission sparsity to the representation hierarchy.Source: §3.3 · p. 6

• CARLA-ToD contains 24 sequences of 200 frames from three cameras, covering four scenes, three times of day, and Static and Dynamic configurations.Source: §4 · p. 6
• The synchronous route and timing make corresponding frames' geometry and camera poses coincide, with only illumination and emission changing.Weather is fixed to clear and sun altitude varies; dynamic actors follow identical trajectories across times of day.Source: §4 · p. 6
• Each RGB frame has a sky mask, while Dynamic frames also include dynamic-object masks and 3D bounding boxes.The three-camera rig records 1920 × 1300 sRGB images; a roof-mounted LiDAR is also included.Source: §4 · p. 6

• Evaluation uses StreetGS with K=1 and K=8 synchronized captures for CARLA-ToD Static comparisons, and renderings of StreetGS for Dynamic diffusion comparisons.GS-W uses frames from multiple times of day; GS-IR, GI-GS, and LumiGauss use separate time-specific models with environment-map swapping.Source: §5 · p. 7
• At K=8, CARLA-ToD Static NVS improves over baselines by up to 0.085 SSIM, 8.87 dB PSNR, and 0.077 LPIPS while running at 53.4 FPS.On CARLA-ToD Dynamic, PAM-ToD also outperforms state-of-the-art diffusion methods; those methods run at 0.07–0.08 FPS and show qualitative consistency or detail problems.Source: §5 · pp. 7–8, Table 2 · p. 8, Figure 3 · p. 8
• All metrics improve monotonically with more anchors and gains diminish around K=8; a single capture is within about 4 dB of K=16.The remaining gap is attributed to unobserved regions or source-dark regions where emission is constrained by anchor coverage.Source: §5 · p. 8, Table 3 · p. 9
• Removing anchor reconstruction causes a noticeable accuracy drop, while removing chromaticity regularization preserves PSNR but degrades SSIM and LPIPS.The appearance transport prior has a complementary effect; the seen-to-unseen prior has a minor effect with eight captures and dense coverage.Source: §5 · p. 8, Table 4 · p. 9
• Both correction channels matter, with multiplicative correction causing the largest drop when removed; hierarchy levels are especially necessary at K=1.At K=8, global gain captures most illumination changes, grids add gains, and residuals offer marginal value; the full model consistently performs best.Source: §5 · pp. 8–9, Table 5 · p. 9
• On Waymo with synthesized anchors, PAM-ToD outperforms UniRelight on ∆E00 (2.69 vs. 8.10), ∆L∗ (1.78 vs. 5.26), and EMDL (1.10 vs. 4.01).PAM-ToD retains target illumination more faithfully but remains susceptible to floaters inherited from the source 3DGS.Source: Figure 4 · p. 9, §5 · p. 9

• PAM-ToD adapts frozen pre-trained 3DGS using a multiplicative correction and non-negative additive emission term, avoiding explicit material-illumination decomposition.Hierarchical parameters and near-identity initialization support stable adaptation from a single synchronized anchor.Source: §6 · p. 9
• On CARLA-ToD, PAM-ToD substantially outperforms inverse-rendering, appearance-embedding, and diffusion baselines while maintaining real-time rendering.Source: §6 · p. 9
• The method requires posed target-time anchor images and struggles with specularities, sharp shadow boundaries, and physically consistent shadow rendering.Future work includes physical shadow priors for severe illumination extremes.Source: §6 · p. 9

A.1 Semantic Labels and Anchors

• Each anchor is a synchronized three-camera capture, while fixed Gaussian semantic features come from visibility-weighted aggregation of source-view Cityscapes probabilities.The view weight combines opacity and an occlusion test against rendered depth; K anchors provide 3K posed images.Source: §A.1 · p. 13

A.2 Color Space and Point of Application

• The correction acts on spherical-harmonics-evaluated view-dependent RGB before rasterization in display-encoded training-image space, without linearization.The appendix treats the image-formation relation as a structural approximation, not a radiometric identity.Source: §A.2 · p. 13

A.3 Sign of the Additive Correction

• Although the additive term in the appearance relation can be negative, the implemented emission is nonnegative; the split is not unique for a single source–target color pair.The shared emission chromaticity and view-shared gain constrain the parameterization, and fixing the sign addresses an ambiguity that destabilized few-anchor optimization.Source: §A.3 · p. 13

A.4 Parameterization and Initialization

• The per-Gaussian residual starts at zero, emission starts near zero with a shared positive unit-mean chromaticity, and sky gain and bias start at zero.The source sky model remains frozen.Source: §A.4 · pp. 13–14

A.5 Loss Definitions

• The appearance-transport prior assigns each Gaussian a rank-matched class-color pseudo target from 32 luminance quantiles, computed once without pixel correspondence.Classes below the stated Gaussian or anchor-pixel thresholds are skipped; the gain-corrected source color used by this prior omits emission, so emission receives no gradient from it.Source: §A.5 · p. 14
• The seen-to-unseen prior uses visibility and pole or traffic-light proxies to form distance-binned reference medians, and applies only where valid references exist.For dynamic scenes, dynamic Gaussians share the global gain and spatial field but have their own residual and emission; several priors apply only to static Gaussians.Source: §A.5 · p. 14

A.6 Regularizers

• Spatial-grid regularization penalizes finer levels more strongly, while semantic k-NN smoothing suppresses but does not eliminate smoothing across class boundaries.All plug-in parameters are optimized jointly for 8,000 iterations; the seen-to-unseen and second chromaticity terms start at iteration 4,000.Source: §A.6 · p. 15, §A.7 · p. 15

B.1 Generation Protocol

• CARLA-ToD fixes weather except for sun altitude across Morning, Noon, and Night, and its Dynamic sequences preserve actor poses and sizes across times of day.The three-camera rig uses synchronized RGB cameras; the dynamic configuration includes vehicles and pedestrians, with masks and bounding boxes provided.Source: §B.1 · p. 16, §B.2 · p. 16

B.3 Evaluation Protocol

• The source 3DGS uses all 200 source-time captures for Reconstruction, while NVS holds out every eighth capture from source training.Each sequence contains 200 synchronized captures.Source: §B.3 · p. 17
• Target-time anchors are spaced uniformly along the route, and NVS anchors falling on held-out frames move to the nearest training frame.Thus, the plug-in never observes an NVS held-out view during training.Source: §B.3 · p. 17
• NVS is scored only on captures held out from both the source 3DGS and the plug-in.Source: §B.3 · p. 17

C.1 Additional Results on Model size

• 7.7 M parameters add 7% over the frozen StreetGS base, with four floats per Gaussian and scene-shared spatial grids totaling 7 MB.Source: §C.1 · p. 17
• A single frozen base with a 31 MB plug-in per target time reduces the three-time-slot footprint from 1.3 GB to 0.49 GB.Source: §C.1 · p. 17
• The paper reports that PAM-ToD preserves high reconstruction fidelity and real-time rendering speed while reusing pre-trained 3DGS assets.Source: §C.1 · p. 17

C.2 Results Details on CARLA-ToD Static

• Inverse-rendering baselines degrade severely for several transitions, including noon-to-morning and transfers to noon, while morning-to-night transfers avoid significant accuracy drops.Source: §C.2 · p. 17
• LumiGauss frequently fails to converge, while GS-W outperforms inverse-rendering baselines but is described as insufficiently expressive.Source: §C.2 · p. 17
• PAM-ToD consistently yields superior average performance, even with one synchronized anchor capture, across the reported time-of-day adaptation results.The cited tables cover Static reconstruction and novel-view synthesis, with Morning, Noon, and Night conditions.Source: §C.2 · p. 17, Table 9 · p. 18, Table 10 · p. 18

Ablation on the number of anchors

• All metrics improve as anchor captures increase, with diminishing gains beyond K=8 because added captures mostly re-observe already constrained Gaussians.A single synchronized capture recovers most of the change; dark regions benefit more from added anchors than lit regions.Source: §C.3 · p. 18, Figure 5 · p. 19

Ablation on our key illumination and emission term

• The illumination and emission terms are complementary: gain models exposure and color shifts, while emission brightens active lights without amplifying near-black color noise.Without illumination, the model cannot darken scenes; without emission, large gains can create chromatic artifacts on road surfaces and walls.Source: Figure 6 · p. 19, §C.3 · p. 19
• With one synchronized capture, per-Gaussian emission is weakly constrained, so the low-capacity illumination term absorbs most of the appearance shift.In the displayed examples, emission concentrates on active light sources and changes little elsewhere, so it does not model fine local appearance shifts on its own.Source: Figure 6 · p. 19, §C.3 · p. 19

CARLA-ToD static scenes

• Additional NVS qualitative results are shown for CARLA-ToD static scenes.Figure 7 presents results across various CARLA-ToD segments.Source: §C.4 · p. 19, Figure 7 · p. 20

CARLA-ToD dynamic scenes

• Additional NVS qualitative results are shown for CARLA-ToD dynamic scenes.Figure 8 presents results across various CARLA-ToD segments.Source: §C.4 · p. 19, Figure 8 · p. 20

Waymo Open Dataset

• Figure 9 presents qualitative NVS results for the Waymo Open Dataset.Source: §C.4 · p. 19, Figure 9 · p. 21