PaperlayerView exact revision on arXiv

Source-linked AI summary

FujinSplat: Seeing Through Smoke with RAW-Domain Gaussian Splatting

Gengjia Chang, Ziteng Cui, Shuhong Liu

arXiv:2609.06017v2cs.CV

TL;DR

Smoke and nonlinear camera processing are entangled in captured images, complicating clean 3D reconstruction. FujinSplat separates them in RAW using a calibrated ISP, a controller for per-view corrections, and a constrained residual during Gaussian reconstruction. On RealX3D, it exceeds the strongest comparable baseline.

  • Problem

    Smoke varies with viewing ray and capture time, while the ISP remaps the recorded signal; existing sRGB restoration and reconstruction approaches address these entangled effects only partially.

    Source: §1 · p. 2

  • Method

    FujinSplat freezes a per-scene camera-calibrated Base ISP, trains a RAW controller on exact labels synthesized by reverse ISP actions, and jointly optimizes static 3D Gaussians with a bounded per-view residual.

    Source: §1 · p. 2, §3.3 · p. 5

  • Results

    FujinSplat reaches 18.42 dB on RealX3D, exceeding the strongest comparable baseline by 2.57 dB.

    Source: §5 · p. 8

  • Takeaways & Limitations

    At unseen poses, the renderer uses pose and intrinsics only, with no per-view parameter predicted or optimized, and inference costs exactly that of the underlying renderer.

    Source: §1 · p. 2

    The method is evaluated on a single benchmark and needs further validation across cameras and smoke conditions.

    Source: §5 · p. 9

Abstract

from arXiv · show

The appearance of a smoky scene is shaped by two processes that a camera records together: the participating medium alters scene radiance in a view-dependent way, and the image signal processor (ISP) then remaps the result through a nonlinear tone and color transformation. Recovering a clean 3D scene requires separating both. Per-view sRGB dehazing acts only after the ISP has entangled them; standard 3D reconstruction ignores the medium and absorbs it into scene geometry and radiance. FujinSplat addresses the problem in the RAW domain, where the two processes remain separable. A per-scene Base ISP is fitted from the scene's hazy RAW captures to its own camera renderings and then frozen, providing a fixed photometric anchor that performs no dehazing. Analyzing expert corrections reveals a compact, low-dimensional correction space identifiable from RAW alone. FujinSplat therefore fits per-view action answers at the training poses and trains a single scene-agnostic controller to regress them from RAW; the corrected views supervise one static 3D Gaussian representation, jointly with a bounded per-view residual that reconciles cross-view photometric inconsistencies. On the RealX3D real-world smoke benchmark FujinSplat clearly outperforms the strongest comparable baseline, ahead of both physics-based reconstruction and restoration-then-3DGS pipelines. Code is available at https://github.com/I2WM/FujinSplat .

1. Introduction

FujinSplat separates smoke effects from camera rendering in RAW, using a fixed per-scene ISP anchor and a RAW-trained controller to correct training views for static 3D reconstruction. On RealX3D, it outperforms the strongest comparable baseline.

1. Introduction

  • Smoke varies by viewing ray and capture time, while the ISP applies nonlinear exposure, white-balance, color, and tone transformations to the recorded signal.Recovering a clean 3D scene therefore requires separating medium effects from camera rendering before reconstruction absorbs them into geometry and appearance.Source: §1 · p. 2
  • Per-view sRGB restoration operates after medium and rendering are entangled, while medium-aware reconstruction and appearance-adaptive methods face limitations on real smoke or a fixed photometric anchor.The cited approaches may yield inconsistent supervision, encode haze in a static representation, or let shared transformations migrate between view codes and scene appearance.Source: §1 · p. 2
  • Paired hazy and clean captures show that one color direction carries 97–99% of average per-pixel residual energy, motivating a shared color action with per-view strength.ISP parameters are gauge-ambiguous, so independently fitted per-view corrections can be noisy and mutually inconsistent.Source: §1 · p. 2
  • FujinSplat reaches 18.42 dB averaged over eight RealX3D scenes, exceeding the strongest comparable baseline by 2.57 dB.The pipeline uses a calibrated Base ISP, per-view color actions, and a bounded residual during reconstruction.Source: §1 · p. 2
  • FujinSplat freezes a per-scene Base ISP that reproduces camera rendering without dehazing, then uses a RAW controller to predict per-view correction actions for static 3D Gaussian supervision.The method also uses a bounded per-view residual to reconcile cross-view inconsistencies during reconstruction.Source: §1 · p. 2

2. Related Work

Prior adverse-environment reconstruction models scattering or uses restoration and generative priors, while RAW-space methods address other capture settings. FujinSplat targets smoke removal and novel-view synthesis directly from RAW.

2.1. 3D Reconstruction in Adverse Environments

  • Adverse-environment reconstruction models participating media or uses restoration and generative priors, but many approaches operate on camera-rendered RGB and can introduce cross-view drift or hallucinated detail.The related work includes scattering-aware radiance fields and Gaussian representations, as well as methods for precipitation, mixed weather, and smoke.Source: §2.1 · p. 3

2.2. RAW-Space Novel-View Synthesis

  • RAW-space novel-view methods address noisy, low-light, or HDR capture, while scattering reconstruction predominantly uses processed RGB rather than jointly removing smoke and synthesizing views from RAW.Photometric methods also use view-specific transformations that can remain ambiguous with shared scene appearance; FujinSplat confines paired clean supervision to training poses without target-view adaptation.Source: §2.2 · p. 3

3. Method

FujinSplat calibrates a scene-specific camera-rendering baseline, learns per-view color corrections from RAW, and jointly optimizes a static Gaussian scene with a constrained residual for cross-view consistency.

3.2. Scene-Specific Base ISP Calibration

  • Each scene receives a calibrated Base ISP because camera devices and configurations can produce distinct RAW-to-RGB mappings.Fitted to hazy camera RGB, it reproduces the camera rendering rather than dehazing and provides a shared RAW-to-RGB baseline across training views.Source: §3.2 · pp. 3–4

3.1. Overview and Problem Formulation

  • The overview maps hazy RAW through a scene-specific Base ISP and predicted color action to corrected training-view RGB, which supervises static 3D Gaussian reconstruction.During reconstruction, the Base ISP and controller remain frozen while the per-view residual refines the scene-shared action.Source: §3.1 · p. 3, Figure 2 · p. 4

3.3. RAW Color-Action Controller

  • The 195 fitted expert corrections occupy a compact space: one direction carries 95% of energy, and the leading residual mode after removing the mean carries 41%.Eight operations at four strengths plus identity span the population; nearest-operation matching has median cosine 0.98, and a RAW-based classifier selects the correct entry with 0.91 accuracy. Reverse ISP-action synthesis generates RAW examples with exact action labels for controller training.Source: §3.3 · pp. 4–5

3.4. 3DGS Optimization with ∆-ISP

  • Because independent RAW-based controller predictions may disagree across views, a bounded per-view Δ-ISP is optimized with the static Gaussian representation.It retains a fraction of each view’s displacement from the scene-shared action, adds one scalar per training view, and enforces zero mean so it reconciles view-dependent disagreement without changing the shared correction.Source: §3.4 · p. 5

4. Experiments

Experiments on RealX3D compare FujinSplat with physics-based reconstruction, restoration-then-3DGS, and hybrid pipelines. FujinSplat leads conventional baselines on average PSNR and SSIM, while ablations examine its RAW input, color-action design, capacity, residual adjustment, and controller training.

4. Experiments

  • Evaluation uses eight paired RAW/RGB RealX3D scenes, compares physics-based, restoration-plus-3DGS, and hybrid baselines, and scores official novel views with PSNR, SSIM, and LPIPS.Clean novel-view images are reserved for evaluation, and metrics use equal scene weighting.Source: §4 · p. 5

4.1. Quantitative and Qualitative Comparisons

  • In novel views, FujinSplat restores visibility while preserving structure and natural color, whereas physics-based and independent-restoration pipelines show residual smoke or inconsistent color and exposure.Figure 4 compares six scenes; cyan insets enlarge ColorChecker regions.Source: Figure 4 · p. 6, §4.1 · p. 7
  • 18.42 dB PSNR and 0.679 SSIM are the best averages, with highest PSNR on all eight scenes; gains are 6.66 dB over physics and 2.57 dB over restoration.FujinSplat has second-best average LPIPS at 0.541, compared with 0.493 for ConvIR.Source: §4.1 · p. 6, Table 1 · p. 7
  • 18.2083 dB without external model calls or ensembling exceeds MSDG by 0.66 dB and remains within 0.17 dB of Dehaze-then-Splat and 0.46 dB of Smoke-GS.FujinSplat completes optimization in 42 minutes, versus approximately 5 to 250 hours for the reported hybrid pipelines.Source: Table 2 · p. 7, §4.1 · p. 7

4.2. Ablation Studies

  • RAW input improves novel-view PSNR from 18.29 to 18.42 dB, while exact-label controller training raises it from 18.21 to 18.25 dB over reconstruction-only training.The RAW comparison replaces RAW with corresponding RGB input while keeping the controller unchanged; exact parameter labels address ambiguity in learning actions from reconstruction alone.Source: §4.2 · pp. 7–8
  • The adopted capacity—seven couplings and 16 knots per curve—outperforms alternatives: other coupling counts lose 0.08–0.12 dB, and other knot counts lose 0.12–0.15 dB.The adopted setting is also best of the seven in SSIM and LPIPS; the operator uses 573 coefficients per view.Source: Table 3 · p. 8, §4.2 · p. 8
  • Using both color-action branches reaches 18.25 dB, versus 16.62 dB for curves alone and 18.08 dB for couplings alone.Curves recover brightness but leave color cast; couplings improve chromatic separation but under-correct the smoke veil.Source: §4.2 · p. 8
  • The bounded zero-mean ∆-ISP raises novel-view PSNR from 18.25 to 18.42 dB while remaining a training-only adjustment.The ablation tests whether it improves 3D consistency among independent controller predictions.Source: Table 4 · p. 8, §4.2 · p. 8

5. Conclusion

FujinSplat uses compact invertible color transformations predicted from RAW observations, with a bounded zero-mean residual to reconcile cross-view discrepancies. On RealX3D, it exceeds the strongest comparable baseline by 2.57 dB, while evaluation is limited to one benchmark.

5. Conclusion

  • On RealX3D, FujinSplat reaches 18.42 dB, exceeding the strongest comparable baseline by 2.57 dB without external closed-source generative models.The method has room for improvement in certain scenes relative to hybrid methods using closed-source multimodal and large generative models with ensemble strategies [3] [5] [32] [55].Source: §5 · pp. 8–9
  • A bounded zero-mean ∆-ISP reconciles residual cross-view discrepancies during Gaussian optimization.FujinSplat represents expert smoke corrections with compact, invertible color transformations inferred from RAW.Source: §5 · p. 8
  • The decomposition confines correction to training and retains standard 3DGS rendering at novel poses, offering an alternative to pipelines requiring per-view processing.The scene-agnostic controller is trained on simulated RAW observations with exact parameter labels to predict per-view dehazing actions.Source: §5 · pp. 8–9
  • Evaluation on a single benchmark leaves validation across cameras and smoke conditions outstanding.The authors also report room for improvement in certain scenes compared with hybrid methods using closed-source generative models and ensembles.Source: §5 · p. 9

A. Rank-One Structure of the Smoke Residual

The smoke residual is largely captured by one scene-specific chromatic direction whose strength varies across views. The section also describes a camera-calibrated base ISP, an invertible monotone color flow, and evidence that action outputs are more identifiable than their parameter vectors.

A. Rank-One Structure of the Smoke Residual

B. Base ISP Construction and Calibration

  • The calibrated base ISP matches the camera’s hazy source-view rendering at 25.37 dB and 0.970 SSIM, then remains frozen as a scene-camera property.Its agreement with the paired clean capture is 12.32 dB, within 0.75 dB of the hazy capture’s own 11.57 dB.Source: B. Base ISP Construction and Calibration · p. 13, Table S2 · p. 14

C. Monotone Color Flow Construction

D. Coefficient Observability

  • Only 69 of 573 directions move any probe at the identity, and coefficient distance does not measure distance between color functions.At two scene-mean actions, only 100 and 102 directions move probe output by at least one quantization level per unit step.Source: D. Coefficient Observability · p. 15
  • Across five refits of each of two source pairs, fitted images agree more closely with one another than with their targets, despite differing coefficients.Fit-to-fit image PSNR is 25.1–35.3 dB, compared with median fit-to-target PSNRs of 18.3 and 20.5 dB; coefficient cosine is 0.06–0.44.Source: D. Coefficient Observability · p. 15, Figure S3 · p. 15

E. Expert Action Family Analysis

FujinSplat analyzes expert corrections in RGB-output space and finds a compact action vocabulary. It synthesizes RAW inputs with exact action labels to train the controller.

E. Expert Action Family Analysis

  • For the analysis of Sec. 3.3, each action is represented by its displacement Tp(q) − q evaluated on the 729 RGB probes from a 9×9×9 Cartesian grid, flattened to 2187 coordinates.Source: E. Expert Action Family Analysis · p. 15
  • The vocabulary contains eight prototype actions at four strengths plus identity, with prototypes selected by farthest-point coverage in probe-displacement space.Each prototype is a fitted action from one source view, and each strength scales its 573 coefficients.Source: E. Expert Action Family Analysis · p. 15
  • 0.906 validation accuracy across 33 action classes, 0.43 above the majority guess, shows the vocabulary classifier predicts fitted-action labels from RAW summaries.The largest validation class contains 0.472 of the 53 validation views.Source: E. Expert Action Family Analysis · p. 15

F. Reverse ISP-Action Synthesis

  • Synthesis uses 1,400 external clean captures developed by a calibrated, frozen base ISP, then analytically inverts the sampled action and base to create RAW inputs with exact labels.The forward round trip returns the clamped target to numerical precision; each capture receives one draw and eight spatial presentations.Source: F. Reverse ISP-Action Synthesis · pp. 15–16
  • Measured actions fit a shared contrast and pivot color in linear RAW: contrast ranges from 0.15–0.90, with median 0.36 across 195 pairs.Synthesis carries the measured pivot direction forward and maps clean-image depth range to contrast, matching the synthetic and measured median contrast.Source: F. Reverse ISP-Action Synthesis · pp. 15–16
  • Each sampled action is compiled deterministically into 573 coefficients, with shared curves handling achromatic veil and contrast and couplings carrying chromatic residuals.The curve removes the veil and restores contrast; coupling offsets are averaged over rising-curve knots and divided among seven stages.Source: F. Reverse ISP-Action Synthesis · p. 16

G. Implementation Details

The implementation uses a shared RAW controller, synthetic exact-label supervision, and a common Gaussian optimization configuration. A bounded, zero-mean per-view residual is optimized before its appearance is compiled into static Gaussians.

  • One 1.51M-parameter controller processes a 3×64×64 RAW summary and emits 573 action coefficients through curve and coupling heads.The summary is demosaiced camera-linear RAW, resized to 64×64, divided by 65535, and clamped to [0, 1].Source: G. Implementation Details · p. 17
  • The per-scene table expands the main ablation across eight scenes and three metrics, with each scene value averaging four held views.The equal-scene mean reproduces the main-paper values.Source: Table S3 · p. 18
  • Controller training regresses exact 573-dimensional synthetic labels, with image reconstruction as an auxiliary loss and eight spatial presentations per observation.Training uses 1,500 steps and no validation split.Source: G. Implementation Details · p. 18
  • Each scene’s Gaussian model starts from 23k–33k benchmark points and runs 18k iterations with spherical-harmonic degree 3 and the standard L1+D-SSIM objective.All eight scenes share the configuration; densification runs from iteration 500 to 6k.Source: G. Implementation Details · p. 18
  • Per-view residual scalars are optimized during iterations 13k–16k, clamped to [0, 1], and re-centered to zero mean; unseen poses use the static Gaussians alone.After iteration 16k, the scalars are frozen and the remaining 2k iterations compile corrected appearance into the static representation.Source: G. Implementation Details · p. 18

H. Evaluation Protocol and Supervision

Evaluation uses paired smoke and clean captures, held-out views, and equal-scene metric averaging. Paired clean source captures enter the method only through expert actions and the correction distribution.

I. Per-Scene Results

The section describes an expanded per-scene ablation and the protocols used to compare physics-based reconstruction and restoration-then-3DGS baselines. The supplied passages establish the comparison setup but do not report outcome values.

I. Per-Scene Results

  • RAW-domain ablation variants share seeds, point cloud, schedule, and frozen base, so comparisons differ only in the component under test.The expanded table covers eight scenes; its equal-scene means reproduce the main-paper values.Source: §I · p. 19
  • The ablation figures show held views under six settings, including vanilla 3DGS on hazy camera RGB and frozen-base output without correction.Each scene’s displayed view is compared with its paired clean capture.Source: §I · p. 19

J. Baseline Protocols

  • All baselines use the same scenes, held poses, clean references, and equal-scene averaging; challenge-method values are quoted rather than rerun.The authors report changing defaults where needed and note their 18k-iteration Gaussian budget is shorter than restoration baselines’ 30k.Source: J. Baseline Protocols · p. 19
  • Physics-based methods use hazy RGB source views and the benchmark camera poses, while restoration baselines replace only training images before vanilla 3DGS.The restoration pipelines use pretrained weights without RealX3D fine-tuning; held images, poses, and initial points remain unchanged.Source: J. Baseline Protocols · p. 19

K. Limitations

FujinSplat’s color-only correction cannot recover smoke-destroyed sensor texture. Its correction distribution has untested transfer scope, and ConvIR and closed-source generative-model pipelines retain advantages on specific reported comparisons.

K. Limitations

  • The global color action moves color values, not pixels, so smoke-destroyed sensor texture is not recovered.Source: K. Limitations · pp. 19–20
  • ConvIR has better average LPIPS, 0.493 against 0.541, while trailing by 2.57 dB in PSNR.Source: K. Limitations · p. 20
  • Transfer to other cameras, expert styles, and denser smoke is untested because the correction distribution was estimated on one benchmark under one development convention.Source: K. Limitations · p. 20
  • On the seven-scene challenge subset, pipelines querying a closed-source generative model per training view remain ahead.Source: K. Limitations · p. 20
Loading 2609.06017v2…