PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Analog: Open-Set Film Emulation with Reference-Conditioned 3D LUTs

Yitong Mu

arXiv:2608.14702v1cs.CVcs.GRcs.LGcs.MM

TL;DR

Open-set film emulation needs to reproduce arbitrary reference looks rather than apply fixed, image-agnostic mappings. Deep Analog predicts a reference-conditioned residual 3D LUT and adds reference-driven tone and optical effects; the full pipeline achieves 21.72 dB PSNR while its color stage runs at 1080p in 5.2 ms.

  • Problem

    Fixed, image-agnostic LUT presets cannot reliably reproduce arbitrary reference film looks across different image content, motivating open-set reference-driven emulation.

  • Method

    Deep Analog predicts a reference-conditioned residual 3D LUT from self-supervised synthetic transforms and augments it with reference-driven tone matching, grain, and halation rendering.

  • Results

    21.72 dB PSNR is achieved by the full pipeline, while the color stage runs in 5.2 ms at 1080p and exports an industry-standard .cube LUT.

  • Takeaways & Limitations

    Entropy regularization counters gated-LUT weight collapse, while conditional residual prediction supports open-set film emulation without a fixed LUT basis or paired film data.

  • Takeaways & Limitations

    Synthetic color-transform pairs test inversion of color distortions but do not establish that recovered LUTs match specific film stocks’ spectral responses.

Abstract

from arXiv · show

Film emulation reproduces the look of an analog film stock on a new digital photograph. We target its open-set form -- matching any reference film frame from a single example -- with a 3D lookup table (LUT) predicted from that reference. Real-time image enhancement predicts per-image weights over a fixed bank of 3D LUTs and blends them. We show this is a gated mixture of experts and inherits its failure: trained end-to-end against reconstruction, the gate collapses onto a single expert, so a bank of K LUTs delivers the capacity of one. An entropy term, the enhancement-setting analogue of mixture-of-experts load balancing, restores utilization and recovers about 1 dB PSNR. The deeper constraint survives: a fixed LUT basis is closed-set, freezing the achievable looks at training time. We therefore discard the basis and predict a single 3D LUT as a residual from a reference image (StyleLUTNet), trained by self-supervision on procedurally generated color transforms. The conditional design removes the gate and generalizes open-set to unseen film stocks without paired data or retraining. Around this color backbone we build Deep Analog, a film-emulation pipeline that adds histogram-based tone matching and a physics-informed optical renderer -- multi-scale grain and per-channel halation driven by parameters an inverse network regresses from the reference. On 350 self-supervised pairs the color stage reaches 22.05 dB PSNR / 0.925 SSIM and the full pipeline 21.72 dB / 0.923; the color path runs in 5.2 ms at 1080p (192 FPS) and exports a portable .cube LUT for standard editing tools. A second degeneracy in conditional LUT training -- residual-scale collapse -- shares the root cause and yields a general principle: auxiliary regularization must stay subordinate to reconstruction.

1. Introduction

The paper diagnoses image-adaptive multi-LUT blending as a gated mixture of experts that collapses and remains closed-set. It replaces the fixed basis with stable residual single-LUT prediction and adds a physics-informed renderer for analog film effects.

  • Structural diagnosis: Multi-LUT blending is a softmax-gated mixture of experts whose end-to-end training can collapse effective capacity to one LUT.The fixed-bank mechanism is presented as image-adaptive but inherits expert-collapse behavior.
  • Structural diagnosis: An entropy penalty restores near-uniform expert utilization and recovers about 1 dB PSNR, while exposing the fixed basis as closed-set.The passage also reports 0.7–1.0 dB recovery at matched training budget.
  • Conditional LUT design: Residual parameterization over the identity map stabilizes direct single-LUT prediction without a basis or gate.A 33×33×33 LUT has over 100K free parameters, and unconstrained regression can produce non-smooth, non-monotonic color maps.
  • Physics-informed rendering: Deep Analog combines conditional LUT prediction with CDF tone matching, multi-scale luminance-dependent grain, and wavelength-scaled per-channel halation.FilmAnalyzerNet regresses the physical rendering parameters from the reference, and the renderer supports portable .cube/.json preset export.
  • Research questions: The investigation asks whether multi-LUT collapse can be prevented, whether one reference-conditioned LUT generalizes to unseen stocks, and whether reference-driven rendering reproduces grain and halation.These questions organize the study around collapse, open-set reconstruction, and unsupervised physical effects.

2. Related Work

Prior enhancement methods commonly predict image-specific 3D LUTs, including reference-conditioned variants, but fixed LUT bases remain collapse-prone and closed-set while pixel-independent mappings cannot model spatial film effects. Deep Analog addresses these gaps with reference-conditioned LUT prediction alongside physically modeled grain and halation, supported by self-supervised transform inversion.

  • Image-adaptive LUT enhancement: Image-adaptive enhancement typically predicts per-image weights over a bank of learnable 3D LUTs, blending them at fixed, resolution-independent application cost.Zeng et al. established this paradigm and reported submillisecond inference after weight prediction.
  • Limitations of LUTs: 3D LUTs cannot express spatially varying grain, halation, or local contrast adaptation because identical RGB inputs receive identical outputs regardless of context.Bilateral-grid modulation restores some spatial adaptivity, but existing objectives remain focused on photo enhancement rather than film effects.
  • Reference-conditioned enhancement: Reference-conditioned methods enable arbitrary style transfer but still gate a fixed LUT basis and inherit pixel-independent limitations.StarEnhancer conditions its weight predictor on an external style reference for multi-style enhancement from one model.
  • Self-supervised transform learning: StyleLUTNet uses self-supervised inversion of known color augmentations to recover a transform as a 3D LUT from a reference and transformed target.Because target pixels are known exactly, training can act directly in pixel and histogram space without contrastive negative sampling.
  • Research gaps: The literature identifies multi-LUT collapse and closed-set looks, reference-conditioned fixed-basis gating, and missing reference-based estimation of physical effect parameters as three gaps.Deep Analog combines a single LUT predicted from an arbitrary reference with physical rendering to address these limitations.

3. Method · 3.1. System Overview

Deep Analog transforms a target photograph according to a reference film scan through a six-stage pipeline. Its architecture combines conditional LUT-based color grading, reference-statistics analysis, and learned physical rendering parameters into a portable preset.

  • 3. Method: Deep Analog processes a target photograph conditioned on a reference film scan through six sequential stages.The pipeline is presented as the system overview for the method.
  • 3.1. System Overview: Reference analysis extracts grain, halation, tonal characteristics, black and white points, and shadow/highlight color casts.FilmAnalyzerNet predicts grain and halation parameters, while a statistical analyzer measures reference tone and color statistics.
  • 3.1. System Overview: StyleLUTNet predicts a conditional 3D LUT from the reference and applies it to the full-resolution target by trilinear interpolation.The color-grading stage uses the reference scan as conditioning input.
  • 3.1. System Overview: StyleLUTNet encodes a resized reference scan into a residual 17^3 LUT, adds it to a fixed identity LUT, and upsamples the result to 33^3.This architecture constructs the final LUT from a reference-conditioned residual.
  • 3.1. System Overview: FilmAnalyzerNet predicts seven groups of physical parameters comprising 11 scalar outputs.These parameters support the optical components of the architecture.
  • 3.1. System Overview: A deterministic analyzer measures reference CDFs, black and white points, and shadow/highlight color casts.These statistics condition the color, tone, and optical components.
  • 3.1. System Overview: The predicted LUT, reference statistics, and physical parameters jointly condition the color, tone, and optical components.The architecture combines learned and deterministic conditioning paths.
  • 3.1. System Overview: The six stages form a portable preset, while Figure 3 presents the visible image progression through them.Figure 2 summarizes the model and conditioning paths.

3.2. StyleLUTNet: Conditional 3D LUT Prediction

StyleLUTNet predicts a reference-conditioned 3D LUT by residual learning from global color statistics, avoiding a fixed LUT basis while remaining stable through identity initialization. Self-supervised random color transforms train it to invert varied looks without paired film-scan data.

  • Architecture: StyleLUTNet predicts a per-reference 3D LUT as a residual from a fixed identity LUT, enabling direct open-set color transformation without a learned fixed basis.A 224×224 ImageNet-1K-pretrained ResNet-18 summarizes the reference into a 512-dimensional feature, while a four-layer MLP decodes the LUT residual.
  • Architecture: 7.3× reduction in decoder output dimensionality comes from predicting a 17^3 LUT and upsampling it to 33^3 by 3D trilinear interpolation.The low-resolution grid reduces outputs from 107,811 to 14,739, while interpolation serves interface compatibility with 33^3 baseline LUTs rather than adding color information.
  • Residual parameterization: Residual prediction stabilizes full-LUT regression by starting at the identity operator and learning only a deviation from it.This preserves the stability of basis-and-blend designs without restricting outputs to a fixed set of learned tables.
  • Residual parameterization: α collapsed from 0.1 to 0.04 over 100 training epochs when regularization outweighed reconstruction at a 15:1 ratio, producing near-identity output.Shrinking the learnable residual scale reduced smoothness and monotonicity penalties more efficiently than improving color accuracy.
  • Self-supervised training: StyleLUTNet trains without paired film-scan datasets by learning to invert random compositions of 2–4 color transformations applied identically to reference and target images.Transformation ranges span and slightly exceed real film-stock variation, encouraging generalization to unseen film looks.

3.3. SwinLUT: Multi-LUT Architecture and Weight Collapse

SwinLUT uses a Swin-T encoder to blend five learnable basis LUTs, but standard softmax training collapses onto one expert. Temperature scaling, entropy regularization, and diversity loss restore weight diversity without overcoming the fixed basis’s closed-set limitation.

  • Architecture: SwinLUT blends K = 5 learnable basis LUTs using softmax weights predicted by a Swin-T encoder.The encoder produces a style representation that an MLP maps to the blending weights.
  • Weight collapse: Standard softmax training collapses the gate onto one dominant expert, leaving the other four basis LUTs effectively unused.The unused LUTs receive negligible gradient because early advantages compound through positive feedback.
  • Mitigation: Three interventions restore weight diversity: temperature scaling, entropy regularization, and pairwise diversity loss.The learnable temperature converges to ≈2.2, while entropy penalizes peaked distributions and diversity loss discourages identical LUT mappings.
  • Mitigation: Entropy regularization accounts for most of the recovery, whereas diversity loss is marginal when entropy regularization is active.This conclusion is supported by the full ablation described for Section 4.5.
  • Limitation: Even after collapse mitigation, the fixed LUT basis remains closed-set, requires paired training data, and limits the achievable looks.This limitation motivates the conditional StyleLUTNet design rather than indicating an architectural defect.

3.4. Film Physics Module

The Film Physics Module complements the neural LUT with reference-derived tone matching and analytic film-tone corrections, then adds spatially varying grain and halation that a 3D LUT cannot represent. Its operations use closed-form reference statistics and luminance-dependent optical effects to reproduce analog film characteristics beyond global color shifts.

  • Photometric Response: CDF-based per-channel matching aligns the LUT-graded image to the reference’s full tonal distribution while Gaussian smoothing and dithering suppress quantization banding.CDFs use 256 uniform sample points; smoothing applies σ = 2 bins with kernel size 9, followed by ±0.5/256 triangular dithering.
  • Analytic Film Tone: Reference statistics drive residual tone corrections: black-point lift, quadratic highlight rolloff, and localized shadow/highlight tinting.The module measures the 2nd and 98th percentile luminance points plus mean colors from the darkest and brightest 15% of pixels; black lift activates above 0.02.
  • Optical Effects: Three-octave, luminance-dependent grain models film texture across spatial scales, with relative octave weights of 50%, 33%, and 17%.A learned sigmoid-over-quadratic mask makes shadows grainier than highlights, while predicted parameters control grain intensity.
  • Optical Effects: Grain and halation remain spatially varying effects, including per-channel spatial spread, and therefore cannot be represented by a 3D LUT alone.The tone stage is a global per-channel mapping, whereas grain and halation encode spatial responses and are separately visualized on controlled synthetic scenes.

3.5. FilmAnalyzerNet: Inverse Parameter Estimation

FilmAnalyzerNet estimates the physical parameters governing grain and halation from a resized reference image. Its constrained outputs are learned self-supervised from synthetic renders with known parameters, avoiding real-film physical measurements.

  • Architecture: FilmAnalyzerNet uses a ResNet-18 backbone and seven independent three-layer MLP heads to regress 11 grain- and halation-rendering parameters.The backbone produces a 512-dimensional feature vector; each head maps 512 → 256 → 128 → output with ReLU activations and dropout p = 0.3.
  • Physical constraints: Output activations enforce physical constraints, including positive grain intensity with a minimum floor and grain size bounded to [0.5, ∞).Grain intensity uses softplus × 0.05 + 0.005, while grain size uses softplus × 1.5 + 0.5.
  • Physical constraints: Halation threshold is confined to [0.5, 0.95], radius has a minimum extent of 2.0, and intensity is capped at moderate strength.The threshold uses sigmoid×0.45+0.5, radius uses softplus × 4.0 + 2.0, and intensity uses sigmoid × 0.5.
  • Training: Synthetic film-look data supervise parameter recovery by rendering grain and halation with known parameters on clean images.This self-supervised loop avoids requiring ground-truth physical measurements of real film stocks.

3.6. Loss Functions

The three networks use related but distinct objectives: SwinLUT adds gate regularization, StyleLUTNet combines reconstruction with LUT and histogram constraints, and parameter regression balances heterogeneous output scales. Regularization is kept subordinate to reconstruction to avoid residual-scale collapse while maintaining well-behaved LUTs.

  • Network objectives: SwinLUT extends reconstruction and LUT-regularity losses with two gate-level regularizers targeting weight collapse.StyleLUTNet retains LUT regularity, drops gate terms, and adds histogram supervision because it predicts a single conditional LUT.
  • StyleLUTNet losses: StyleLUTNet’s losses constrain smoothness, monotonicity, and per-channel histogram transfer alongside L1 reconstruction.Total variation suppresses oscillatory LUT mappings, monotonicity loss prevents tone reversal, and histogram loss supervises distribution transfer.
  • Regularization balance: ∼10−4 weighted smoothness and monotonicity terms converge orders of magnitude below reconstruction loss, preserving LUT behavior without competing with the task.Excessively large λs and λm relative to Lpixel cause residual-scale collapse.
  • Multi-LUT probe: The multi-LUT probe carries over the published objective and parameter values while adding gate-focused modifications.The passage identifies this objective as introduced in Section 3.3 and states that baseline values isolate modifications to the gate.
  • Parameter regression: w = 100, 50, 10, 5, and 1 weight parameter-regression errors to compensate for heterogeneous scales across grain, halation, bias, and luminance outputs.Grain intensity is near 10−2 while halation radius exceeds 2, so uniform weighting would let large-magnitude parameters dominate.

3.7. Differentiable Trilinear Interpolation

StyleLUTNet and SwinLUT apply predicted LUTs with the same differentiable trilinear interpolation operator, implemented in pure PyTorch for modern hardware and training features.

  • Operator: Both StyleLUTNet and SwinLUT apply their predicted LUTs through the same trilinear interpolation operator.The operator treats each pixel as a 3D sampling coordinate in the LUT volume.
  • Implementation: The pure-PyTorch implementation replaces deprecated C++/CUDA extensions and supports automatic mixed precision without modification.It introduces no measurable throughput penalty at the training batch size B =4 on modern PyTorch and A100-generation GPUs.

4. Experiments

Experiments show that entropy regularization prevents LUT-gating collapse, while excessive auxiliary regularization causes residual-scale collapse. The complete pipeline preserves real-time color processing and transfers reference film appearance across stocks and scenes, with texture stages intentionally trading pixel metrics for realism.

  • LUT ablations: 21.27 dB PSNR and 0.9026 SSIM accompanied LUT-weight collapse, with one basis LUT receiving mean weight 0.947 and four effectively ignored.Mean weight entropy was 0.16 of the possible 1.61, indicating under 10% of gate capacity was used.
  • LUT ablations: 22.19 dB PSNR and 0.9289 SSIM in V2 improved over V1 by +0.92 dB and +0.026 as entropy rose to 1.588 and the dominant weight fell to 0.208.Removing entropy caused entropy to reach zero, the dominant weight to reach 1.0, and PSNR to fall to 20.96 dB; diversity removal retained entropy at 1.571.
  • Conditional LUT training: Residual-scale collapse produced near-identity LUTs when auxiliary regularization outweighed reconstruction, revealing the shared principle that regularizers must remain subordinate to the primary task.With λs = 10 and λm = 5, the combined regularization-to-pixel ratio was 15:1 and the residual scale α shrank from 0.1.
  • Pipeline evaluation: 21.72 dB was achieved by the jointly trained color, grain, and halation pipeline, only 0.04 dB below the separately trained pipeline.Color grading raised PSNR by +0.86 dB and reduced LPIPS by 40%; grain reduced PSNR by 0.23 dB while SSIM changed from 0.925 to 0.923, and halation contributed a further 0.06 dB decrease.
  • Efficiency: 5.2 ms at 1080p (192 FPS) and 12.0 ms at 4K (83 FPS) establish real-time StyleLUT color grading, while GPU acceleration remains important for full-pipeline inference.On RTX 4060, StyleLUT was 2.1× faster than Apple MPS and the full pipeline 2.6× faster at 1080p; CPU-only StyleLUT inference was 20× slower.
  • Generalization and qualitative results: A single reference reproduced distinct film palettes across five stocks and seven scene conditions with unchanged model weights, while grain and halation added physically motivated texture effects.StyleLUTNet’s 22.05 dB self-supervised color-transfer result is not meaningfully comparable to FiveK closed-set baselines because it addresses a different task.

5. Discussion

The discussion shows that auxiliary regularization can induce collapse in LUT-based models, while conditional single-LUT prediction provides open-set generalization and practical deployment advantages. It also identifies evaluation limitations, boundary failures, and directions for improving transfer robustness.

  • Mixture-of-experts collapse: Softmax-gated LUT mixtures predictably collapse onto one expert when trained end-to-end, reducing a bank’s effective capacity to a single LUT.The optimizer specializes one expert and routes all inputs to it, motivating load-balancing losses.
  • Mixture-of-experts collapse: About 1 dB of PSNR is recovered when temperature scaling and entropy regularization restore full LUT-weight diversity.The intervention adds one learnable scalar and one entropy-loss term.
  • Regularization principle: Residual-scale collapse and weight collapse show that auxiliary regularization must remain subordinate to reconstruction.In both cases, optimization exploits weight distributions or scale parameters to reduce a secondary loss at reconstruction’s expense.
  • Model comparison: 22.16 dB for five basis LUTs and 22.05 dB for StyleLUTNet are not directly comparable because they use different tasks, ground truths, and evaluation protocols.Their proximity suggests that basis LUTs primarily increase effective LUT resolution, but a same-protocol comparison is needed.
  • Open-set generalization: A conditional single-LUT architecture generalizes to arbitrary references without retraining, unlike a multi-LUT bank whose target styles are fixed during training.This open-set capability is essential for film emulation, where users may provide any film scan.
  • Validity and limitations: The synthetic evaluation tests inversion of generated color distortions and renderer parameters, not fidelity to measured film-stock spectra or physical responses.Open-set generalization is shown qualitatively, while generalization beyond tested stocks and scenes remains unmeasured.
  • Validity and limitations: PSNR decreases as grain and halation are added because deliberate optical distortions move pixels from the clean reference, making PSNR an unreliable aesthetic-quality proxy.SSIM is more stable across the optical stages because it evaluates local luminance, contrast, and structure.
  • Failure cases and extensions: Mean saturation increased from 0.14 to 0.64 and clipped-pixel fraction from 12.5% to 26.2% in automatically selected boundary cases.These failures motivate gamut-aware LUT constraints, highlight-preserving tone matching, adaptive transfer strength, and scene-adaptive conditioning.

6. Conclusion

The conclusion identifies expert collapse and residual-scale collapse as related degeneracies, then motivates Deep Analog’s open-set, reference-conditioned LUT and physics-informed rendering pipeline. It reports practical performance and portability while noting limitations in temporal consistency, scene awareness, optical modeling, feature strength, and model size.

  • Degeneracies and regularization: Entropy regularization counters softmax-gated multi-LUT collapse onto one expert, recovering about 1 dB of PSNR with a one-line loss modification.Residual-scale collapse in conditional LUT training shares the same root cause, motivating the principle that auxiliary regularization should remain subordinate to reconstruction.
  • Open-set design: StyleLUTNet predicts one reference-conditioned 3D LUT through self-supervision, enabling arbitrary film-look transfer in the open set without paired data.Deep Analog adds a physics-informed optical branch that renders grain and halation from parameters regressed from the reference.
  • Performance and deployment: 21.72 dB PSNR is achieved by the full pipeline, while color grading runs in 5.2 ms at 1080p and exports to the industry-standard .cube format.The unoptimized CDF tone-matching stage concentrates the remaining latency and could be accelerated with a compiled kernel.
  • Limitations and future work: Video use would require temporal consistency constraints for grain and halation, while the extracted look remains affected by the reference capture’s illumination.The optical branch renders grain and halation but does not model the film’s modulation transfer function (MTF).
  • Limitations and future work: Stronger self-supervised ViT reference features and a lighter decoder could improve transferability and enable on-device inference, respectively.The decoder MLP accounts for most of StyleLUTNet’s 82.6M parameters.
Loading 2608.14702v1…