PaperlayerView exact revision on arXiv

Source-linked AI summary

Color Pass-Through via Camera-Display Coupling

Ruikang Li, Molin Li, Jiarui Wu, Zhe Wei, Pengpeng Liu, Tianfan Xue

arXiv:2607.12746v1cs.CV

TL;DR

Smartphone camera–display pipelines still leave a perceptual color gap, especially across diverse real-world scenes. Color Pass-Through couples both devices in an end-to-end learned system, enabling observer-specific calibration and improved pass-through across scenes and illuminants.

  • Problem

    Smartphone capture and display still produce noticeable chromaticity and lightness discrepancies, while conventional calibration and post-processing lack consistent faithful reproduction across diverse, unconstrained scenes.

  • Method

    Color Pass-Through couples camera and display through an end-to-end learnable neural projector, followed by one-time coefficient calibration for a target observer.

  • Results

    The system demonstrates robust color pass-through across diverse scenes and illuminants using quantitative measurements and subjective user studies.

  • Takeaways & Limitations

    Coupling the camera–display pair enables observer-specific color correction across diverse scenes and illuminants.

  • Takeaways & Limitations

    Color pass-through is guaranteed only when the camera itself is the observer; differing observers can produce noticeable displayed color shifts.

Abstract

from arXiv · show

When a real-world scene is captured by a smartphone camera and viewed on its screen, the displayed image often differs noticeably from the original scene in color, brightness, and contrast. This gap persists despite substantial advances in both modern cameras and displays. A key reason is that most pipelines factor the high-dimensional capture-to-display process into two separately calibrated camera and display stages, and then connect them through low-dimensional color transforms, leading to information bottlenecks and inevitable error accumulation. To address this systemic challenge, we propose Color Pass-Through, an end-to-end learned framework that operates directly on captured images. Our key insight is to treat the camera and display as a coupled system rather than calibrating them in isolation. Coupling the camera and display yields two practical advantages: (1) it brings the entire real-world scenes to the display via end-to-end optimization, and (2) it allows efficient one-step calibration for each distinct observer via complete capture-to-display path. We validate Color Pass-Through using both digital and human observers. Compared with representative baselines, our method achieves an average gain of +2.0 points on a 5-point user study and more than 2x improvement on quantitative metrics, demonstrating improved reproduction of the perceived color of the original scene.

1 Introduction

Color Pass-Through addresses perceptual color and brightness mismatches by jointly learning the camera–display mapping for a specific observer. It further corrects residual camera-metamerism errors through a one-step observer-specific calibration and validates the system with objective measurements and user studies.

  • Problem: Smartphone capture-and-display pipelines can shift scene chromaticity and lightness, producing images that appear overly bright, desaturated, or washed out.Existing auto-white-balance and color-checker calibration methods reduce color casts or map images to standard color spaces but do not reproduce the same color.
  • Core approach: Color Pass-Through couples the camera and display as one system bridged by an end-to-end learnable neural projector.The approach directly maps displayed colors to original scene colors for a specific observer, reducing error accumulation and information bottlenecks.
  • Calibration: A lightweight pixel-wise neural network learns the device-specific projector from re-captured samples rendered on the target display and re-imaged by the paired camera.The re-capture protocol supplies measurements that supervise the coupled camera–display mapping.
  • Residual correction: Residual observer-dependent color casts arise from camera metamerism, especially spectral components in the camera’s metameric-black subspace.The camera-null space is approximated by a dominant component and compensated using a learned predictor with one observer-specific calibration coefficient.
  • Experimental validation: The method is evaluated through DSLR-based objective measurements and subjective user studies across diverse scenes and illuminants.The DSLR serves as a fixed digital observer for quantitative results and figures, while human observers provide subjective evaluation.

2 Related Work

Related work typically separates color reproduction into calibrated camera and display stages connected through a device-independent reference space. It also includes computational color constancy and display-specific modeling of illumination, viewing conditions, device response, and ambient adaptation.

  • Traditional color reproduction pipelines: Traditional ICC workflows use CIE XYZ to mediate device-to-device color transforms through decoupled capture and output calibration stages.Capture devices map sensor measurements into the reference space, while displays and printers apply inverse mappings to device-dependent signals.
  • Computational color constancy: Computational color constancy corrects scene-illumination effects under the assumption that perceptual colors remain consistent across lighting conditions.Traditional approaches use hand-crafted assumptions, while learning-based methods estimate illuminants directly from data.
  • Display color management: Display color management transforms device-dependent signals through a device-independent reference space and uses chromatic adaptation for illuminant differences.Examples include Bradford and von Kries transforms.
  • Display color management: Display characterization models device input-to-emitted-radiance mappings with parametric transfer functions and adapts them to ambient lighting conditions.Gain–Offset–Gamma models and Apple’s True Tone are representative examples.

3 Derivation of Color-Accurate Pass-Through

The derivation shows that exact radiance pass-through is impossible with three-channel camera and display representations, so the method instead targets observer-dependent perceptual color equivalence. This goal leads to a correction mapping conditioned on observer sensitivities, whose direct calibration is impractical because observers vary and are unknown.

  • Radiance pass-through limitation: Exact radiance pass-through requires DFC to equal the identity, but its rank is at most 3 and therefore cannot reconstruct arbitrary L-dimensional radiance when L ≫ 3.The camera–display pair acts as an autoencoder that discards spectral information during transfer.
  • Color pass-through objective: Color pass-through instead requires the display to appear similar to the scene for a target observer, which may be a human or a three-channel camera.The display radiance need not match the scene radiance exactly; only the observer’s perceived colors must agree.
  • Observer-conditioned formulation: For observer sensitivities M, the correction satisfies M s*_i = M D F_M C s_i for every scene point.This enforces equality between the observer’s displayed and scene colors after correcting the captured image.
  • Observer-conditioned formulation: The formal observer-specific correction is F_M = (MD)†MC†, where † denotes the Moore–Penrose pseudoinverse.The formulation uses MD ∈ R3×3 and MC† ∈ R3×L.
  • Practical calibration limitation: Directly calibrating F_M is difficult because observer sensitivity M is unknown, high-dimensional, and varies across observers, making (MD)† unreliable to calibrate.The method therefore approximates F_M through a decomposition described subsequently.

4 Learning Color Pass-Through

Color Pass-Through decomposes observer-specific color pass-through into a learned camera–display projector and a camera-null correction, enabling independent training and one-step observer calibration. The projector is learned from re-captured data to handle nonlinear camera–display behavior, while the correction compensates observer-dependent spectral deviations invisible to the camera.

  • Model decomposition: The method decomposes observer-specific pass-through into a camera–display projection term FC and a camera-null correction δi for camera-invisible spectral deviations.The two predictors are trained independently and combined end-to-end at inference.
  • Learning the projector: Nonlinear camera–display projector FC is learned by rendering sampled RGB supervision on the display, re-capturing it with the camera, aligning pixels, and training a network to invert the process.The learned mapping approximates (CD)† from re-captured pairs rather than enforcing linearity.
  • Projector limitations: The learned projector captures higher-frequency details and improves quantitative results over alternative learned fits, but guarantees pass-through only when the camera itself is the observer.For a different observer, such as a DSLR, the projector alone produces noticeable color shifts.
  • Observer-specific correction: Camera-null correction estimates residual observer-specific color cast with a single calibration coefficient and compensates it to achieve pass-through for target observer M.The correction addresses deviations that remain perceptually visible to viewers even when a same-model digital observer sees matching colors.
  • Practical calibration: 93% of variance is explained by the first principal component, motivating a rank-1 correction with K = 1; φ ∈R3×1 therefore contains only 3 unknown calibration variables.The coefficient φ relates only to observer M and display D and remains nearly stable across varying in-the-wild scenes.

5 RESULTS

The results evaluate Color Pass-Through against camera-side and ICC-style baselines using quantitative, qualitative, and human-observer experiments. The method preserves more consistent colors, remains effective with limited training data, and shows calibration stability across illuminants.

  • Baseline comparisons: Color Pass-Through preserves more consistent colors than other baselines in separated-view comparisons and reproduces closer colors in direct in-scene comparisons.The baselines include default smartphone processing, learned multi-illuminant auto-white-balance, and ICC-style correction pipelines.
  • Evaluation: Evaluation spans ten correlated color temperatures, five RGB-LED illuminants, 24 ColorChecker patches, and diverse real scenes.Quantitative results are summarized in Table 2, while qualitative comparisons appear in Figures 8 and 9.
  • Human evaluation: Ten human observers rate brightness and color accuracy across ten unseen scenes after selecting personalized calibration coefficients from a one-time grid search.Ratings measure how well each displayed image matches the real scene on a 5-point Likert scale.
  • Ablations and robustness: With 100 training images, the learned bFC still preserves similar colors, demonstrating learning efficiency under limited data.The experiment progressively subsamples the training set to test performance with reduced supervision.
  • Ablations and robustness: Observers usually retain their original φ after each entry is perturbed by ±0.015, indicating cross-illuminant calibration stability across 10 scenes.The stability study visualizes selections from three representative observers.

6 Conclusion

Color Pass-Through is an end-to-end learned correction applied to captured images through a coupled camera–display pair. Combining two predictors at inference time with one-time target-observer calibration enables color pass-through across diverse scenes and illuminants, supported by evaluation metrics and user studies.

  • Color Pass-Through applies end-to-end learned correction to captured images through a coupled camera–display pair.
  • Two optimized predictors are combined at inference time, followed by one-time coefficient calibration for a target observer.
  • The calibrated system enables color pass-through across diverse scenes and illuminants, as confirmed by evaluation metrics and user studies.

Supplementary Material · A Model Details

The supplementary material extends Color Pass-Through with model implementation details, mathematical derivations, additional experiments, applications, and limitations. Section A focuses on the full model implementation and key component designs.

  • Supplementary Material: The supplementary document contains five sections extending the main paper’s Color Pass-Through method.These sections cover model details, derivations, experiments, applications, and future improvements.
  • A Model Details: Section A describes the model in detail, including its full implementation and key component designs.
  • Supplementary Material: Section B derives the full model in the linear case and extends the camera–display projector to nonlinear settings with low-rank camera-null correction.These derivations lead to the final formulation.
  • Supplementary Material: Section C reports extra experiments evaluating AvgPool augment effectiveness and comparing baselines for learning \protect \mathcal {F}_{\mathbf C}.
  • Supplementary Material: Section D discusses applications including compatibility with in-phone camera processing for richer details and lower noise.
  • Supplementary Material: Section D also extends Color Pass-Through to high-dynamic-range scenes to alleviate overexposure in highlight regions.
  • Supplementary Material: Section E discusses limitations of the current version and outlines potential directions for future improvement.

A.1 Full Model Implementation · A.2 Estimation and Pretraining · Estimating camera spectral sensitivity C.

Color Pass-Through uses separate estimation/pretraining, observer-specific calibration, and inference stages. Camera sensitivity is estimated with a low-dimensional PCA model, then used to derive camera null-space components for the coupled predictors.

  • A.1 Full Model Implementation: The full framework comprises estimation and pretraining, calibration, and inference stages.The calibration coefficient φ is estimated once for the target human or digital observer and then fixed for evaluation.
  • A.2 Estimation and Pretraining: The estimation and pretraining stage estimates camera sensitivity C, derives the first null-space component e, and independently trains two predictor networks.The predictors are a learned camera–display projector bFC(·) and a second predictor combined with it for pass-through.
  • A.2 Estimation and Pretraining: Calibration executes the pretrained full model on a raw Pro-mode image and estimates φ ∈R3×1 through a 3D-grid procedure.The resulting coefficient is fixed for the target observer before validation and evaluation.
  • Estimating camera spectral sensitivity C.: Camera sensitivity C is modeled over 400–720 nm with a low-dimensional PCA basis using ColorChecker reflectances, illuminant spectra, and measured RGB responses.Each channel sensitivity is normalized and represented in a PCA subspace learned from measured camera sensitivities.
  • Estimating camera spectral sensitivity C.: More than 97% of the sensitivity variance is explained by the first two principal components, making a two-dimensional model sufficient for each RGB channel.The model includes channel-dependent gains and PCA coefficients.
  • Estimating camera spectral sensitivity C.: Under an unknown daylight illuminant, daylight basis parameters and PCA coefficients are iteratively optimized to minimize forward-model error.The daylight parameters are determined by correlated color temperature t, using the CIE daylight basis.
  • Estimating camera spectral sensitivity C.: 15 ColorChecker images are captured under daylight illumination and averaged to estimate camera sensitivity.The estimated sensitivity is aligned with a hyperspectral grid containing L = 31 spectral bands.
  • Estimating camera spectral sensitivity C.: The camera null space has dimension d = 28, and spectra are decomposed into camera-invisible metameric-black components by orthogonal projection.The null-space basis N is orthonormal and computed for null(C⊤), with emphasis on the first principal component e.

Estimating the first principal component e. · A.3 Observer-Specific Calibration via a 3D-grid

The method compresses null-space variation into a low-dimensional representation, finding that the first principal component captures most variance. It then calibrates an observer-specific 3D coefficient once by selecting the displayed result that best matches the real scene.

  • Estimating the first principal component e.: s_null is projected into null-space coordinates z = s^T N ∈ R^d before PCA is performed in the reduced space.The covariance is estimated over ARAD-1K, CAVE, and ICVL hyperspectral fitting sets.
  • Estimating the first principal component e.: Over 1,182 hyperspectral images, the first principal component explains over 90% of total null-space variance.The images contain 6.02 × 108 pixels in total.
  • Estimating the first principal component e.: The first three components explain over 98% of null-space variance cumulatively.This indicates that null-space variation is highly concentrated along the leading principal directions.
  • Estimating the first principal component e.: Reconstruction using the top three components preserves nearly 99% of the total energy.Thus, three principal components retain nearly all observed null-space signal energy.
  • A.3 Observer-Specific Calibration via a 3D-grid: After estimation and pretraining, the full Color Pass-Through model calibrates φ ∈ R^(3×1) once for the observer.Under the low-rank approximation, φ is independent of scene radiance and remains nearly stable across diverse real-world scenes.
  • A.3 Observer-Specific Calibration via a 3D-grid: Calibration samples results under different φ values, renders them on the display, and has the observer select the value providing the best match.The selected φ is then fixed for subsequent experiments.
  • A.3 Observer-Specific Calibration via a 3D-grid: For human observers, φ selection uses overall tone and fine-grained color details; for digital observers, it minimizes MSE over ColorChecker patches.Fig. 16 illustrates the one-time 3D-grid calibration workflow in an arbitrary scene.

B Detailed Mathematical Proof · B.1 Review of Full Model Interpretation in the Linear Case · B.2 Nonlinear Extension of the Camera–Display Projector

The mathematical proof interprets the linear observer-aware mapping as a camera–display projector applied to a residual-corrected camera color, then extends this composition to nonlinear display and learned projector operators. The nonlinear formulation preserves the same full-model structure and remains consistent with the linear case.

  • B.1 Review of Full Model Interpretation in the Linear Case: The observer-induced spectral projector is defined as P_M := D(MD)†M, yielding the bridge F_M C s_i = F_C C P_M s_i.This connects the observer-aware mapping to the camera–display projector through the projected signal.
  • B.1 Review of Full Model Interpretation in the Linear Case: The residual term δ_i := C(I−P_M)s_i makes the projected camera color equal to C s_i−δ_i before applying F_C.The resulting display color observed by M closely matches the scene color M s_i.
  • B.1 Review of Full Model Interpretation in the Linear Case: The linear derivation shows that the observer-aware mapping F_M equals the camera–display projector F_C applied to a residual-corrected camera color.The correction term is δ_i ∈ R^3 in the input space of F_C.
  • B.1 Review of Full Model Interpretation in the Linear Case: The resulting linear full model renders M D through learned F_C after subtracting learned δ_i from the camera color C s_i.This is identified as the linear version of Eq. (11) in the main paper.
  • B.2 Nonlinear Extension of the Camera–Display Projector: The nonlinear extension replaces the linear display matrix D with a nonlinear emission operator 𝒟: R^3→R^L and the effective process T_CD(σ) := C𝒟(σ).This accounts for gamma correction, tone mapping, and other device-dependent nonlinear processing.
  • B.2 Nonlinear Extension of the Camera–Display Projector: The nonlinear camera–display projector 𝓕_C is a learned right inverse of T_CD on the realizable camera-color domain Ω_C.It satisfies T_CD(𝓕_C(c)) = c for every c ∈ Ω_C.
  • B.2 Nonlinear Extension of the Camera–Display Projector: The nonlinear full model retains the linear composition: M observes 𝒟(𝓕_C(C s_i−δ̂_i)), with learned projector and residual correction.The corrected signal must lie in the domain where the camera–display projector is valid.
  • B.2 Nonlinear Extension of the Camera–Display Projector: When 𝒟(σ)=Dσ and 𝓕_C is linear, the nonlinear formulation reduces exactly to the linear model in Eq. (24).In practice, the learned projector is implemented as a nonlinear pixel-wise network, while the resulting display color remains an approximation of M s_i.

B.3 Low-rank Approximation of Camera–Null Correction

The correction term vanishes under an extended Luther–Ives null-space condition, while in the general case it originates entirely from the camera-null component of scene radiance. This structure places the correction in a low-dimensional subspace, motivating a low-rank approximation.

  • Zero correction: The extended Luther–Ives condition null(C) ⊆ null(M) is sufficient for δ_i = 0 for all scene radiances.When M = TC for invertible T, the null spaces are equal, recovering the classical Luther–Ives condition up to an invertible linear transform.
  • Hardware route: Adding camera channels can shrink null(C), making null(C) ⊆ null(M) easier to satisfy and potentially reducing or eliminating the correction term by design.The passage identifies RGBW, other four-channel CFA designs, and multispectral sensors as possible hardware strategies.
  • Radiance decomposition: For the general case, scene radiance decomposes into a displayable component r_i in range(D) and a camera-null component n_i in null(C).The displayable component is r_i := P_C s_i, while n_i := s_i − r_i is the metameric-black component invisible to the camera.
  • Correction source: The displayable component contributes no correction, yielding δ_i = C(I − P_M)n_i.Thus, the correction originates entirely from the camera-null component rather than the displayable component.
  • Low-rank approximation: The correction term lies in a low-dimensional subspace of R^3 parameterized by the same coefficients a_i, supporting a low-rank approximation.Estimating it otherwise requires access to the unobserved spectral component n_i and a high-dimensional operator.

C Additional Experiments … D.2 Extension to High Dynamic Range

The additional experiments show that lightweight spatial cues improve camera–display projector learning, while extensions incorporating smartphone ISP outputs and tone mapping enhance robustness to imaging and display limitations. In particular, AvgPool reduces colored-edge artifacts, ISP augmentation improves visual quality, and tone-mapping-like input processing partly alleviates highlight overexposure.

  • C.2 Comparison of Different Baselines for Learning F_C: Visual fitting comparisons show the proposed network achieves superior quantitative results and lower errors than BPAM [57] and CSRNet [39].BPAM is a CNN-based baseline, whereas CSRNet is an MLP-based baseline.
  • C.1 Evaluate the Effectiveness of Avgpool Layer for Learning FC: AvgPool augmentation provides a lightweight spatial cue that helps identify demosaicing-affected edge pixels while preserving the MLP’s pixel-wise formulation.The method augments the green channel with AvgPool before feeding it to the MLP.
  • C.1 Evaluate the Effectiveness of Avgpool Layer for Learning FC: AvgPool-augmented input makes edge pixels easier to identify and produces fewer colored-edge artifacts than vanilla input.The design consistently improves learning of the camera–display projector F_C.
  • D.1 Augment with In-phone Camera Processing: The model currently supports only single-frame input with relatively simple post-processing, motivating integration with commercial image signal processing pipelines.This is presented as a possible extension rather than a current capability.
  • D.1 Augment with In-phone Camera Processing: Using an HDR-fused image as a condition while retaining the original raw image as input, the design in Fig. 20(d) yields the best results among tested variants.The tested designs are inspired by NiLUT [25], CSRNet [39], SVDLUT [51], SABLUT [50], HDRNet [33], and DeepUPE [75].
  • D.2 Extension to High Dynamic Range: A tone-mapping-like curve uses identity mapping at low intensities and compression at high intensities to address display dynamic-range limitations.The curve is intended to mitigate failures in out-of-range regions where displays cannot represent the full scene range.
  • D.2 Extension to High Dynamic Range: Tone-mapping-like input augmentation can alleviate overexposure to some extent and better preserve high-intensity highlight regions, although a more effective solution remains necessary.Current displays fall far short of the upper range of human visual perception, especially for scenes with strong highlights.

E Limitation

Color Pass-Through is limited to the specific camera–display pair it was trained for and may struggle to reproduce colors faithfully under extreme illumination because displays have limited dynamic range.

  • Generalization across different cameras: The learned model cannot be directly applied to images captured by a different camera because it is designed for a specific camera–display pair.Raw-to-raw translation could potentially be combined with the full model to enable cross-device transfer in the future.
  • Limited dynamic range: Under extreme illumination, such as direct sunlight, limited display dynamic range can cause clipping or compression in bright regions.Faithful reproduction may require HDR capture, specific tone-mapping techniques, and rendering.
Loading 2607.12746v1…