PaperlayerView exact revision on arXiv

Source-linked AI summary

Invertible Image Signal Processing

Yazhou Xing, Zian Qian, Qifeng Chen

arXiv:2103.15061v2eess.IVcs.CV

TL;DR

RAW images retain information useful for editing and vision tasks but are difficult to access because of their size and the losses introduced by conventional ISP and JPEG compression. InvISP redesigns the ISP as a single invertible neural network with a differentiable JPEG simulator, and experiments on two DSLR cameras show substantially better RAW recovery while preserving RGB rendering quality.

  • Problem

    RAW data support image editing and computer vision but are difficult to access, while lossy ISP operations and JPEG compression make accurate recovery from sRGB challenging.

  • Method

    InvISP uses one invertible neural network to jointly map RAW to RGB and RGB to RAW, incorporating a differentiable JPEG simulator.

  • Results

    InvISP outperforms state-of-the-art baselines on two DSLR cameras, recovering much better RAW data without sacrificing RGB rendering performance.

  • Takeaways & Limitations

    The framework enables RAW data compression, image retouching, and HDR reconstruction from the reconstructed RAW data.

Abstract

from arXiv · show

Unprocessed RAW data is a highly valuable image format for image editing and computer vision. However, since the file size of RAW data is huge, most users can only get access to processed and compressed sRGB images. To bridge this gap, we design an Invertible Image Signal Processing (InvISP) pipeline, which not only enables rendering visually appealing sRGB images but also allows recovering nearly perfect RAW data. Due to our framework's inherent reversibility, we can reconstruct realistic RAW data instead of synthesizing RAW data from sRGB images without any memory overhead. We also integrate a differentiable JPEG compression simulator that empowers our framework to reconstruct RAW data from JPEG images. Extensive quantitative and qualitative experiments on two DSLR demonstrate that our method obtains much higher quality in both rendered sRGB images and reconstructed RAW data than alternative methods.

1. Introduction

InvISP redesigns the camera ISP as an invertible pipeline that renders appealing RGB images while recovering RAW data nearly identical to the originals, including from compressed RGB images.

  • Motivation: RAW data preserve scene irradiance and support image editing and computer vision, but their large size makes access difficult.The cited applications include photometric stereo, denoising, reflection removal, and super resolution.
  • Motivation: Existing RAW-reconstruction methods rely on lossy ISP steps, producing recovered RAW images that may differ substantially from the originals.Prior approaches either store mapping parameters, use camera priors, or synthesize RAW data by reversing the ISP order.
  • Approach: InvISP redesigns the camera image signal processing pipeline as an invertible learned mapping for both RAW-to-RGB and RGB-to-RAW conversion.The framework uses one invertible neural network rather than separate forward and reverse networks.
  • Approach: InvISP addresses information loss, visual artifacts, and JPEG compression while preserving visually appealing sRGB rendering.Its design uses bijective transformations, affine coupling layers, invertible 1 × 1 convolution, and a differentiable JPEG simulator.
  • Contribution: The framework reconstructs nearly identical RAW data and supports applications including RAW compression, image retouching, and HDR reconstruction.Experiments on two DSLR cameras demonstrate improved RAW reconstruction without sacrificing RGB rendering performance.

2. Related Work

Prior work studies RAW reconstruction, learned ISP processing, and invertible neural networks, while InvISP applies invertibility directly to the camera ISP for accurate RAW recovery.

  • RAW Image Reconstruction: RAW reconstruction methods encode ISP parameters, use camera priors, map to CIE-XYZ, or synthesize RAW through RGB-RAW-RGB cycles.These approaches address recovering RAW or related representations from sRGB images through explicit metadata, priors, or learned mappings.
  • Image Signal Processing: Existing ISP research learns or reorganizes operations such as demosaicing, denoising, low-light enhancement, and super-resolution with conventional networks.The cited approaches include end-to-end CNNs, two-stage CNN pipelines, and U-net-based processing.
  • Position of InvISP: InvISP differs from prior encoder-decoder ISP methods by using invertible neural networks to enable accurate RAW reconstruction.Its contribution is applying invertibility to the ISP pipeline rather than only learning forward image-processing operations.
  • Invertible Neural Networks: Normalizing-flow research provides invertible transformations through coupling layers, ActNorm, invertible 1 × 1 convolution, and related architectures.These models transform distributions using compositions of tractable invertible functions.

3. Traditional ISP analysis

Traditional ISP operations lose RAW information through quantization, clipping, and JPEG compression, motivating InvISP’s jointly optimized forward and inverse pipeline.

  • Traditional ISP: Traditional ISP converts RAW sensor data into RGB through operations including white balance, demosaicing, denoising, color transformation, and tone mapping.These operations are difficult to invert because several are lossy and are traditionally tuned independently.
  • Quantization and tone mapping: Tone mapping compresses high- and low-intensity values, so subsequent quantization can merge distinct RAW intensities into one RGB value.A 14-bit RAW intensity range of [16313, 16383] can be quantized to the single 8-bit RGB value 255.
  • Quantization and tone mapping: Quantization errors make direct synthesis of 14-bit RAW data from 8-bit sRGB especially difficult in over-exposed regions.The paper reports that InvISP preserves more RAW detail at high-intensity pixels than previous approaches.
  • Out-of-range value clipping: Value clipping discards out-of-range pixels in over- and under-exposed regions and can accumulate errors across manually isolated ISP steps.InvISP jointly optimizes the ISP steps to alleviate clip-error accumulation.
  • JPEG compression: JPEG compression adds further information loss because quantization is its lossy and non-differentiable step.InvISP incorporates a differentiable JPEG simulator that replaces JPEG quantization with differentiable Fourier transformations.
  • Invertible ISP: InvISP’s forward pass demosaics Bayer RAW and applies bijective functions, while its inverse pass reverses these operations to recover RAW from compressed RGB.The framework is designed to account for JPEG compression during optimization and accepts real JPEG images at test time.

4. Method

InvISP uses a single invertible network to map between RAW and sRGB spaces while preserving reversibility. The method incorporates coupling layers, channel permutations, bilinear demosaicing, and a differentiable JPEG simulator to support RAW recovery from compressed images.

  • Invertible Image Signal Processing (InvISP): InvISP composes invertible bijective functions to learn a single RAW-to-sRGB mapping whose reverse recovers RAW data.This avoids using separate networks for the two directions and reduces accumulated mapping error.
  • Invertible Image Signal Processing (InvISP): Affine coupling layers split inputs into parts and use scale and translation transformations, while enhanced coupling updates otherwise unchanged channels.Invertible 1 × 1 convolutions provide learnable channel permutations between coupling layers.
  • Invertible Image Signal Processing (InvISP): The implementation uses bilinear-demosaiced RAW input because invertible networks require identical input and output sizes, while reversing bilinear demosaicing is trivial.The model also omits explicit latent-distribution modeling for the image-to-image translation task.
  • Invertible Image Signal Processing (InvISP): InvISP is trained bidirectionally with L1 loss to produce sRGB images in the forward pass and recover realistic RAW data in the reverse pass.The RGB and RAW reconstruction accuracies are balanced with λ, set to 1 in the main experiments.
  • Differentiable JPEG Simulator: The differentiable JPEG simulator replaces non-differentiable rounding with a Fourier-series approximation, enabling end-to-end optimization for JPEG-distorted inputs.Increasing K improves approximation accuracy but increases runtime; the method empirically sets K to 10, and its rounding curve is shown in Figure 4.
  • Differentiable JPEG Simulator: The proposed rounding function achieves a better balance between RGB rendering and RAW reconstruction than the comparison rounding function.The comparison is reported in Table 1.

5. Experiments

Experiments evaluate InvISP on two DSLR datasets against synthetic-RAW and invertible or encoder-decoder baselines. Results show stronger RAW recovery while maintaining competitive RGB rendering, with applications demonstrated in retouching and HDR reconstruction.

  • Experimental setup: The evaluation uses Canon EOS 5D and Nikon D700 subsets from MIT-Adobe FiveK, training each camera-specific model with an 85:15 train-test split.The datasets contain 777 Canon image pairs and 590 Nikon image pairs.
  • Quantitative comparison: Table 1 compares InvISP with UPI, CycleISP, U-net, and Invertible Grayscale using perceptual metrics for rendered RGB and recovered RAW images.The evaluation uses PSNR and SSIM for RGB images and PSNR for recovered RAW images.
  • Quantitative comparison: More than 13 dB PSNR improvement over UPI and CycleISP demonstrates more accurate RAW recovery than synthetic-RAW baselines.InvISP jointly optimizes RGB rendering and RAW recovery, addressing information loss from quantization, JPEG compression, and saturated-value clipping.
  • Qualitative comparison: InvISP recovers RAW information better than UPI and CycleISP in over-exposed and highlight regions, where their synthetic RAW images differ substantially from ground truth.The ground-truth RAW is visualized through bilinear demosaicing, while other RAW images are visualized through error maps.
  • Applications: The framework is demonstrated for image retouching and HDR reconstruction, with retouching using a pretrained model.These experiments illustrate downstream uses of the reconstructed RAW data.
  • Qualitative comparison: Compared with Invertible Grayscale and U-net, InvISP preserves visually pleasing RGB rendering while producing higher-quality RAW images, including on edges and over-exposed areas.The U-net achieves comparable RGB rendering but performs worse at RAW recovery, while Invertible Grayscale performs relatively poorly on both outputs.

6. Applications

InvISP supports RAW data compression and downstream image applications by reconstructing RAW data from stored JPEG images. Experiments report substantial file-size reduction and demonstrate use in image retouching and HDR reconstruction.

  • 6.1. RAW data compression: JPEG-only storage enables reconstruction of corresponding RAW data for later applications.The framework is presented as a camera RAW compression approach that avoids explicitly storing RAW data.
  • 6.1. RAW data compression: The framework significantly reduces file size compared with lossy DNG.Table 2 compares compression ratio and bit per pixel between InvISP and lossy DNG.
  • 6.2. Image retouching: Recovered RAW data can be used as input for high-quality image retouching.The experiment applies a pretrained deep-learning retouching method after demosaicing and white balancing recovered and ground-truth RAW data.
  • 6.3. HDR reconstruction and tone manipulation: InvISP enables single-image HDR reconstruction and tone mapping that displays more detail than the original RGB input.The framework addresses information loss in saturated and under-exposed regions through its invertible processing design.

7. Conclusion

The paper proposes an end-to-end InvISP framework that renders visually pleasing RGB images while recovering nearly perfect RAW data. It combines invertible neural-network structure with a differentiable JPEG simulator and demonstrates applications in compression, retouching, and HDR reconstruction.

  • 7. Conclusion: InvISP jointly generates visually pleasing RGB images and recovers nearly perfect RAW data.The framework is evaluated against other frameworks and RAW data synthesis methods.
  • 7. Conclusion: A differentiable JPEG simulator improves the framework's stability to JPEG compression.The simulator is integrated into the invertible processing framework during optimization.
  • 7. Conclusion: The framework enables RAW data compression, image retouching, and HDR reconstruction tasks.These applications are demonstrated alongside comparisons with alternative frameworks and RAW synthesis methods.
Loading 2103.15061v2…