Source-linked AI summary
RawGen: Learning Camera Raw Image Generation
Dongyoung Kim, Junyong Lee, Abhijith Punnappurath, Mahmoud Afifi, Sangmin Han, Alex Levinshtein, Michael S. Brown
TL;DR
Large-scale raw data is scarce, while diffusion models primarily generate photo-finished sRGB images rather than physically consistent linear representations. RawGen bridges these domains through many-to-one inverse-ISP learning, outperforming fixed-ISP methods and supporting scalable text-driven raw synthesis for downstream low-level vision.
Problem
Large-scale raw datasets are scarce, hardware-specific, and difficult to collect, while existing diffusion models primarily generate nonlinear sRGB rather than physically meaningful linear representations.
Method
RawGen learns many-to-one latent- and pixel-level unprocessing from diverse sRGB renditions to a shared canonical linear representation, then maps outputs to target camera raw domains.
Results
RawGen consistently outperforms fixed-ISP inverse-ISP methods and supports scalable text-driven raw synthesis for downstream low-level vision tasks.
Takeaways & Limitations
RawGen provides camera-agnostic scene-referred linear synthesis and enables diverse synthetic raw data for low-level vision without additional scene capture.
Takeaways & Limitations
Accurate device-specific raw synthesis depends on factors beyond color mapping, including device-specific noise characteristics and point-spread functions.
Abstract
from arXiv · showhide
Cameras capture scene-referred linear raw images, which are processed by onboard image signal processors (ISPs) into display-referred 8-bit sRGB outputs. Although raw data is more faithful for low-level vision tasks, collecting large-scale raw datasets remains a major bottleneck, as existing datasets are limited and tied to specific camera hardware. Generative models offer a promising way to address this scarcity -- however, existing diffusion frameworks are designed to synthesize photo-finished sRGB images rather than physically consistent linear representations. This paper presents RawGen, to our knowledge the first diffusion-based framework enabling text-to-raw generation for arbitrary target cameras, alongside sRGB-to-raw inversion. RawGen leverages the generative priors of large-scale sRGB diffusion models to synthesize physically meaningful linear outputs, such as CIE XYZ or camera-specific raw representations, via specialized processing in latent and pixel spaces. To handle unknown and diverse ISP pipelines and photo-finishing effects in diffusion-model training data, we build a many-to-one inverse-ISP dataset where multiple sRGB renditions of the same scene generated using diverse ISP parameters are anchored to a common scene-referred target. Fine-tuning a conditional denoiser and specialized decoder on this dataset allows RawGen to obtain camera-centric linear reconstructions that effectively invert the rendering pipeline. We demonstrate RawGen's superior performance over traditional inverse-ISP methods that assume a fixed ISP. Furthermore, we show that augmenting training pipelines with RawGen's scalable, text-driven synthetic data can benefit downstream low-level vision tasks.
1 Introduction
RawGen addresses the scarcity and camera-specificity of raw data by adapting diffusion models from photo-finished sRGB synthesis to physically meaningful, scene-referred linear generation. Its many-to-one reconstruction framework suppresses diverse unknown ISP effects and supports both sRGB-to-linear inversion and text-driven camera-specific raw synthesis.
- Motivation: Camera sensors record scene-referred linear raw data, but lossy ISP rendering produces 8-bit sRGB outputs while large raw datasets remain costly, scarce, and camera-specific.Raw captures preserve physically meaningful radiometric information, yet collecting them at scale requires costly, labor-intensive recapture for different camera models.
- Motivation: Existing diffusion models provide semantic, prompt-controlled sRGB synthesis but operate primarily in the 8-bit sRGB domain rather than physically meaningful linear representations.Text-to-image systems enable scalable and diverse image synthesis through textual prompts.
- Motivation: Conventional inverse-ISP methods assume paired data from a fixed, known imaging pipeline, making them unreliable for diffusion outputs containing diverse, unknown, nonlinear photo-finishing effects.This domain gap prevents consistent recovery of scene-referred linear representations from generated sRGB images.
- RawGen: RawGen uses latent- and pixel-level unprocessing to map diverse sRGB inputs into a canonical scene-referred linear domain while preserving diffusion priors and promptability.The framework is camera-agnostic and suppresses heterogeneous photo-finishing effects embedded in diffusion-generated images.
- RawGen: A many-to-one objective maps multiple photo-finished sRGB observations of one scene to a shared linear reference, encouraging invariance to ISP-induced variability.RawGen supports sRGB-to-canonical-linear inversion and text-driven canonical-linear generation, with transformation into camera-specific raw domains when required.
- Results: RawGen outperforms fixed-ISP inverse methods and enables scalable text-driven raw data generation that supports downstream low-level vision tasks without additional capture effort.Applications include illuminant estimation, neural ISP learning, and denoising; the framework targets arbitrary cameras.
2 Related Works
Prior inverse-ISP and diffusion-based methods generally assume fixed or one-to-one rendering pipelines and focus on display-oriented outputs, limiting reliable recovery of physically linear camera data. RawGen addresses these gaps with many-to-one training for text-driven, camera-centric raw synthesis.
- Forward & Inverse Camera ISP Pipelines: Traditional camera ISPs combine engineered stages such as denoising, demosaicing, white balancing, color correction, and tone mapping, often producing manufacturer-specific aesthetics.Proprietary pipelines make the photo-finishing operations in internet sRGB images diverse and unknown.
- Forward & Inverse Camera ISP Pipelines: Learning-based ISP networks replace handcrafted modules with end-to-end mappings, typically converting raw images from a specific camera to display-referred outputs.These methods assume a one-to-one relationship between sRGB images and their originating ISPs.
- Forward & Inverse Camera ISP Pipelines: Existing data-driven inverse-ISP approaches largely assume one-to-one image-to-ISP correspondence, which breaks for diffusion imagery with unknown and highly variable rendering parameters.Consequently, conventional inverse-ISP strategies cannot reliably recover consistent scene-referred representations from such generative outputs.
- Diffusion Models for Low-Level Vision Tasks: Diffusion-based ISP methods typically use synthetic pairs generated by fixed one-to-one ISP logic, while related HDR methods generally output tone-mapped sRGB or display-oriented HDR formats.These approaches therefore do not reconstruct camera-centric physically linear representations such as CIE XYZ or raw from in-the-wild sRGB inputs, nor do they explore text-driven raw synthesis.
- Diffusion Models for Low-Level Vision Tasks: RawGen adopts many-to-one training that anchors diverse photo-finished observations to a shared linear target for physically consistent raw generation at scale.It synthesizes physically meaningful raw images directly from open-world text prompts without assuming a fixed imaging pipeline.
3 Method
RawGen repurposes pretrained sRGB diffusion models to generate canonical, scene-referred CIE XYZ representations that can be mapped into arbitrary camera raw spaces. It trains on many-to-one data linking diverse sRGB renderings to shared XYZ anchors, then fine-tunes the denoiser and VAE decoder for image-to-raw and text-to-raw inference.
- Canonical representation: RawGen uses CIE XYZ as a linear, device-independent canonical space that enables camera-agnostic synthesis and conversion to camera-specific raw representations.Standard color transforms relate CIE XYZ to camera-specific raw spaces without retraining.
- Many-to-one training data: The many-to-one dataset anchors multiple sRGB renditions of each scene to one illumination-neutral, scene-referred CIE XYZ image.Variants are produced by perturbing white balance, tone mapping, and contrast parameters while preserving the underlying scene content.
- Denoiser fine-tuning: Denoiser fine-tuning teaches a pretrained rectified-flow DiT to suppress ISP-induced variability and recover the shared XYZ anchor latent from randomly sampled sRGB-conditioned variants.The model predicts the rectified-flow velocity target v_gt = ϵ − z_XYZ, using LoRA adapters while freezing pretrained backbone weights.
- Decoder fine-tuning: Decoder fine-tuning retargets the pretrained VAE decoder from sRGB to linear XYZ by reconstructing anchor images from XYZ latents with an ℓ1 loss.This preserves the spatial representations learned during pretraining while adapting the decoder to linear XYZ outputs.
- Inference: At inference, both image-to-raw and text-to-raw use the same conditional DiT to generate an XYZ latent, decode it to CIE XYZ, and map it into a target camera’s linear raw space.Image-to-raw obtains conditioning from an encoded sRGB input, whereas text-to-raw uses the base model’s text-to-latent pathway.
4 Experiments
RawGen is evaluated for many-to-one inverse-ISP reconstruction, camera-specific raw synthesis, and downstream low-level vision augmentation. Across these experiments, it generalizes to unseen photo-finishing styles, aligns with real device-specific raw statistics, and improves performance across three downstream tasks.
- Many-to-One Inverse-ISP: RawGen reconstructs scene-referred XYZ from sRGB images with diverse, unknown photo-finishing, outperforming prior inverse-ISP methods and its one-to-one training variant on unseen styles.The experiments use expert-retouched MIT-Adobe FiveK images and compare against CIE XYZ Net, InvISP, Raw-Diffusion, and RawGenone-to-one.
- Many-to-One Inverse-ISP: For text-guided color variations, RawGen maps diverse renditions of the same scene to a tight, well-separated latent cluster, suppressing photo-finishing variability.The evaluation generates 100 color-graded variants per prompt and measures latent-space distance to the centroid using PCA, t-SNE, and UMAP projections.
- Device-Specific Raw Synthesis: RawGen generates camera-agnostic XYZ outputs that can be mapped to target-camera raw domains with device metadata, color correction matrices, and injected heteroscedastic Gaussian noise without retraining.This decoupled process supports device-specific raw synthesis from a shared canonical linear representation.
- Device-Specific Raw Synthesis: Neural ISPs trained on Samsung Galaxy S24 raw data produce plausible renderings from RawGen-generated S24-domain raw images without retraining, with Modular Neural ISP suppressing injected synthetic noise.The result indicates alignment between synthesized and real device-specific raw statistics.
- Downstream Applications: RawGen-based text-to-raw synthesis improves performance across all three evaluated downstream low-level vision tasks over prior synthetic raw generation approaches.The results support scalable, physically grounded raw synthesis as a way to reduce device-specific raw-data acquisition bottlenecks.
5 Conclusion and Discussion
RawGen bridges display-referred diffusion models and scene-referred linear representations through camera-agnostic raw generation. Its many-to-one objective recovers canonical XYZ representations, supports device-specific raw synthesis, and enables scalable text-to-raw data synthesis for downstream improvement.
- Contributions: RawGen is a diffusion-based framework for camera-agnostic raw generation that bridges display-referred generative models and scene-referred linear representations.The framework targets linear raw representations rather than only display-referred outputs.
- Contributions: Its many-to-one objective suppresses photo-finishing effects and recovers a canonical XYZ representation consistent across diverse sRGB variants.Multiple sRGB variants are mapped toward a shared scene-referred representation.
- Contributions: RawGen demonstrates many-to-one linear XYZ reconstruction and device-specific raw synthesis.The demonstrated outputs include both a shared linear XYZ representation and camera-specific raw data.
- Contributions: Scalable text-to-raw data synthesis enables RawGen to improve downstream low-level vision tasks.The conclusion identifies synthetic text-driven raw data as a way to augment downstream pipelines.
A Photo-Finishing Simulation Details
The photo-finishing simulator converts DNG raw data through a physically grounded ISP pipeline and generates multiple sRGB renditions per scene by independently randomizing white balance, tone mapping, and contrast. Outputs are standardized at 1024 × 1024, with 16-bit PNG XYZ anchors and 8-bit PNG sRGB variants.
- ISP Pipeline: The ISP processes DNG raw data through normalize, lens-shading correction, white balance, demosaic, XYZ, sRGB, and gamma stages, using edge-aware demosaicing and a D50 white point.Each scene is rendered multiple times with independently sampled photo-finishing parameters.
- White Balance: White-balance gains perturb red and blue channels independently by multiplicative factors sampled from U(0.7, 1.3), while green remains fixed at g = 1.0.Red and blue gains are initialized from DNG AsShotNeutral metadata.
- Tone Mapping: Tone mapping samples β from N(0.6, 0.12) clipped to [0.1, 2.0] and γ from N(0.9, 0.12) clipped to [0.5, 1.5], then re-encodes the result to sRGB.The operator is applied per channel after inverse-sRGB decoding to linear RGB.
- Contrast and Output Format: Contrast applies c sampled from U(0.7, 1.3) around a 0.5 mid-gray pivot in display-referred sRGB, while images are center-cropped and resized to 1024 × 1024.XYZ anchors use 16-bit PNG, whereas sRGB variants use 8-bit PNG.
B XYZ-to-Camera Raw Mapping
RawGen renders camera-agnostic CIE XYZ outputs into target-camera linear raw-RGB using DNG calibration metadata and CCT-dependent matrix interpolation. It can additionally simulate camera noise with a calibrated heteroscedastic model controlled by a global noise-strength factor.
- Camera-specific mapping: DNG color-matrix pairs under two reference illuminants are interpolated by correlated color temperature to render XYZ images into camera-specific linear raw-RGB.This preserves a camera-agnostic XYZ generation stage while enabling target-camera rendering.
- Camera-specific mapping: The mapping inverts the forward model to obtain white-balanced camera RGB, converts the XYZ illuminant through the color matrix, normalizes by green, and applies the relative gain.The resulting clipped image is treated as the target camera’s linear raw-RGB output.
- Noise simulation: Generated raw images are exported as normalized 16-bit images for dataset generation unless otherwise noted.The pipeline optionally adds sensor noise after raw conversion.
- Noise simulation: Noise variance is modeled per channel as a signal-dependent and signal-independent term scaled by global noise strength factor s.The shot- and read-noise parameters are obtained from camera noise calibration or the camera’s noise profile.
- Noise simulation: Zero-mean Gaussian noise sampled from the modeled variance is added to the raw image, with examples spanning different CCTs and noise-strength values for the Samsung Galaxy S24 main camera.Figure 7 illustrates these variations after gamma correction for visualization.
C Downstream Task Setup Details
RawGen is evaluated as scalable synthetic-data supervision for illumination estimation, neural ISP, and raw-domain denoising. Across tasks, architectures, protocols, and testing conditions are controlled to isolate the impact of generated raw data.
- Illumination Estimation: Illumination estimation uses 3K text-prompt-generated synthetic raw images to increase scene diversity on the nine-camera NUS dataset.The Graphics2RAW architecture and optimization settings remain unchanged, with performance measured by angular error.
- Neural ISP: Neural ISP training pairs 3K RawGen-generated synthetic raw images with rendered sRGB targets for supervision.The nighttime dataset and Graphics2RAW testing configuration are retained, testing occurs on real raw inputs, and architecture, losses, and metrics remain fixed.
- Image Denoising: Raw-domain denoising trains on 3K clean linear images synthesized from text prompts and noisy counterparts generated with the baseline’s heteroscedastic Gaussian noise model.The unchanged denoiser architecture and training configuration are evaluated exclusively on real noisy captures at ISOs 1600 and 3200.
D Additional Results
Additional results show RawGen’s qualitative CIE XYZ reconstruction, sRGB-to-raw synthesis, text-to-raw generation across target cameras, and compatibility with pretrained neural ISPs. Rendered outputs remain visually plausible and closely aligned with real Samsung Galaxy S24 raw data despite distribution-shift limitations.
- CIE XYZ Reconstruction: Additional CIE XYZ comparisons evaluate RawGen against Raw-Diffusion and CIE XYZ Net across sRGB renderings with Expert A–E preferences.The figures include corresponding ground-truth CIE XYZ images.
- sRGB-to-Raw Synthesis: RawGen reconstructs raw images from input sRGB images and generates additional raw images of the same semantic scenes using InstructBLIP-2 descriptions.The generated outputs include raw reconstructions and text-driven samples.
- sRGB-to-Raw Synthesis: RawGen maps generated CIE XYZ images into arbitrary target-camera raw domains, with text-to-raw examples spanning Canon 1Ds MkIII, Olympus EPL6, Nikon D5200, and Samsung NX2000.Additional examples are generated from text prompts for three target cameras and camera-specific demonstrations cover four DSLR models.
- Compatibility on Pre-Trained ISP: Pretrained Lite ISP, Invertible ISP, and Modular Neural ISP models render RawGen-generated S24 raw images with visually plausible outputs.Synthetic sensor noise is added before rendering, and the results indicate close alignment with real Samsung Galaxy S24 training data despite raw-distribution generalization limitations.