Source-linked AI summary
Rethinking Learning-based Demosaicing, Denoising, and Super-Resolution Pipeline
Guocheng Qian, Yuanhao Wang, Jinjin Gu, Chao Dong, Wolfgang Heidrich, Bernard Ghanem, Jimmy S. Ren
TL;DR
Image capture combines incomplete color sampling, noise, and limited resolution, yet the effect of task order on this mixture problem remains unclear. The paper evaluates pipelines in sequential and joint solutions and proposes a way to impose one in an end-to-end network. DN→SR→DM consistently improves sequential performance, while pipeline supervision yields only marginal but consistent gains in joint networks.
Problem
The effect of pipeline order on the combined DN, DM, and SR problem is unclear, despite the problem being common in practical imaging.
Source: §1 · p. 1Method
The paper studies pipelines in sequential and joint solutions and uses intermediate supervision to make an end-to-end network follow a chosen pipeline.
Source: §1 · pp. 1–2, §3.2 · p. 3Results
DN→SR→DM consistently improves sequential image quality across architectures, datasets, and SR factors, while pipeline supervision brings marginal but consistent improvements in end-to-end networks.
Source: §1 · pp. 1–2Takeaways & Limitations
Pipeline choice matters in sequential solutions, but investigating task order is not very necessary for end-to-end networks given the insignificant improvement.
Source: §8 · p. 10PixelShift200 contains only static objects and 200 unique samples, and the study considers only single-frame image processing.
Source: §9 · p. 10
Abstract
from arXiv · showhide
Imaging is usually a mixture problem of incomplete color sampling, noise degradation, and limited resolution. This mixture problem is typically solved by a sequential solution that applies demosaicing (DM), denoising (DN), and super-resolution (SR) sequentially in a fixed and predefined pipeline (execution order of tasks), DM$\to$DN$\to$SR. The most recent work on image processing focuses on developing more sophisticated architectures to achieve higher image quality. Little attention has been paid to the design of the pipeline, and it is still not clear how significant the pipeline is to image quality. In this work, we comprehensively study the effects of pipelines on the mixture problem of learning-based DN, DM, and SR, in both sequential and joint solutions. On the one hand, in sequential solutions, we find that the pipeline has a non-trivial effect on the resulted image quality. Our suggested pipeline DN$\to$SR$\to$DM yields consistently better performance than other sequential pipelines in various experimental settings and benchmarks. On the other hand, in joint solutions, we propose an end-to-end Trinity Pixel Enhancement NETwork (TENet) that achieves state-of-the-art performance for the mixture problem. We further present a novel and simple method that can integrate a certain pipeline into a given end-to-end network by providing intermediate supervision using a detachable head. Extensive experiments show that an end-to-end network with the proposed pipeline can attain only a consistent but insignificant improvement. Our work indicates that the investigation of pipelines is applicable in sequential solutions, but is not very necessary in end-to-end networks. \RR{Code, models, and our contributed PixelShift200 dataset are available at \url{https://github.com/guochengqian/TENet}
1 INTRODUCTION
The paper frames denoising, demosaicing, and super-resolution as a common but underexplored mixture problem, with pipeline order still uncertain. It studies sequential and joint solutions, finding a consistently stronger sequential pipeline but only marginal gains from adding pipeline supervision to end-to-end networks.
1 INTRODUCTION
- Pipeline order for the combined DN, DM, and SR problem remains underexplored, and its effect on performance is unclear.Earlier sequential solutions used DM→DN→SR, while other work moved DN and SR before demosaicing,,.Source: §1 · p. 1
- The proposed DN→SR→DM pipeline consistently improves sequential image quality across architectures, datasets, and SR factors.The paper reports that it alleviates problems from task interactions, including magnifying demosaicing artifacts through later super-resolution.Source: §1 · p. 1
- End-to-end networks show marginal but consistent gains when intermediate supervision enforces a pipeline, making pipeline investigation less necessary in joint solutions.The approach is tested on TENet++ and other architectures,,.Source: §1 · p. 2
- The paper studies pipeline effects in both sequential and joint solutions for learning-based DN, DM, and SR.It identifies the mixture problem as practically common and more complicated than any single task.Source: §1 · pp. 1–2
- The paper introduces TENet++ for joint DN, DM, and SR, and contributes PixelShift200, a full-color-sampled dataset for raw image processing.PixelShift200 contains red, green, and blue channels without color interpolation.Source: §1 · p. 2
2 RELATED WORK
Related work covers learning-based methods for demosaicing, denoising, and super-resolution, then identifies limited attention to the order of these tasks when combined. The paper takes a holistic view of that pipeline question, particularly for end-to-end methods.
Demosaicing
- Deep-learning demosaicing methods improved on earlier model-based and data-driven approaches, with later work exploring different architectures,,.Model-based filters and priors remain common in camera systems and software, including DCRaw.Source: §2 · p. 2
Denoising
- Learning-based denoisers use CNNs, but methods trained on synthetic benchmarks can generalize poorly to real shots under oversimplified additive white Gaussian noise assumptions [5],,,.Raw-image denoising with Poisson-Gaussian noise improved results on the real-world DND dataset,,.Source: §2 · p. 2
Super-resolution
- Learning-based SR methods advanced beyond example-based methods,, but most focus on color images, with relatively few addressing raw-image SR,,.Source: §2 · p. 2
ISP and Mixture Problem
- Most image-processing pipelines perform DN, DM, and SR sequentially in the predefined DM→DN→SR order, while prior alternatives and end-to-end methods leave task order underexplored,, [10],,,,.The paper revisits the mixture problem from a holistic perspective, focusing on pipeline execution order.Source: §2 · p. 2
3 METHODOLOGY
The methodology studies DN→SR→DM for sequential processing and introduces an end-to-end network that represents joint DN and SR before demosaicing. It uses intermediate supervision to impose that pipeline and develops TENet++ to avoid a bottleneck in the naive design.
3.1 A New Pipeline for DN, DM and SR
- The sequential pipeline denoises the noisy raw mosaic, super-resolves the mosaic, then demosaics it into a high-resolution color image.The authors motivate denoising first because raw-image noise is better modeled and processing beforehand can complicate denoising; super-resolution before demosaicing can reduce artifacts.Source: §3.1 · pp. 2–3
3.2 Inserting Our Pipeline into An End-to-end Network
- Intermediate supervision makes an end-to-end network follow DN+SR→DM, with joint DN and SR in FM followed by demosaicing in CM.The intermediate mosaic output and loss focus FM on joint DN and SR, while the final-output loss controls fidelity.Source: §3.2 · pp. 3–4
- DN+SR→DM is the best overall joint pipeline, while its improvement is marginal; additional denoising supervision does not help.The authors identify processing DN and SR in mosaic space, rather than RGB space, as the pipeline’s central idea.Source: §3.2 · p. 4
3.3 Trinity of Pixel Enhancement Network
- TENet++ uses an attachable convolutional branch for intermediate supervision, avoiding TENet’s channel bottleneck from latent features to the four-channel raw image.The branch produces the intermediate raw image during training and can be removed for inference.Source: §3.3 · pp. 4–5
4 PIXELSHIFT200 DATASET
PixelShift200 provides real-world, full-color-sampled images intended to address artifacts and information loss in synthesized demosaicing datasets. The dataset also supports better raw image-processing quality than training on incompletely color-sampled data.
4.1 Motivation of PixelShift200
- Synthesized demosaicing datasets can inherit ISP interpolation artifacts, teach an average camera-ISP demosaicing algorithm, and lose information through 8-bit depth.The authors argue that real-world, high-resolution, uncompressed images with full-color sampling are needed.Source: §4.1 · p. 5
- PixelShift200 contains 200 4K images with known 14-bit red, green, and blue values per pixel, collected without demosaicing.Four captures of the same scene shift the sensor by one pixel horizontally or vertically to obtain full color sampling.Source: §4.1 · p. 5
- Training raw image-processing networks on PixelShift200 produces better image quality than training on incompletely color-sampled datasets such as DIV2K.PixelShift200 also shows less aliasing and fewer moirés than Camera Raw interpolation of a mosaic sample.Source: §4.1 · p. 5
4.2 PixelShift200 Collection Procedure
- The collection uses 200 4K training images and 20 1K test images selected to cover a wide range of scenes.Training images were cropped into 9,444 overlapping 512 × 512 patches.Source: §4.2 · p. 5
5 EXPERIMENTS
The experiments compare pipelines under shared PixelShift200 conditions and find that task order matters most in sequential processing. The proposed DN→SR→DM pipeline leads among sequential solutions, while DN+SR→DM offers smaller gains in joint solutions.
5.1 Experimental Setup
- Evaluation uses PSNR, SSIM, and FreqGain to measure fidelity, structure similarity, and fine-grained artifacts, respectively.FreqGain is a scalar adaptation of a moiré-detection metric.Source: §5.1 · p. 5
- The comparison implements candidate pipelines with common modules and training data, then validates the proposed pipelines across datasets and module designs.The tested setup includes single-task, partially joint, and fully joint models; the reported common comparison uses PixelShift200, Gaussian-Poisson noise, and ×2 SR.Source: §5.1 · p. 6
5.2 Pipeline Comparison Experiments
- 3.10 dB higher PSNR than DM→DN→SR; DN→SR→DM is the strongest sequential pipeline in the PixelShift200 comparison.With DN first, performing SR before DM still raises PSNR by 0.28 dB; the table covers Gaussian-Poisson noise and ×2 SR.Source: Table 1 · p. 6, §5.2 · p. 7
- DN+SR→DM improves PSNR by 0.39 dB over other partially joint pipelines and by 0.65 dB over the DN+DM+SR end-to-end counterpart.Joint-solution gains are less than 1 dB and are described as less significant than sequential gains.Source: Table 1 · p. 6, §5.2 · pp. 7–8
- Qualitative comparisons show fewer color artifacts and less noise for DN→SR→DM, while the proposed joint pipeline suffers less moiré.The comparisons use a low-resolution noisy mosaiced input; the table comparison uses shared models and ×2 SR with Gaussian-Poisson noise.Source: §5.1 · p. 6, Figure 4 · p. 7, §5.2 · p. 8
6 ABLATION STUDY
The ablations test whether pipeline gains persist across architectures, datasets, SR factors, and noise models. Gains remain consistent in sequential solutions, while joint-solution improvements are generally small; PixelShift200 training also reduces artifacts in real-shot comparisons.
6.1 Ablate Proposed Pipeline
- DN+SR→DM consistently improves joint performance across TENet++ and alternative NLSA-, JDSR-, and JDnDmSR-based architectures.TENet++ also outperforms the tested SOTA module and networks when using the same pipeline.Source: §6.1 · p. 8
- Across DIV2K, Urban100, and CBSD68, the proposed pipelines improve performance in sequential and joint solutions; joint PSNR gains remain below 0.2 dB.These evaluations use DIV2K training data, Gaussian-Poisson noise, and ×2 SR.Source: §6.1 · p. 8, Table 3 · p. 10
- With ×4 super-resolution, the proposed pipeline outperforms alternatives in both sequential and partially joint solutions.Source: §6.1 · p. 8, Table 4 · p. 10
- Under additive Gaussian noise with sigma 10, DN+SR→DM only marginally outperforms DN+DM+SR in joint solutions.Models were trained on DF2K and evaluated on DIV2K test, Urban100, and CBSD68; TENet++ is described as closest to ground truth qualitatively.Source: §6.1 · p. 8, Table 5 · p. 10
6.2 Ablate proposed Dataset PixelShift200
- On real-shot images, TENet++ trained on PixelShift200 shows less moiré and fewer color artifacts than the same model trained on DIV2K.The authors attribute the improvement to PixelShift200’s full-color sampling and natural-image distribution characteristics.Source: §6.2 · pp. 8, 10, Figure 6 · p. 9
7 REAL-SHOT EXPERIMENTS
The real-shot comparisons evaluate TENet++ against raw-processing software and other methods, with fewer artifacts reported than JDnDmSR and stronger high-frequency texture reconstruction than JDSR.
7 REAL-SHOT EXPERIMENTS
- The comparison includes an iPhone X raw image processed with TENet++, DCRaw, and Camera Raw.DCRaw and Camera Raw outputs are demosaiced and then superresolved using a 6-RRDB model.Source: §7 · p. 10
- TENet++ reconstructs high-frequency texture more successfully than JDSR in real-shot comparisons.The compared models use the same DN+SR→DM pipeline and are trained on DIV2K.Source: §7 · p. 10
- TENet++ produces fewer moiré and color-aliasing artifacts than JDnDmSR in real-shot comparisons.The examples cited are scarf texture and steel railing, respectively.Source: §7 · p. 10
8 CONCLUSION
The conclusion finds pipeline choice consequential for sequential DN, DM, and SR but marginal for joint solutions, and presents intermediate supervision and PixelShift200 as contributions.
8 CONCLUSION
- Pipeline effects are significant for sequential DN, DM, and SR, but marginal for joint solutions.The authors conclude that shuffling task order is not very necessary for end-to-end networks.Source: §8 · p. 10
- Intermediate supervision is presented as a way to enforce a chosen pipeline in an end-to-end network.Source: §8 · p. 10
- PixelShift200 is contributed as a full-color sampled dataset for training and evaluating raw image-processing tasks.Source: §8 · p. 10
9 LIMITATION AND FUTURE WORK
The authors identify limits in PixelShift200's size and scene content, and restrict this study to single-frame processing while pointing to multi-frame work as a future direction.
9 LIMITATION AND FUTURE WORK
- PixelShift200 contains only static objects and 200 unique samples, and the authors say more samples would benefit the community.Source: §9 · p. 10
- The study considers only single-frame processing, leaving multi-frame DN, DM, and SR as a proposed future direction.The authors describe multi-frame processing as promising and under-explored.Source: §9 · p. 10