Source-linked AI summary
Compression Artifacts Reduction by a Deep Convolutional Network
Chao Dong, Yubin Deng, Chen Change Loy, Xiaoou Tang
TL;DR
Lossy compression produces coupled artifacts that degrade image quality and downstream low-level vision. The paper introduces AR-CNN, a compact four-layer convolutional network with feature enhancement, and studies easy-to-hard feature transfer for deeper models. It reports effectiveness on benchmarks and real-world use cases, while also showing utility as preprocessing for other low-level vision routines.
Problem
Lossy compression introduces blocking, ringing, and blurring that reduce perceptual quality and adversely affect low-level image processing routines.
Method
AR-CNN uses a four-layer convolutional network with feature enhancement and jointly optimized feature extraction, enhancement, mapping, and reconstruction, with feature transfer for deeper models.
Results
AR-CNN shows effectiveness over state-of-the-art methods on benchmark datasets and real-world use cases, including Twitter, and facilitates super-resolution and contrast enhancement.
Takeaways & Limitations
Compression-artifact reduction can serve as preprocessing for other low-level vision routines that take compressed images as input.
Takeaways & Limitations
The paper focuses on restoration of the luminance channel in YCrCb space.
Abstract
from arXiv · showhide
Lossy compression introduces complex compression artifacts, particularly the blocking artifacts, ringing effects and blurring. Existing algorithms either focus on removing blocking artifacts and produce blurred output, or restores sharpened images that are accompanied with ringing effects. Inspired by the deep convolutional networks (DCN) on super-resolution, we formulate a compact and efficient network for seamless attenuation of different compression artifacts. We also demonstrate that a deeper model can be effectively trained with the features learned in a shallow network. Following a similar "easy to hard" idea, we systematically investigate several practical transfer settings and show the effectiveness of transfer learning in low-level vision problems. Our method shows superior performance than the state-of-the-arts both on the benchmark datasets and the real-world use case (i.e. Twitter). In addition, we show that our method can be applied as pre-processing to facilitate other low-level vision routines when they take compressed images as input.
1. Introduction
Lossy compression creates multiple coupled artifacts that degrade visual quality and downstream image processing. The paper proposes AR-CNN and feature-transfer strategies to reduce these artifacts effectively, including in real-world settings.
- Motivation: Lossy compression introduces blocking, ringing, and blurring that reduce perceptual quality and impair low-level image processing routines.These routines include contrast enhancement, super-resolution, and edge detection.
- Motivation: Existing methods often trade artifact removal for blurred output or sharpened images with noisy edges and unnatural smooth regions.Deblocking-oriented approaches remove blocking and ringing but tend to blur, while restoration-oriented methods can introduce other visual defects.
- Motivation: Directly applying a three-layer SRCNN is poorly suited to compressed images because coupled artifacts produce noisy features and undesirable reconstruction patterns.The paper identifies difficulties in handling blocking artifacts and smooth regions.
- Approach: AR-CNN adds feature-enhancement layers after the first layer and jointly optimizes feature extraction, enhancement, mapping, and reconstruction.The four-layer network is designed to clean noisy features before reconstruction.
- Approach: Feature transfer from a shallow network enables training a deeper model, following an easy-to-hard strategy across compression settings.The paper also examines transfer from a standard compression model to a harder real use case.
- Results: Experiments report effectiveness over state-of-the-art methods on benchmark datasets and real use cases, while also supporting downstream low-level vision routines.The reported applications include super-resolution and contrast enhancement on compressed inputs.
2. Related work
Prior work separates compression-artifact restoration into deblocking-oriented and restoration-oriented methods, while deep-network transfer learning has mainly been studied in high-level vision. AR-CNN adapts deep convolutional modeling to the coupled artifacts of compression restoration.
- Prior methods: Existing algorithms are classified as deblocking-oriented or restoration-oriented methods.Deblocking methods target artifact removal, whereas restoration methods treat compression as distortion.
- Prior methods: SA-DCT is widely acknowledged as a state-of-the-art deblocking method but cannot reproduce sharp edges and tends to oversmooth texture regions.Its limitations motivate methods that address restoration quality beyond block-boundary filtering.
- Deep models: SRCNN formulates independent sparse-coding restoration steps as convolutional layers optimized in a unified network, demonstrating deep models' potential for low-level vision.Compression restoration differs from super-resolution because it contains multiple artifact types.
- Deep models: Directly applying SRCNN to compression restoration produces undesirable noisy patterns in reconstructed images.The paper attributes this to the need for deeper understanding of compression-specific artifacts.
- Transfer learning: Transfer learning has been systematically investigated in high-level vision, but not in low-level vision tasks.The paper positions its transfer experiments as addressing this gap.
3. Methodology
The paper builds AR-CNN by adding feature enhancement to SRCNN, then uses easy-to-hard transfer learning to train deeper or harder compression-restoration models.
- SRCNN foundation: SRCNN learns an end-to-end mapping from an input image through convolutional feature extraction, nonlinear mapping, and reconstruction.Its convolutional layers use ReLU responses and no pooling or fully connected layers, preserving output size.
- Motivation: Compression artifacts produce noisy and ambiguous extracted features, while blocking artifacts in flat regions can be mistaken for normal edges.The feature enhancement layer is introduced to address these coupled artifact effects.
- AR-CNN architecture: AR-CNN inserts a feature enhancement layer between feature extraction and mapping to transform noisy features into a cleaner feature space.The four-layer network comprises feature extraction, feature enhancement, mapping, and reconstruction.
- AR-CNN architecture: Unlike simply adding nonlinear mapping layers, AR-CNN improves mapping accuracy by enhancing extracted low-level features.The distinction separates AR-CNN from a deeper SRCNN architecture.
- Transfer learning: The study investigates easy-to-hard transfer because deeper low-level vision networks can be difficult to converge and are sensitive to initialization.The base task is a four-layer AR-CNN, whose features initialize target networks for deeper or harder tasks.
- Transfer learning: Transfer settings include shallow-to-deeper transfer, high-quality-to-low-quality transfer, and standard-compression-to-real-use-case transfer.The target settings reuse selected base-network features while randomly initializing remaining layers and training toward the target task.
4. Experiments
The experiments train on BSDS500 JPEG-compressed images and evaluate restoration using PSNR, SSIM, and PSNR-B on LIVE1.
- Data and setup: 400 BSDS500 images are used for training, with JPEG quality settings q = 20 and q = 10 and luminance-channel restoration in YCrCb space.The validation set contains 100 images, while the disjoint training and test sets each contain 200 images used for training.
- Evaluation: Table 1 reports average PSNR (dB), SSIM, and PSNR-B (dB) results on the LIVE1 dataset.The supplied table passage identifies the reported metrics and dataset but does not provide the numerical table entries.
- Data and setup: Training images are decomposed into 32 × 32 patches, producing 537,600 training samples from the 400 images.AR-CNN produces a 20 × 20 output from each 32 × 32 input, so loss is computed against the central ground-truth region.
4.1. Comparison with the State-of-the-Arts
AR-CNN is evaluated against state-of-the-art deblocking and restoration methods, SRCNN variants, and RTF variants using quantitative and qualitative image-quality comparisons.
- Comparison with SA-DCT: AR-CNN is evaluated on LIVE1 using PSNR, SSIM, and PSNR-B at JPEG quality q = 10.PSNR-B is designed for blocky and deblocked images and is more sensitive to blocking artifacts than SSIM.
- Comparison with SA-DCT: AR-CNN outperforms SA-DCT on all JPEG qualities and evaluation metrics by a large margin.The gains are larger on PSNR-B, indicating fewer blocking artifacts.
- Comparison with SA-DCT: AR-CNN processes a 256 × 256 image in 0.5 second, compared with 3.4 seconds for SA-DCT.Both implementations use C++ on a PC with an Intel I3 CPU and 16GB RAM.
- Comparison with SRCNN: The original and deeper SRCNN networks are inferior to AR-CNN on all evaluation metrics and produce noisy edges and unnatural smooth regions.AR-CNN also reaches higher PSNR from the beginning of training.
- Comparison with RTF: AR-CNN outperforms plain RTF and achieves better performance than RTF combined with SA-DCT, especially under PSNR-B.The comparison uses half-sized BSDS500 images and training conditions matched to the RTF evaluation.
4.2. Experiments on Easy-Hard Transfer
The experiments test transfer learning from shallower, higher-quality, and standard-compression models to harder compression-artifact reduction settings, including Twitter images.
- Transfer shallow to deeper model: The easy-hard transfer experiments compare different initialization settings using a four-layer AR-CNN base network trained on BSDS500 at q = 10.The deeper model is a five-layer AR-CNN with structure 9-7-3-1-5.
- Transfer shallow to deeper model: Features transferred from a four-layer network enable successful training of a five-layer network.Direct training with conventional initialization did not converge reliably, while transferred training converged faster and performed better than He et al.’s method.
- Transfer high to low quality: The “transfer 1 layer” setting reaches an average PSNR of 27.77dB in 1.54 × 10^8 backprops, roughly half the backprops required by “base-q10”.It also slightly outperforms “base-q10” throughout training.
- Transfer standard to real use case: Transferred features converge faster than training from scratch for the Twitter restoration task.“Transfer q10” reaches 25.1dB in 6 × 10^7 backprops, compared with 10 × 10^7 for “base-Twitter”.
- Transfer standard to real use case: Features learned under standard compression schemes transfer to real-world Twitter compression and yield higher PSNR than training from scratch.Both transferred and baseline networks produce satisfactory quality improvements over the compressed images.
5. Application
AR-CNN is presented as a preprocessing step for low-level vision routines affected by JPEG compression, with benefits visible after subsequent processing. It also outperforms SA-DCT in the shown application results and is evaluated on blocking, ringing, and Twitter-compressed images.
- 5. Application: JPEG artifacts distort super-resolution and contrast-enhancement outputs, but AR-CNN largely eliminates these effects before processing.The application compares JPEG, SA-DCT-deblocked, and AR-CNN-restored inputs.
- 5. Application: The figures compare AR-CNN with SA-DCT for blocking-artifact removal and with SRCNN for ringing-effect removal.These comparisons address distinct compression-artifact types.
- 5. Application: The application section includes restoration results on Twitter-compressed images, where the original 8MP image is only partially displayed for visualization.The Twitter case is presented as a real-world restoration example.
- 5. Application: AR-CNN is positioned as preprocessing that facilitates other low-level routines when their inputs are JPEG images.This role is stated directly in the figure description.
- 5. Application: AR-CNN achieves much better results than SA-DCT in the illustrated low-level vision applications.The differences between the methods become more evident after super-resolution and contrast enhancement.
6. Conclusion
The paper concludes that AR-CNN is an effective four-layer convolutional network for varied compression artifacts and that easy-to-hard transfer settings support deeper or better low-level vision models.
- 6. Conclusion: AR-CNN is a four-layer convolutional network designed to handle various compression artifacts effectively.The conclusion frames the architecture as requiring detailed understanding of the compression process.
- 6. Conclusion: Easy-to-hard transfer settings can facilitate training a deeper or better network for low-level vision problems.The paper reports systematic investigation of several such settings.
- 6. Conclusion: Larger filter sizes also help improve performance, although their further investigation is left to future work.This is identified as an additional conclusion related to network design.