Source-linked AI summary
Deep Generative Adversarial Compression Artifact Removal
Leonardo Galteri, Lorenzo Seidenari, Marco Bertini, Alberto Del Bimbo
TL;DR
Compression artifacts remove details and add noise, harming perceived image quality and potentially object detection. The paper uses fully convolutional residual CNNs and a conditional GAN for post-processing decompressed images, with SSIM providing a baseline. The GAN produces more photorealistic details and improves object-detector performance on highly compressed images relative to MSE- or SSIM-trained networks.
Problem
Lossy compression degrades images through lost details and added artifacts, which can reduce human-perceived quality and computer-vision performance.
Method
The paper applies fully convolutional residual CNNs to decompressed images, using SSIM as a baseline and conditional adversarial training with sub-patch discrimination.
Results
The GAN generates finer, more photorealistic details and performs better than MSE- or SSIM-trained networks for object detection on highly compressed images.
Takeaways & Limitations
GAN-based artifact removal can serve as a pre-processing step for object detection and can improve reconstructed-image fidelity for humans and machines.
Takeaways & Limitations
The method converts all images to YCrCb for processing and omits the compression quality-factor parameter from reconstruction.
Abstract
from arXiv · showhide
Compression artifacts arise in images whenever a lossy compression algorithm is applied. These artifacts eliminate details present in the original image, or add noise and small structures; because of these effects they make images less pleasant for the human eye, and may also lead to decreased performance of computer vision algorithms such as object detectors. To eliminate such artifacts, when decompressing an image, it is required to recover the original image from a disturbed version. To this end, we present a feed-forward fully convolutional residual network model trained using a generative adversarial framework. To provide a baseline, we show that our model can be also trained optimizing the Structural Similarity (SSIM), which is a better loss with respect to the simpler Mean Squared Error (MSE). Our GAN is able to produce images with more photorealistic details than MSE or SSIM based networks. Moreover we show that our approach can be used as a pre-processing step for object detection in case images are degraded by compression to a point that state-of-the art detectors fail. In this task, our GAN method obtains better performance than MSE or SSIM trained networks.
1. Introduction
Compression artifacts trade smaller files and lower transmission demands for degraded visual quality and potentially weaker computer-vision performance. The paper develops CNN- and GAN-based post-processing methods, finding SSIM-based restoration effective while GAN reconstruction better preserves perceptual detail and detection performance.
- Motivation: Lossy compression reduces storage and transmission requirements but introduces blocking, posterizing, contouring, blurring, and ringing artifacts.Degradation generally becomes stronger as the compression factor increases, while low compression may be impractical for web, streaming, drone, surveillance, and sensor-network applications.
- Motivation: Prior artifact-removal methods span DCT-coefficient optimization, image priors, patch models, and relatively few CNN approaches.The paper positions CNN-based artifact removal as comparatively underexplored.
- Approach: The proposed CNN post-processing method targets decompressed images and can handle JPEG, H.264/AVC intra-frame coding, and H.265/HEVC.This makes the approach applicable across multiple compression algorithms.
- Approach: SSIM is used as an alternative to MSE because PSNR and MSE correlate imperfectly with perceived image quality.The paper reports state-of-the-art results on several benchmarks for deep CNNs trained by directly optimizing SSIM.
- Results: GAN artifact removal improves SSIM and object-detector performance on highly compressed images while producing reconstructions judged more faithful by human viewers.The reported perceptual advantage is especially strong for the GAN method.
2. Related Work
The paper reviews processing-based and learning-based artifact removal, then proposes residual CNN and conditional GAN designs to address the smoothing and detail-loss limitations of MSE- and SSIM-trained restoration.
- Prior methods: Existing compression-artifact removal methods are predominantly processing-based, while only a smaller set are learning-based.Processing-based approaches include DCT-domain methods and regularized restoration formulations.
- Prior methods: Learning-based methods train DCNNs to transform degraded images into restored versions using original images as targets.Their learned image manifold can incorporate semantics rather than relying only on DCT coefficients or other image statistics.
- Prior methods: Related DCNN restoration work includes AR-CNN for artifact reduction and feed-forward CNNs for image restoration, alongside applications to super-resolution, style transfer, and denoising.These neighboring tasks provide architectural and methodological precedents for image transformation networks.
- Contributions: The paper introduces a fully convolutional residual generative network trained with MSE and SSIM alternatives, noting that both can produce overly smoothed images.The network restores images at arbitrary resolution and SSIM improves perceptual results relative to MSE.
- Contributions: A conditional GAN is proposed to model sharper reconstructions, with sub-patch discrimination intended to reduce high-frequency mosquito noise.The paper presents this as its first GAN application to compression-artifact recovery.
3. Methodology
The methodology reconstructs uncompressed images from compressed inputs with a convolutional generator, while aiming for a compression-parameter-independent post-processing function.
- Images are converted to YCrCb before processing because compression algorithms separate luminance and chrominance and subsample chrominance.Luminance-only metrics use grayscale images, whereas full processing uses all three YCrCb channels.
- The task is to reconstruct IHQ from compressed ILQ by learning an approximate inverse G ≈ A−1.The compression quality factor is omitted from the reconstruction algorithm so the function remains independent of compression parameters.
- Fully convolutional networks perform local nonlinear transformations and can process images of any size, supporting efficient training on localized artifacts.The approach treats artifact removal as an image-transformation problem similar to super-resolution and style transfer.
- The proposed generator can be trained with direct supervision or coupled to a discriminator in a generative adversarial framework.The network is optimized using a custom artifact-removal loss over N training images.
3.1. Generative Network
The generator is a deep residual fully convolutional network built from convolutional layers and LeakyReLU nonlinearities, with downsampling and upsampling to reduce computation.
- The generator uses residual blocks containing convolutional layers and LeakyReLU nonlinearities.Its architecture is shown in Figure 2.
- Convolutions use 3 × 3 kernels and 64 feature maps, with stride-2 downsampling followed by nearest-neighbor upsampling.Padding of 1 pixel maintains image dimensions across the residual blocks.
- A convolution after upsampling reduces artifacts introduced by the upsampling process.The final one-feature-map convolution uses tanh to constrain reconstructed values to [−1, 1].
3.2. Loss Functions for Direct Supervision
Direct supervision trains the generator by comparing reconstructed images with original images using pixel-wise MSE or structurally oriented SSIM objectives.
- Direct supervision computes the loss from the reconstructed image IRQ and the target original image IHQ.Weights are updated using classical backpropagation.
- MSE serves as the baseline loss for image reconstruction and restoration.It is effective at recovering low-frequency details but suppresses many high-frequency details.
- SSIM provides an alternative objective because MSE and PSNR have been shown to be inconsistent with human visual perception of image similarity.The network is optimized for structural similarity between uncompressed and reconstructed images using a differentiable SSIM loss.
3.3. Generative Adversarial Artifact Removal
The adversarial method couples the generator with a discriminator and adds perceptual feature-space comparison to encourage realistic reconstructed patches and details.
- Adversarial training couples the generative network with a discriminator to support photo-realistic image generation.The generator architecture from Section 3.1 is reused in this framework.
- The discriminator distinguishes reconstructed patches IRQ from real patches IHQ conditioned on the compressed input ILQ.Positive and negative examples concatenate each candidate with ILQ along the channel dimension.
- Discrimination operates on 16 × 16 sub-patches because compression artifacts are typically created within image patches.The paper reports a beneficial effect of this design on reducing mosquito noise.
- Perceptual loss compares feature representations rather than pixels by projecting IHQ and IRQ through a differentiable function φ before measuring Euclidean distance.The VGG19 feature maps are extracted from the second convolutional layer before the last max-pooling layer.
- The adversarial loss rewards generator outputs that fool the discriminator.The discriminator parameters are learned with a separate discriminator loss.
4. Experiments
Experiments compare supervised MSE/SSIM and adversarial restoration across image-quality, perceptual, and object-detection evaluations. The GAN produces more convincing details and improves detection most strongly, including under aggressive compression and across codecs.
- Experimental setup: The networks are trained on random augmented MSCOCO patches compressed with MATLAB JPEG at multiple quality factors for 70,000 iterations.Training uses 16 random 128×128 patches per mini-batch, Adam with momentum 0.9, and learning rate 10^-4.
- Experimental setup: Evaluations cover PSNR, PSNR-B, and SSIM on BSD500 and LIVE1, subjective quality on BSD500, and object detection on PASCAL VOC2007.The study compares JPEG restoration methods and evaluates detection across reconstruction algorithms.
- Comparison with state of the art: The non-adversarial residual networks outperform other approaches across quality measures, with MSE improving PSNR and PSNR-B and SSIM training improving SSIM.These comparisons include JPEG compression, SA-DCT, AR-CNN, and Svoboda et al.'s method.
- Comparison with state of the art: The GAN has lower standard quality-index performance but produces more perceptually convincing images with realistic textures rather than the smoother details of MSE/SSIM models.Human viewers find the GAN reconstruction more convincing in the reported qualitative comparison.
- Object detection: The GAN improves object-detector mAP by 7.4 points on compressed images, compared with 2.1, 2.4, and 2.5 points for AR-CNN, MSE, and SSIM generators.The compressed-image result remains 6.8 points below the original-dataset upper bound.
- Object detection: GAN gains are especially large for cat (+16.6), cow (+12.5), dog (+18.6), and sheep (+14.3), where articulated structure and texture provide informative cues.MSE and SSIM generators deteriorate performance in these classes, while the sub-patch discriminator reaches .623 mAP versus .605 for the full-patch discriminator.
- Object detection: GAN restoration outperforms other restoration algorithms across compression quality factors, including aggressive QF=10, although the gap narrows at QF=40.The approach also improves mAP by WebP(+4%), JPEG2000 (+2.9%), and BPG (+2.1%) at matched average bitrate.
- Subjective evaluation: A subjective study with 10 viewers compares SSIM and GAN reconstructions, with GAN better on some textures and SSIM better on some faces.Figure 7 includes an elephant-skin example favoring GAN and a face example favoring SSIM.
5. Conclusion
Deep convolutional residual networks remove compression artifacts, while the conditional GAN framework produces finer details that improve reconstruction fidelity for both human viewers and object detectors.
- Deep convolutional residual networks can remove compression artifacts from images.
- SSIM-trained reconstruction achieves state-of-the-art results on standard image-similarity metrics.
- SSIM-based reconstructions can appear blurry and lack high-frequency details.
- The conditional GAN alternates full-size patch generation with sub-patch discrimination to produce finer, consistent details.
- Human evaluation and object-detection experiments show that GAN-generated details improve outcomes for both machines and humans.