PaperlayerView exact revision on arXiv

Source-linked AI summary

Quantization Guided JPEG Artifact Correction

Max Ehrlich, Larry Davis, Ser-Nam Lim, Abhinav Shrivastava

arXiv:2004.09320v2eess.IVcs.CVcs.LGstat.ML

TL;DR

JPEG compression loses information, and existing artifact-correction methods often require a separate model for each quality setting. This paper uses a single model parameterized by the JPEG quantization matrix. It reports state-of-the-art results while beating methods trained separately for each quality level.

  • Problem

    Deep-learning JPEG artifact correction is limited by methods that require a separate model for each quality setting, complicating practical use when quality is unknown at inference.

    Source: §1 · p. 1

  • Method

    The paper performs fully DCT-domain image regression with a single network whose convolutional filters are parameterized by the JPEG quantization matrix.

    Source: §1 · pp. 1–2, §3.1 · p. 4

  • Results

    The single network achieves state-of-the-art results, beating methods trained with a different network for each quality level.

    Source: §5 · p. 14

  • Takeaways & Limitations

    The network relies on information available at inference time and addresses a practical deployment problem for JPEG artifact correction.

    Source: §5 · p. 14

    GAN-restored texture is generally unlike the regression target at inference and produces worse numerical results than regression, despite looking more realistic.

    Source: §3.4 · p. 9

Abstract

from arXiv · show

The JPEG image compression algorithm is the most popular method of image compression because of its ability for large compression ratios. However, to achieve such high compression, information is lost. For aggressive quantization settings, this leads to a noticeable reduction in image quality. Artifact correction has been studied in the context of deep neural networks for some time, but the current state-of-the-art methods require a different model to be trained for each quality setting, greatly limiting their practical application. We solve this problem by creating a novel architecture which is parameterized by the JPEG files quantization matrix. This allows our single model to achieve state-of-the-art performance over models trained for specific quality settings.

1 Introduction

JPEG’s high compression can reduce image quality, while existing correction methods are limited by requiring a separate model for each quality setting. The paper proposes a single quantization-guided model and identifies its DCT-domain formulation and Convolutional Filter Manifolds as contributions.

1 Introduction

  • JPEG’s lossy compression is especially noticeable at low quality, motivating artifact correction for space-constrained scenarios using aggressive compression.Artifact correction has been studied in classical literature and with deep neural networks.Source: §1 · p. 1
  • Separate models for each JPEG quality level are costly to train and deploy, and unknown quality at inference makes selecting the correct model impractical.The paper notes that quality is not stored with the JPEG image.Source: §1 · p. 1
  • The proposed approach processes images in the DCT domain, uses the reconstructed Y channel to guide color reconstruction, and fine-tunes with a GAN loss to restore texture.The network is parameterized by the 8 × 8 quantization matrix and aims to produce realistic reconstructions.Source: §1 · p. 2
  • A single model parameterized by the JPEG quantization matrix is presented as state of the art in color restoration, alongside fully DCT-domain regression and Convolutional Filter Manifolds.The quantization matrix is stored in the JPEG file, and the network adapts its weights to that input.Source: §1 · pp. 1–2

2 Prior Work

Prior work spans classical transform-based correction, CNN restoration, dual-domain models, and methods for blind restoration. These approaches use different techniques to address JPEG artifacts and, in blind settings, to handle unknown quality.

2 Prior Work

  • Classical methods include Pointwise Shape-Adaptive DCT, which reconstructs local estimates from thresholded DCT coefficients, and a lapped-transform approximation to inverse DCT.The lapped-transform method operates on quantized coefficients.Source: §2 · p. 2
  • CNN approaches include regression, hierarchical skip connections, wavelet-based restoration, GANs, scalable architectures, dense residuals, and persistent memory [8] [40] [5] [27] [6] [12] [52] [42].Galteri et al. [12] use a GAN formulation for more visually appealing results; S-Net [52] varies output quality with computation complexity.Source: §2 · pp. 2–3
  • Dual-domain methods combine pixel and DCT processing, including DCT error rectification, implicit DCT loss, and separate low- and high-frequency streams [49] [51].Jin et al. report competitive results with a fraction of the parameters.Source: §2 · p. 3
  • Blind-restoration approaches include single networks, quality-factor estimation with ensembles, and per-pixel quality maps [48].Galteri et al. use an auxiliary classifier to select a quality-specific GAN; AGARNET uses a single network with image and per-pixel quality inputs.Source: §2 · p. 3

3 Our Approach

The approach uses the JPEG quantization matrix to guide fully DCT-domain restoration, first correcting the Y channel and then using it to guide Cb/Cr correction. Its building blocks preserve frequency relationships while parameterizing convolutional filters with quantization data.

3 Our Approach

  • A single architecture uses the JPEG quantization matrix to guide fully DCT-domain image-to-image regression, restoring Y before using it to guide the more degraded color channels.The quantization matrix records rounding applied to DCT coefficients; JPEG color channels are subsampled and more heavily quantized than Y.Source: §3 · p. 3

3.1 Building Blocks

  • Convolutional Filter Manifolds replace a multilayer perceptron with a lightweight three-layer CNN that maps the spatial quantization matrix to convolutional kernels.The resulting model matches state-of-the-art pixel and dual-domain results using only DCT coefficients as input and output.Source: §3.1 · pp. 4, 6

3.2 Y Channel Correction Network

  • The Y-channel network uses BlockNet to form quantization-invariant block representations and FrequencyNet to process individual DCT frequencies.The final network applies BlockNet, FrequencyNet, and a second BlockNet in sequence, then fuses their outputs into a residual added to the input coefficients.Source: §3.2 · pp. 6–7

3.3 Color Correction Network

  • The color network processes Cb and Cr independently, concatenating their block representations with a restored-Y representation to supply structural information.It upsamples the color representation before concatenation and adds the predicted residual to nearest-neighbor-upsampled input coefficients.Source: §3.3 · pp. 7–8

3.4 Training

  • Training proceeds in stages: train Y correction, train color correction with Y weights frozen, then train the full network with the GAN loss.The regression objective combines l1 error and SSIM maximization; the GAN-stage texture loss uses a MINC-trained network and is reported as critical for restoring texture to blurred regions.Source: §3.4 · pp. 8–9

4 Experiments

The experiments evaluate a single quantization-guided model across JPEG qualities, datasets, and color restoration, then test generalization and architectural choices. The reported results show competitive restoration, while ablations favor quantization-adaptive CFM layers and fusion.

4.1 Experimental Setup

  • Training uses DIV2k and Flickr2k patches compressed at qualities 10–100; evaluation uses Live1, Classic-5, BSDS500, and ICB.The training set contains 1,065,000 patches; ICB comprises 15 high-quality lossless images designed to measure compression quality.Source: §4.1 · p. 9

4.2 Results: Artifact Correction

  • The color model consistently outperforms recent baselines on Live1, BSDS500, and ICB, using native color processing alongside IDCN.Other compared models process YCbCr channels independently after conversion.Source: §4.2 · p. 10
  • Intermediate Y-channel results match or outperform prior state-of-the-art models in many cases, with consistently high SSIM across quality factors from one model.The evaluation covers Live1, Classic-5, BSDS500, and ICB.Source: §4.2 · p. 10, Table 2 · p. 11
  • GAN fine-tuning restores texture to blurred flat regions and sharpens edges, but produces higher-quality images at the expense of quantitative metrics.The authors report that Table 4 FID scores indicate significantly more realistic images than the regression model.Source: §4.2 · p. 11, Table 4 · p. 13

4.3 Results: Generalization Capabilities

  • Across three color datasets, average PSNR improvement is shown for qualities 10–100 and drops steeply at quality 90.Figure 11 reports the improvement trend; no exact values are supplied here.Source: Figure 11 · p. 12
  • IDCN’s quality-10 and quality-20 models perform significantly worse on mismatched qualities; at quality 50, the quality-10 model is worse than the input JPEG.The paper notes that JPEG files do not store the quality setting, leaving a deployed system no way to select the correct IDCN model.Source: §4.3 · pp. 11–12, Table 3 · p. 12

4.4 Design and Ablation Analysis

  • The CFM unit performs considerably better than concatenating the quantization matrix with the input or ignoring it.The ablation experiments hold model parameter counts approximately equal.Source: §4.4 · pp. 12–13, Table 5 · p. 13
  • BlockNet alone performs significantly better than FrequencyNet alone, while removing fusion leaves the network at input PSNR after training.The network with fusion makes considerable progress; the authors posit fusion supports gradient flow to early layers.Source: §4.4 · pp. 13–14

5 Conclusion

The conclusion presents a quantization-guided JPEG artifact correction network that achieves state-of-the-art results with one model across quality levels. It uses inference-time information to address a practical deployment problem.

5 Conclusion

  • The single network achieves state-of-the-art results, beating methods that train a different network for each quality level.The authors state that the network relies only on information available at inference time.Source: §5 · p. 14
  • The network’s inference-time information addresses a practical deployment problem for artifact-correction methods.The conclusion frames real-world deployment as the practical scope of this benefit.Source: §5 · p. 14
  • The paper presents a design for quantization-guided JPEG artifact correction.Source: §5 · p. 14

A Additional Evaluation Details

The evaluation adapts prior methods to color images and discusses settings that affect comparison fairness. It also flags sensitivity in PSNR-B evaluation and provides code and pretrained weights.

A Additional Evaluation Details

  • For prior methods without native color handling, the authors evaluate YCbCr and RGB channel-wise variants and report the better YCbCr results.They preserve published network structures rather than modifying them for three-channel input.Source: A Additional Evaluation Details · p. 18
  • SSIM comparisons depend on using the ARCNN evaluation setting: an 8 × 8 uniformly weighted window rather than the default 11 × 11 Gaussian window.The authors say prior works have not applied this setting uniformly.Source: A Additional Evaluation Details · p. 18
  • PSNR-B is asymmetric because its blocking effect factor is computed only on the degraded image, making argument order critical.The authors report a prior work that reversed the arguments and obtained PSNR-B nearly equal to PSNR.Source: A Additional Evaluation Details · p. 18
  • The authors provide model and evaluation code plus pretrained weights, with evaluation code checked against ARCNN MATLAB code.Source: A Additional Evaluation Details · p. 18

B Further Analysis

The further analysis covers learned CFM filters, model interpolation, JPEG storage savings, frequency-domain results, and runtime throughput. The supplied passage outlines these analyses without reporting their findings.

B Further Analysis

  • The analysis visualizes CFM layers and what they learn to examine their contribution to the results.Source: B Further Analysis · p. 18
  • The authors compare qualitative outputs at varying interpolation strengths between the regression and GAN models.Source: B Further Analysis · p. 19
  • The analysis studies storage savings from keeping low-quality JPEG images and restoring them with the method.Source: B Further Analysis · p. 19
  • The authors examine frequency-domain qualitative results, including whether the GAN model generates more high-frequency content than the regression model.Source: B Further Analysis · p. 19
  • Runtime throughput is compared with the other tested methods.Source: B Further Analysis · p. 19

B.1 Understanding Convolutional Filter Manifolds

The visualizations show how CFM filters and their activations vary with JPEG quality, while interpolation and equivalent-quality experiments examine model behavior across settings and potential space savings.

B.1 Understanding Convolutional Filter Manifolds

  • For the same channel, CFM filter patterns remain roughly consistent as quality rises, while their response shrinks because high-quality JPEGs need less correction.Figure 1 compares quality levels 10, 50, and 100 across channels.Source: Figure 1 · p. 19, §B.1 · p. 19
  • The t-SNE embeddings form an ordered path from low to high quality; low-quality filters remain distinct, while higher-quality filters converge.The embeddings use weights generated for all 101 quantization matrices; circled points mark matrices seen during training.Source: Figure 3 · p. 20, §B.1 · p. 20
  • Maximally activated CFM weights show prominent blocking artifacts at quality 10, suppressed blocking but remaining structural artifacts at 50, and mostly input noise at 100.The visualization shows Y-channel responses for the same channels and layer as Figure 1.Source: §B.1 · pp. 19–20

B.2 Model Interpolation

  • Blending GAN and regression model parameters suppresses artifacts introduced by the GAN alone in qualitative examples generated from quality-10 JPEGs.The interpolation uses α values 0.0, 0.7, 0.9, and 1.0; α=0 is regression and α=1 is GAN.Source: §B.2 · pp. 20–21

B.3 Equivalent Quality

  • On Live-1, equivalent JPEG quality and file-size savings are assessed by matching restored-image SSIM to progressively higher-quality JPEGs.The study averages quality increase and space savings across the dataset; it notes that SSIM-matched JPEGs can look worse than the model output.Source: §B.3 · p. 21

B.4 Frequency Domain Analysis

The frequency-domain analysis compares the original image, a quality-10 JPEG, and regression and GAN restorations using DCT frequency content.

B.4 Frequency Domain Analysis

  • The regression model fills in high frequencies, while the GAN model nearly matches the original images in frequency saturation.For each image, the analysis plots the probability that each of 15 spatial frequencies in a DCT block is nonzero.Source: §B.4 · p. 21
  • The comparison uses colormapped DCTs of original images, quality-10 JPEGs, and regression and GAN restorations.The analysis uses the Y channel from several images.Source: §B.4 · p. 21
  • The frequency plots report, for each image, the probability that each of 15 spatial frequencies in a DCT block has magnitude greater than zero.Source: §B.4 · p. 21

B.5 Runtime analysis

Runtime analysis plots inference speed against SSIM increase for quality-10 Live-1 images, with specified hardware, image, and comparison constraints.

B.5 Runtime analysis

  • Inference speed is measured on an NVIDIA Pascal GPU using 100 frames at 1280 × 720 resolution.Source: §B.5 · pp. 21, 27
  • ARCNN is omitted because its authors provide no GPU-accelerated inference code; grayscale-only models are tested on single-channel images, not run three times for RGB.Source: §B.5 · p. 27
  • Figure 8 plots frames per second against SSIM increase for quality-10 Live-1 images.Source: Figure 8 · p. 27, §B.5 · p. 27

C Qualitative Results

The paper presents regression-network examples for quality-10 and quality-20 images and outlines JPEG compression and decompression, including where compression loses information.

C Qualitative Results

D JPEG Compression Algorithm

  • JPEG compression converts RGB to YCbCr, centers channels, applies a blockwise 2D DCT, quantizes coefficients, and zigzag-orders them before run-length and entropy coding.The account also describes padding and chroma subsampling; quantized coefficients are computed using Y and color quantization tables.Source: D JPEG Compression Algorithm · pp. 27, 30, 31
  • Decompression reverses entropy coding, run-length coding, and zigzag ordering, then scales blocks using the stored quantization table before reversing the DCT and restoring the image.The decompressor reads the table from the JPEG file rather than deriving it from a scalar quality setting.Source: D JPEG Compression Algorithm · pp. 31–32
  • Quantization loses fractional remainders, and coefficients rounded to zero remain lost after scaling; the resulting zero runs support higher compression ratios.Source: D JPEG Compression Algorithm · p. 31

Related papers

Loading 2004.09320v2…