Source-linked AI summary
Real-Time Adaptive Image Compression
Oren Rippel, Lubomir Bourdev
TL;DR
Existing codecs are difficult to adapt, while ML-based compression must achieve compact representations and efficient deployment. The paper proposes an autoencoder with multiscale analysis, adaptive coding, codelength regularization, and compression-specific adversarial training, reporting smaller files than established codecs and real-time GPU processing.
Problem
Existing codecs are hard-coded, while ML-based compression must produce compact representations and operate efficiently under computation, memory, and battery constraints.
Method
The paper uses an autoencoder with pyramidal feature extraction, adaptive arithmetic coding, expected-codelength regularization, and compression-specific adversarial training.
Results
The codec typically produces files 2.5 times smaller than JPEG and JPEG 2000, 2 times smaller than WebP, and 1.7 times smaller than BPG across quality levels.
Takeaways & Limitations
The approach combines improved compression performance with lightweight deployment and convincing reconstructions at very low bitrates.
Takeaways & Limitations
Comparisons with Theis et al. and Johnston et al. rely on results transcribed from their graphs because reconstruction access was unavailable.
Abstract
from arXiv · showhide
We present a machine learning-based approach to lossy image compression which outperforms all existing codecs, while running in real-time. Our algorithm typically produces files 2.5 times smaller than JPEG and JPEG 2000, 2 times smaller than WebP, and 1.7 times smaller than BPG on datasets of generic images across all quality levels. At the same time, our codec is designed to be lightweight and deployable: for example, it can encode or decode the Kodak dataset in around 10ms per image on GPU. Our architecture is an autoencoder featuring pyramidal analysis, an adaptive coding module, and regularization of the expected codelength. We also supplement our approach with adversarial training specialized towards use in a compression setting: this enables us to produce visually pleasing reconstructions for very low bitrates.
1. Introduction
Existing codecs are difficult to adapt and deploy efficiently, motivating a lightweight ML-based approach that improves compression, runs in real time, and produces convincing low-bitrate reconstructions.
- Motivation: Existing commercial codecs are hard-coded and cannot be customized beyond high-level hyperparameter tuning.They serve diverse transmission settings despite changing requirements and applications.
- Motivation: ML-based compression must address both compact representations and efficient deployment under computation, memory, and battery constraints.The paper frames progress on performance and computational feasibility as joint goals.
- Results: 2.5 times smaller than JPEG and JPEG 2000, 2 times smaller than WebP, and 1.7 times smaller than BPG across quality levels.These comparisons were reported on the Kodak PhotoCD and RAISE-1k 512×768 datasets.
- Results: 9ms encode and 10ms decode times on a GTX 980 Ti GPU demonstrate the codec’s lightweight deployment target.The paper contrasts these runtimes with slower JPEG 2000 and WebP processing.
- Low-bitrate reconstruction: Adversarial training specialized for compression produces convincing reconstructions at very low bitrates.The approach supplements the main algorithm with compression-specific adversarial training.
2. Background & Related Work
Traditional codecs use carefully engineered but fixed pipelines, while ML methods learn representations; this paper combines learned compression with adversarial training to improve very-low-bitrate reconstructions.
- Traditional compression: Traditional codecs use transformation, quantization, and encoding modules whose components are hard-coded and manually engineered to fit together.JPEG uses block DCT and run-length encoding, while JP2 uses wavelets and adaptive arithmetic coding.
- Traditional compression: Fixed transformations cannot adapt to input statistics, and traditional pipelines cannot directly optimize reconstructions for a chosen image-quality metric.Loss is also largely compartmentalized within quantization.
- ML-based compression: ML-based compression automatically discovers structure and commonly uses autoencoders with bitrate bottlenecks, reconstruction losses, and entropy coding.Prior work explored binary or quantized representations, entropy coding, SSIM-weighted losses, and spatially adaptive bit allocation.
- Generative adversarial networks: Generative adversarial networks train a generator to synthesize target-like outputs while a discriminator distinguishes generated samples from ground truth.This framework has also supported progress in photo-realistic image generation and related image tasks.
- Generative adversarial networks: Multiscale adversarial training encourages reconstructions to match ground-truth statistics, producing sharp results at very low bitrates.The paper presents this as the first proposed use of GANs for image compression.
3. Model
The model learns multiscale image structure, compresses quantized features with adaptive arithmetic coding and codelength regularization, and uses adversarial loss to pursue realistic reconstructions.
- Feature extraction: The encoder’s feature extractor uses pyramidal decomposition and interscale alignment to recognize structure across channels, individual scales, and scales.The decoder performs counterpart inverse operations.
- Code computation: The code module quantizes extracted features and applies adaptive arithmetic coding to their binary expansions.Adaptive arithmetic coding exploits structure in the resulting bitplanes, including sparsity and spatial correlation.
- Codelength regularization: Adaptive codelength regularization penalizes feature entropy to support better compression while targeting an expected code length.The broader architecture modulates expected codelength toward a prescribed bitrate.
- Adversarial training: Adversarial training supplies a discriminator loss that encourages realistic reconstructions.The loss penalizes discrepancies between reconstruction and target distributions.
3.1. Feature extraction
The feature extractor learns nonlinear, scale-specific representations inspired by wavelet analysis, then aligns and jointly processes information across scales.
- PYRAMIDAL DECOMPOSITION: The pyramidal encoder learns nonlinear feature extractors separately at each scale, generalizing wavelet decomposition beyond fixed linear filters.Unlike JPEG 2000's hard-coded wavelet filters, the extractors are learned and nonlinear.
- PYRAMIDAL DECOMPOSITION: For each scale m, the encoder produces coefficient maps c_m from x_m and downsamples x_m to form the input x_m+1 for the next scale.The first scale receives the model input x, and each extractor outputs coefficients with scale-specific dimensions.
- PYRAMIDAL DECOMPOSITION: The pyramid uses six scales, with convolutional feature extractors and learned 4 × 4 stride-2 downsamplers.Feature extractors use 3 × 3 or 1 × 1 convolutions and leaky ReLUs with leak 0.2.
- INTERSCALE ALIGNMENT: Interscale alignment combines coefficients from different scales to exploit shared information unavailable in classic wavelet analysis.The module maps each coefficient tensor to a common spatial size and channel count before joint processing.
- INTERSCALE ALIGNMENT: After resampling each coefficient tensor, the module sums the transformed maps and applies a nonlinear transformation to produce output y.The resampling functions use convolutions or deconvolutions with suitable strides, followed by a sequence of 3 × 3 convolutions.
3.2. Code computation and regularization
The codec quantizes learned features, losslessly decomposes them into binary bitplanes, and adaptively arithmetic-codes the resulting tensor. Adaptive codelength regularization shapes the representation so the coder can exploit sparsity and spatial structure while targeting an expected bit count.
- Quantization: Quantization maps feature tensor y into 2^B equal-sized bins, with B = 6 used for all models in the paper.The quantized tensor is denoted ˆy.
- Bitplane decomposition: Each quantized value is losslessly expanded into B binary bitplanes, producing b ∈ {0, 1}^B×C×H×W.Each spatial feature map becomes B binary planes through the value’s binary expansion.
- Adaptive arithmetic coding: Higher bitplanes are sparser and neighboring bits often agree, creating structure that adaptive arithmetic coding exploits for compact representations.The bitplane decomposition is lossless, while the coder removes redundancy from the structured binary tensor.
- Adaptive arithmetic coding: Adaptive arithmetic coding predicts each bit from position and previously decoded neighboring bits, then uses those probabilities for variable-length compression.During decoding, contexts are restricted to already decoded bits.
- Adaptive codelength regularization: Adaptive codelength regularization modulates the quantized representation to target an expected code length ℓtarget across inputs.The regularizer encourages structure where the adaptive arithmetic coder can exploit it.
- Adaptive codelength regularization: The regularizer penalizes element magnitudes and spatial-neighbor differences, while feedback adjusts αt until the effective bit count reaches equilibrium.If the measured bit count is too high, αt increases; if too low, it decreases.
- Adaptive codelength regularization: A total-to-target ratio BCHW/ℓtarget = 4 works well across all explored architectures, exposing a trade-off between representation capacity and sparsity.Increasing channels or spatial map size raises available bits while requiring greater sparsity.
4. Realistic Reconstructions via Multiscale Adversarial Training
The paper adapts adversarial training to compression by comparing each target-reconstruction pair jointly and aggregating discriminator evidence across scales. An adaptive training scheme stabilizes the competing discriminator and reconstructor updates.
- 4.1. Discriminator design: The discriminator treats the target and reconstruction jointly as one example, asking which image is real rather than labeling each independently.This differs from GAN formulations that assign separate real/fake labels to the target and reconstruction.
- 4.1. Discriminator design: Uniformly swapping target and reconstruction within each discriminator input pair prevents the input order from determining the real image.The swapped pairs are propagated through the discriminator before their outputs are aggregated.
- 4.1. Discriminator design: Multiscale discriminator outputs are averaged across branches at different depths to detect artifacts that vary with image scale.The architecture is motivated by scale-dependent artifacts such as noise and blurriness.
- 4.1. Discriminator design: The discriminator is applied to the aggregate sum across scales before the objectives are optimized.The complete discriminator architecture is described as a multiscale network.
- 4.2. Adversarial training: Training balances reconstruction and confusion gradients by magnitude and alternates discriminator and reconstructor updates according to prediction accuracy.The supplied procedure defines lower and upper accuracy thresholds for freezing or propagating the confusion signal.
5. Results
The evaluation measures perceptual quality, bitrate, runtime, and reconstruction appearance across Kodak and RAISE-1k datasets. It uses MS-SSIM-based rate-distortion analyses and compares the approach with commercial and machine-learning codecs.
- Evaluation setup: MS-SSIM is used for training and testing because it is designed to match human visual perception and is more representative than PSNR-like losses.The metric is evaluated with color-space-appropriate component weighting.
- Results: The reported result types are average MS-SSIM versus BPP, relative compressed file size versus MS-SSIM, encode/decode timing versus MS-SSIM, and visual reconstruction examples.These results are presented in figures and tables across the evaluation datasets.
- Test sets: Kodak contains 24 images, so the study supplements it with RAISE-1k, which contains 1,000 raw images resized to 512 × 768.The authors note that Kodak may be susceptible to overfitting and may not capture broader natural-image statistics.
- Codecs: The study compares JPEG, JPEG 2000, WebP, BPG, and available recent machine-learning codecs using best-performing configurations and header-adjusted bitrates.BPG is evaluated in 4:2:0 and 4:4:4 configurations where applicable.
- Evaluation setup: The evaluation records BPP, weighted MS-SSIM, and encode/decode computation times for each approach, color space, image, and available rate.These measurements cover both compression performance and runtime.
- Performance evaluation: Rate-distortion curves are computed per image, interpolated densely across rates, and then averaged at fixed independent-variable values.The procedure addresses the fact that different summaries of individual rate-distortion curves can produce disparate results.