PaperlayerView exact revision on arXiv

Source-linked AI summary

An Introduction to Neural Data Compression

Yibo Yang, Stephan Mandt, Lucas Theis

arXiv:2202.06533v3cs.LGcs.ITeess.IV

TL;DR

Neural compression addresses how to learn effective compression algorithms from data using neural and generative models. This monograph organizes the field’s information-theoretic and machine-learning foundations, reviews its main methods and extensions, and reports rapid progress over handcrafted codecs alongside substantial practical challenges.

  • Problem

    Neural compression requires connecting generative modeling with compression objectives while addressing the field’s essential information-theoretic and perceptual concepts.

  • Method

    The monograph reviews neural compression through lossless and lossy coding, generative models, learned quantization, diffusion, perceptual quality, task-oriented compression, and video compression.

  • Results

    Neural image compression outperformed the best known handcrafted codecs in 2018, and classical codecs were absent from the 2021 learned-image-compression challenge’s top 10.

  • Takeaways & Limitations

    Neural compression has made significant strides in both lossless and lossy compression and is being applied to emerging data types such as point clouds, implicit 3D surfaces, and neural radiance fields.

  • Takeaways & Limitations

    Computational complexity remains a major obstacle to wider adoption, with additional open problems in loss functions, evaluation, quantization-free compression, and miscommunication risk.

Abstract

from arXiv · show

Neural compression is the application of neural networks and other machine learning methods to data compression. Recent advances in statistical machine learning have opened up new possibilities for data compression, allowing compression algorithms to be learned end-to-end from data using powerful generative models such as normalizing flows, variational autoencoders, diffusion probabilistic models, and generative adversarial networks. The present article aims to introduce this field of research to a broader machine learning audience by reviewing the necessary background in information theory (e.g., entropy coding, rate-distortion theory) and computer vision (e.g., image quality assessment, perceptual metrics), and providing a curated guide through the essential ideas and methods in the literature thus far.

Introduction

Neural compression connects data compression with modern generative modeling, building on earlier learning-based approaches while addressing compression-specific objectives. This monograph introduces essential concepts and methods for machine-learning readers and highlights these connections.

  • Introduction: Learning-based compression predates deep learning, with neural networks applied to image compression as early as the late 1980s and 1990s.Modern methods differ in scale, neural architectures, and encoding schemes.
  • Introduction: Current neural compression research draws on GANs, VAEs, normalizing flows, and autoregressive models to capture complex data distributions.These models are often developed for realistic samples or high log-density, objectives that are not necessarily aligned with compression.
  • Introduction: The monograph introduces essential neural-compression concepts and methods for machine-learning readers who may not have a data-compression background.It complements more specialized or applied surveys.
  • Introduction: It emphasizes connections between neural compression, generative modeling, and machine learning more broadly.The article aims to facilitate exchange between generative modeling and compression research.

116 Introduction

The introduction presents compression as a pipeline that transforms, quantizes, entropy-codes, and reconstructs data, using JPEG to motivate learned alternatives. It then outlines the monograph’s coverage of lossless, lossy, perceptual, task-oriented, and video compression.

  • JPEG as a motivating example: JPEG transforms pixels into coefficients, quantizes them, and losslessly compresses the resulting integers using run-length and entropy coding.Its color channels are processed independently, divided into 8 × 8 blocks, and transformed with the DCT.
  • JPEG as a motivating example: A typical lossy-compression pipeline maps an image x to coefficients z, quantizes them to ˆz, entropy-codes them, and reconstructs ˆx with decoder g.Neural compression may additionally use an adversarial critic to encourage realism and perceptual quality.
  • Learned transforms: Learned neural networks can replace JPEG’s linear transforms, linking lossy compression to autoencoders and variational autoencoders.Compression coefficients correspond to latent variables in generative-model terminology.
  • Overview: The monograph covers lossless and lossy compression, with lossy compression relying on lossless coding for compressed representations.It begins with coding theory and learned discrete distributions, then reviews generative-model classes and their rate–efficiency trade-offs.
  • Scope of the monograph: The monograph extends transform coding to end-to-end learning, compression without quantization, perceptual quality, downstream tasks, and video compression.Its central setting assumes i.i.d. data, while video compression lifts that assumption along the temporal dimension.

Lossless Compression

Lossless compression models data probabilities and uses entropy coding to convert outcomes into compact bit strings while preserving exact reconstruction. The section connects classical coding schemes with neural methods, including autoregressive, latent-variable, and other generative models, while highlighting trade-offs between rate and decoding speed.

  • Lossless compression reconstructs data perfectly by modeling its probability distribution and feeding those probabilities into an entropy-coding scheme.Entropy coding converts data into compact bit strings; non-likelihood-based models such as GANs are difficult to use because probabilities are unavailable or may be zero.
  • Entropy measures uncertainty through expected surprise, with an outcome’s information content given by −log2 P(x).A fair coin has entropy of 1 bit, while entropy approaches 0 as one outcome becomes nearly certain.
  • Cross-entropy is the average coding cost when samples from P use a code optimized for Q, and its excess over entropy is the KL divergence.Maximum-likelihood estimation is equivalent to minimizing cross-entropy, which is never below the entropy.
  • Huffman coding: Huffman coding builds a binary tree that assigns shorter codewords to more probable symbols, but incurs up to 1 bit per symbol beyond information content.Streaming codes reduce this overhead to 1 bit for an entire message rather than each symbol.
  • Arithmetic coding: Arithmetic coding assigns intervals proportional to message probabilities and amortizes coding overhead across the sequence, reaching at most H[P, Q] + 2 average bits.Its per-symbol overhead is 2/n bits, compared with up to 1 bit per symbol for Huffman coding.
  • Neural compression extends lossless coding with autoregressive, latent-variable, and other generative models, but sequential decoding and rate–speed trade-offs remain important constraints.Autoregressive decoding is inherently sequential; a two-part latent-variable code achieved much faster decoding than autoregressive models at approximately 20% worse bit-rate, while other models target parallelism or faster compression.

Lossy Compression

Lossy compression encodes data into a discrete representation and reconstructs it imperfectly to trade reconstruction fidelity against bit-rate. Rate-distortion theory characterizes this trade-off, while practical codecs balance theoretical performance with computational constraints.

  • Lossy compression communicates a discrete representation using few bits while measuring reconstruction quality with a distortion function.
  • A lossy codec consists of an encoder, decoder, and entropy code, with rate measured by expected code length and distortion by reconstruction error.
  • Rate-distortion theory defines the lowest achievable bit-rate for a given distortion through the information rate-distortion function RI(D), which is non-increasing and convex.
  • Practical codec design replaces constrained rate-distortion optimization with the Lagrangian L(λ, c) = R(c) + λD(c), selecting codecs for different trade-offs through λ.
  • The operational R-D curve may be non-convex, lies above the information curve, and differs from the asymptotic multi-sample definition used in usual R-D theory.
  • Vector quantization can approach the theoretical limit but scales poorly in high dimensions, whereas transform coding sacrifices unconstrained optimality for substantially better scalability.

3.2 Neural lossy compression

Neural lossy compression replaces transform-coding components with learned nonlinear functions and entropy models optimized end-to-end. Its design connects rate-distortion objectives to autoencoders and VAEs while supporting progressive and variable-rate representations.

  • Nonlinear transforms are more flexible than linear transforms, adapt better to data distributions, and can be optimized for custom losses such as perceptual losses.
  • Neural lossy compression replaces analysis and synthesis transforms and entropy models with learned functions, optimized end-to-end on the target data.
  • The encoder produces continuous transform coefficients z, quantization yields discrete latents ˆz, and an entropy model losslessly transmits them.
  • The rate-distortion objective combines reconstruction distortion with the information cost of the quantized representation, but quantization and rate terms are non-differentiable for SGD.
  • Deterministic nonlinear transform coding can be interpreted as a variational autoencoder with a degenerate posterior concentrated on the quantized transform coefficients.
  • Recurrent and hierarchical architectures encode residual information across stages, enabling progressive reconstruction as additional latent representations are transmitted.

3.3 Learned quantization and rate control

Learned lossy compression replaces non-differentiable quantization with trainable approximations, while jointly designing entropy models and rate-control mechanisms. The surveyed methods range from binarization and vector quantization to uniform quantization and stochastic alternatives, with empirical comparisons favoring different approximations for rate and distortion.

  • Motivation: End-to-end rate-distortion training was hindered because quantization and the discrete rate loss are non-differentiable.Quantization has zero derivatives almost everywhere and is undefined at discontinuities.
  • Binarization: Early methods binarized encoder outputs, controlled rate through representation constraints or masking, and often fit separate autoregressive entropy models after training.Stochastic and deterministic binarization used straight-through estimation, while later approaches modeled nonzero elements with PixelCNN-style entropy models.
  • Vector quantization: Vector quantization learns codebook vectors and approximates hard assignments with differentiable soft assignments weighted by distances to codebook entries.The soft assignments are annealed toward hard quantization, while computation is restricted to small latent blocks because full vector quantization is expensive.
  • Uniform quantization: Uniform quantization rounds latent elements to integers and is computationally cheaper than vector quantization, relying on flexible transforms to accommodate the fixed grid.Its integer-valued representations can be paired with entropy models and rate-control schemes.
  • Comparisons: SGA+STE achieved improved rate-distortion performance over UQ+STE, and comparisons favored rounding-based distortion approximations with uniform-noise rate approximations.These findings support using different approximations for the distortion and rate terms.
  • Rate control: A learned quantization-grid method assigns more bits to latent dimensions with higher posterior variance and fewer bits where dKL[q(z_i|x)∥p(z_i)] is small.Adjusting λ at compression time enables variable bit-rate compression; the method can outperform JPEG but is weaker than end-to-end optimization at lower bit-rates.
  • Implementation constraint: Exact sender–receiver agreement on entropy-model probabilities is required for entropy coding, making floating-point round-off and nondeterministic GPU operations practical concerns.The requirement becomes especially stringent when conditional entropy models are computed on the fly.

3.4 Compression without quantization

Compression without quantization communicates a continuous stochastic latent instead of a discretized representation, accepting noise in the transmitted representation. This avoids the training–testing mismatch caused by differentiable quantization surrogates, but stochastic coding introduces its own requirements and overheads.

  • Motivation: Differentiable training surrogates create a mismatch between soft quantization during training and hard quantization at test time, generally causing sub-optimal performance.Annealing can reduce the mismatch but may produce high-variance gradients and requires an annealing schedule.
  • Stochastic coding: Continuous stochastic latents can be transmitted using finite bits, framing the problem as channel simulation, reverse channel coding, or relative entropy coding.This approach avoids quantization by communicating a sample z ∼q(z | x) rather than a quantized latent.
  • Limitation: Bits-back coding is not directly applicable to reverse channel coding because the decoder must eventually have access to the exact data x.Thus, bits-back solves lossless source coding but not the problem of communicating a stochastic latent sample.

lossless compression.

Bits-back is a solution to lossless source coding but not to reverse channel coding because it requires the decoder to obtain the exact data x.

  • Scope: Bits-back coding does not solve reverse channel coding because its decoder must eventually have access to the exact data x.The method is therefore applicable to lossless source coding, not direct communication of stochastic latent samples.

3.4. COMPRESSION WITHOUT QUANTIZATION

Reverse channel coding can communicate stochastic latent samples at a cost close to their mutual information, but unavoidable logarithmic overhead and computational limits constrain large-scale transmission. The section presents dithered quantization as an efficient special case for additive uniform noise.

  • Coding cost: I[X, Z] + log2(I[X, Z] + 1) + 5 bits is an upper bound on the average cost of communicating z.The bound is close to the information contained in z, while further significant reduction is generally impossible.
  • Overhead: Even optimal encoders and decoders may incur overhead logarithmic in mutual information, although the relative overhead is small when mutual information is large.Bundling more information can increase mutual information and reduce relative overhead, but may be computationally expensive.
  • Computational limitation: No general reverse channel coding algorithm has computational cost polynomial in the information content when transmitting large amounts of information at once.This limits the computational practicality of minimizing bit cost through large-scale bundling.
  • Strategies: The reviewed strategies include an efficient method for simulating additive uniform-noise channels and a general method for arbitrary distributions.The section positions dithered quantization as the specialized efficient approach.
  • Dithered quantization: Dithered quantization communicates a noisy latent by subtracting shared uniform noise, rounding, and adding the noise back at the decoder.The encoder entropy-codes the rounded value, while the decoder uses the shared noise to recover a sample with the desired distribution.
  • Modeling condition: The expected coding cost reaches equality when the entropy model p_Z matches the true marginal distribution of Z.Under a faithful model, the coding cost equals the amount of information transmitted.

3.4. COMPRESSION WITHOUT QUANTIZATION

Neural compression can avoid explicit quantization through stochastic communication schemes, including minimal random coding and reverse channel coding. These approaches support differentiable optimization and can achieve strong rate-distortion performance, though computational cost and sampling requirements remain important considerations.

  • Differentiable quantization: Differentiable quantization enables neural encoders and decoders to be optimized with backpropagation without the usual train-test mismatch.Agustsson and Theis used these properties to train image compression models.
  • Minimal random coding: Minimal random coding samples from a proposal distribution, encodes the selected index using log2 N bits, and reconstructs the corresponding sample at the decoder.The procedure’s complexity depends on the number of generated proposal samples N.
  • Minimal random coding: Concentration of log q(Z | x)/p(Z) helps the selected sample converge rapidly, and minimal random coding has been applied to neural network and image compression.Theis and Yosri further reduced its coding cost without loss of quality.
  • Minimal random coding: Minimal random coding communicates approximately from arbitrary distributions using an average of I[X, Z] bits.The communicated sample converges in total variation to q(z | x)p_data(x) as the number of samples increases.
  • Reverse channel coding: Communication without quantization requires noise, motivating comparisons between deterministic quantized encoders and stochastic encoders.Ballé et al. argue that deterministic encoders can improve on dithered quantization under rate-distortion evaluation.
  • Reverse channel coding: Reverse channel coding enables compression schemes beyond classical transform coding, including diffusion-based approaches that communicate noisy data before generating an estimate.Diffusion coding performed especially well on small images when realism was considered, despite adding Gaussian noise directly to the data.

3.5 Perceptual losses

Perceptual losses address the difficulty of deciding which information to discard in lossy compression and how reconstruction quality should be measured. Existing metrics capture different aspects of perception, but none fully predicts human judgments across all distortions, especially for generative compression.

  • Motivation: Lossy compression must balance bit savings against reconstruction errors that are as imperceptible as possible.This makes the training objective more complicated than minimizing bits in lossless compression.
  • Quality assessment: Full-reference metrics use both an image and its reconstruction, whereas no-reference metrics judge the reconstruction alone.MOS is associated with no-reference judgments, while DMOS uses both unprocessed and reconstructed data.
  • Quality assessment: MSE and PSNR are widely used but do not predict perceived quality well, while SSIM generally correlates better with human visual perception.SSIM measures luminance, contrast, and structural correlation over local image patches.
  • Open limitations: SSIM predicts human judgments better than MSE for common distortions but tends to fail on reconstructions from generative compression.Directly optimizing perceptual metrics can also produce mixed results and blind spots across distortions.
  • Learned perceptual metrics: Deep representations support perceptual distortions such as VGG-based losses, LPIPS, and the unsupervised PIM metric.LPIPS fine-tunes pretrained classifier representations to match human responses, while PIM learns representations with a contrastive loss.
  • Open limitations: No metric yet matches humans predicting other humans, and randomly initialized networks can perform as well as VGG-based distortions for human-response prediction.These findings leave the usefulness of neural representations for perceptual distortion an open question.
  • Distortions and divergences: Distortion-only optimization can produce artefacts for some data unless the compressed representation preserves sufficient information.This provides a theoretical argument for considering divergences alongside distortions.

3.6 Task-oriented compression

Task-oriented compression preserves performance on downstream tasks rather than prioritizing perceptual reconstruction quality. It formalizes compression around task supervision and extends neural methods to video through choices about prediction, residual coding, latency, and temporal structure.

  • Task-oriented compression: For machine-processed data, task-oriented compression targets downstream performance instead of perceptual reconstruction quality.Operating directly on compressed representations can reduce both bit-rate and computational requirements.
  • Task-oriented compression: Task-oriented compression encodes X while using a shared target Y as supervision, such as pixel-level semantic labels for an image.The compressed representation is optimized in relation to the task target rather than reconstruction alone.
  • Task-oriented compression: The Information Bottleneck uses R = I[X, Z] and D = −I[Z, Y] to encourage Z to be a minimal sufficient statistic for predicting Y.This is presented as an early example of task-oriented compression.
  • Task-oriented compression: Practical task-oriented systems often use discrete Z, measure rate with H[Z], and define distortion through task losses such as cross-entropy or squared error.A decoder may compute predictions from z with a neural network.
  • Neural video compression: Neural video codecs typically combine motion compensation with residual compression, while differing in predictive models, residual domains, and latency settings.Online systems operate frame by frame, whereas offline systems can encode a whole video using future frames.
  • Neural video compression: Neural video compression has explored key-frame interpolation, recurrent prediction, structured priors, learned optical flow, and scale-space flows.These designs trade modeling flexibility, bit-rate, runtime, and predictive accuracy in different ways.
  • Neural video compression: Classical codec components remain influential in neural video compression, but simplified approaches such as VCT have achieved state-of-the-art results.The field includes both hybrid motion-and-residual architectures and approaches that reduce this inherited complexity.

Discussion and Open Problems

Neural compression has progressed from barely surpassing JPEG 2000 to outperforming leading handcrafted codecs and dominating recent learned-compression benchmarks. Wider adoption is still constrained by computational complexity and unresolved questions about losses, evaluation, and new data modalities.

  • Progress: Neural image compression progressed from barely beating JPEG 2000 in 2016 to outperforming the best known handcrafted codecs by 2018.In the 2021 Challenge on Learned Image Compression, classical codecs did not enter the top 10.
  • Progress: The leading codec in the Large Text Compression Benchmark relied solely on neural networks.This extends neural compression’s reported progress beyond image compression.
  • Open problems: Computational complexity remains a primary obstacle to wider adoption of neural compression.It is also a critical issue when applying neural methods to modalities such as point clouds and VR content.
  • Open problems: Open problems include designing loss functions and evaluation criteria and developing more efficient compression methods.These challenges are especially relevant to neural lossy compression.

176 Discussion and Open Problems

Neural compression faces open problems spanning deployment constraints, data representation, perceptual objectives, coding methods, and semantic reliability. Progress depends on meeting practical resource requirements while resolving unresolved design and evaluation questions.

  • Deployment constraints: Neural compression methods require substantially more computation than traditional codecs, creating barriers for latency-, power-, and hardware-constrained applications.The paper specifically identifies mobile video streaming as a setting where these constraints matter.
  • Representation and architecture: New data types require application-specific representations and neural architectures; naive 3D convolutional processing of point clouds can quickly exhaust memory.Point-cloud compression may use voxelization followed by nonlinear transform coding, but implementations must apply neural networks strategically.
  • Losses and perceptual quality: The appropriate objectives for perceptual realism remain poorly understood, and no loss is yet trusted to judge reconstruction quality reliably during training and evaluation.Open questions include the roles of divergences, adversarial approaches, crafted distortions, and no-reference metrics.
  • Coding methods: Quantization creates a training–testing performance mismatch whose effect on compression remains unclear; reverse channel coding is a promising but relatively recent alternative.Efficient coding schemes and their performance relative to quantization-based approaches remain open questions.
  • Semantic reliability: Highly realistic reconstructions at extremely low bit-rates can still misrepresent semantic content, creating a risk of miscommunication in semantically constrained domains.The paper also notes that stochastic reconstructions may differ across users and time.
Loading 2202.06533v3…