Source-linked AI summary
Transformer-based Image Compression
Ming Lu, Peiyao Guo, Huiqing Shi, Chuntong Cao, Zhan Ma
TL;DR
Image compression seeks compact representations with high reconstruction quality, while CNN-based learned methods face limitations in modeling long-range correlations and can require many parameters. TIC addresses this by combining Swin Transformer blocks, convolutional layers, and causal attention within a VAE framework. It achieves competitive or leading compression performance with substantially fewer parameters than existing learned approaches, and TIC+ surpasses VVC Intra by 2.6% BD-Rate improvement.
Problem
CNN-based learned image compression is limited in long-range correlation modeling, content adaptability, and parameter efficiency, motivating a more compact high-efficiency approach.
Method
TIC combines Swin Transformer blocks and convolutional layers in VAE encoder-decoder paths, using causal attention to fuse autoregressive neighbors and hyper priors.
Results
2.6% BD-Rate improvement enables TIC+ to surpass VVC Intra, while TIC requires only about half the parameters of Cheng et al.
Takeaways & Limitations
TIC provides leading image-compression performance with much fewer model parameters than existing learning-based approaches, supporting its attractiveness for practical applications.
Abstract
from arXiv · showhide
A Transformer-based Image Compression (TIC) approach is developed which reuses the canonical variational autoencoder (VAE) architecture with paired main and hyper encoder-decoders. Both main and hyper encoders are comprised of a sequence of neural transformation units (NTUs) to analyse and aggregate important information for more compact representation of input image, while the decoders mirror the encoder-side operations to generate pixel-domain image reconstruction from the compressed bitstream. Each NTU is consist of a Swin Transformer Block (STB) and a convolutional layer (Conv) to best embed both long-range and short-range information; In the meantime, a casual attention module (CAM) is devised for adaptive context modeling of latent features to utilize both hyper and autoregressive priors. The TIC rivals with state-of-the-art approaches including deep convolutional neural networks (CNNs) based learnt image coding (LIC) methods and handcrafted rules-based intra profile of recently-approved Versatile Video Coding (VVC) standard, and requires much less model parameters, e.g., up to 45% reduction to leading-performance LIC.
1 Introduction
The paper motivates TIC by limitations in CNN-based learned image compression and proposes combining Swin Transformer attention with convolution to improve information embedding. TIC retains the VAE-based compression framework while targeting competitive efficiency with fewer parameters.
- CNN filters characterize only short-range correlations, use fixed parameters across content distributions, and incur intensive element-by-element computation.
- Handcrafted attention can capture long-range correlations but often requires excessive parameters, while multi-model optimization requires caching multiple models.
- TIC combines Swin Transformer blocks with convolutional layers to embed long-range and short-range information within the existing VAE architecture.
- TIC uses hyper priors and autoregressive neighbors through a causal attention module for latent-feature context modeling.
- TIC achieves competitive compression efficiency with about half the model parameters of Cheng et al., while TIC+ can outperform VVC Intra with a slight parameter increase.
2 Related Work
Prior learned image compression replaces traditional transforms with trainable convolutional models and improves entropy modeling through hyper priors and autoregressive neighbors. TIC extends this direction by combining local convolution with nonlocal Transformer attention rather than relying only on handcrafted attention mechanisms.
- Learned image compression uses stacked convolutions as trainable transforms and later improves efficiency with hyper priors and autoregressive-neighbor context modeling.
- Handcrafted attention mechanisms guide encoding toward important areas, but TIC combines convolution and attention to jointly model short-range and long-range correlations.
- CNN locality limits information aggregation to a restricted receptive field, motivating vision Transformers for broader spatial dependencies.
- Swin Transformer uses window-based and relative-position attention and can process arbitrary-size high-resolution images.
- The paper migrates Swin Transformer into the image-compression pipeline to pursue better performance.
3 Method
TIC uses an end-to-end VAE compression pipeline whose encoders produce quantized latent symbols for entropy coding and whose decoders reconstruct the image. Its NTUs combine Swin Transformer blocks with convolution, while CAM fuses autoregressive and hyper-prior information for context prediction.
- The encoder transforms an image into latent features, quantizes them into symbols, and entropy-codes them; the decoder reconstructs the image from the compressed bitstream.
- The rate-distortion objective minimizes compressed bit rate plus λ-weighted reconstruction distortion, with λ adjusted for different bit rates.
- CNN-based learned compression stacks strided convolutional layers for compact bottleneck representations and commonly uses masked CNNs to fuse autoregressive neighbors with hyper priors.
- TIC constructs paired main and hyper encoder-decoder paths in a VAE, with each NTU applying a Swin Transformer block and convolutional layer.
- The Swin Transformer block computes window-based and shifted-window self-attention before remapping features, while convolution performs resolution scaling and local aggregation.
- CAM unfolds quantized features into patches, applies masked attention to preserve causality, and uses an MLP to fuse autoregressive neighbors with hyper priors.
- TIC+ increases the number of bottleneck-near Swin Transformer blocks in the main coder while keeping one block per hyper-coder NTU.
4 Results
The TIC delivers competitive image-compression efficiency with substantially fewer parameters, while TIC+ surpasses VVC Intra and reconstructions show sharper textures with less noise. Gradient visualizations further indicate more compact latent representations than a CNN-based model.
- Quantitative evaluation: TIC requires only half as many model parameters as Cheng et al. while maintaining competitive compression efficiency against leading LICs and VVC Intra.Evaluation uses rate-distortion curves and BD-Rate gains over HEVC Intra.
- Quantitative evaluation: 2.6% BD-Rate improvement enables TIC+ to surpass VVC Intra with only a slight increase in model parameters.TIC+ still consumes fewer parameters than Cheng et al.
- Qualitative evaluation: TIC reconstructions show sharper textures and less noise than the compared traditional and learned codecs.The subjective visualization agrees with the reported objective improvement.
- Model scaling: Adding more Swin Transformer Blocks in TIC+ is associated with additional coding gains, while the sufficient number for optimal efficiency remains open.The paper identifies this as a topic for future study.
- Latent-feature compactness: Gradient maps indicate that TIC produces a more compact latent representation than Cheng et al.’s CNN-based model.The maps visualize each input pixel’s contribution to a latent feature generated by the main encoder.
5 Conclusion
The paper presents a state-of-the-art image-compression method that combines Swin Transformers with convolutions to model short- and long-range information. It reports leading performance with substantially fewer parameters, making the method attractive for practical applications.
- Method: The method combines Swin Transformer Blocks and convolutional layers to aggregate short-range and long-range information into compact image representations.The conclusion identifies this combination as the basic processing unit.
- Conclusion: The method achieves leading performance against existing learning-based approaches and VVC Intra while consuming fewer model parameters.The conclusion characterizes the method as state of the art and highlights its parameter efficiency.