PaperlayerView exact revision on arXiv

Source-linked AI summary

FLM: Frequency-Aware Language Models for Generative Image Compression

Jiarun Chen, Kejun Wu, Li Li, Chengtao Cai, Zhengguo Li, Chia-Wen Lin

arXiv:2608.28687v1cs.CV

TL;DR

Generative compression can improve low-bitrate efficiency but may reconstruct content that differs from the source. FLM instead models quantized DCT coefficients with an LLM for arithmetic coding, and reports superior rate-distortion performance across datasets while remaining applicable to lossless JPEG recompression.

  • Problem

    Generative compression can produce textures and semantic details that deviate from the source, while pixel-domain LLM coding spends bitrate on barely perceptible noise and textures.

  • Method

    FLM uses JPEG-style quantized DCT coefficients, macroblock tokenization, autoregressive next-coefficient prediction, arithmetic coding, and two-stage frequency-domain fine-tuning.

  • Results

    FLM achieves overall superior rate-distortion performance against conventional and generative codecs across Kodak, Tecnick, and CLIC2020, with competitive lossless JPEG recompression.

  • Takeaways & Limitations

    FLM combines deterministic frequency-domain reconstruction with LLM-based probabilistic modeling and supports both lossy compression and lossless JPEG recompression.

  • Takeaways & Limitations

    Autoregressive frequency-domain prediction requires slow computation and high computational and memory resources, limiting real-time image-transmission deployment.

Abstract

from arXiv · show

Generative models have significantly improved the performance ceiling of image lossy compression at low bitrates by exploiting learned priors. However, the generated textures and semantic details may deviate from the source content, thereby affecting the fidelity of image reconstruction. To solve these challenges, we propose FLM, a frequency-aware language model that improves compression efficiency through frequency-domain probabilistic modeling while retaining deterministic reconstruction. At the encoder, the input image is transformed into quantized DCT coefficients, which are organized into discrete sequences using macroblock-based coefficient tokenization. FLM then performs next-coefficient prediction to autoregressively estimate token-wise conditional probability distributions for arithmetic coding, thereby generating a compact bitstream. At the decoder, the LLM and arithmetic decoder jointly recover the frequency-domain data, followed by inverse transformations for image reconstruction. A task-specific frequency-domain dataset and a two-stage fine-tuning strategy are further developed to enable the model to operate across multiple bitrate settings. FLM is a versatile compressor that is compatible with both lossy compression and lossless JPEG recompression frameworks. Experiments show that FLM exceeds conventional and generative lossy compression methods in rate-distortion performance. FLM achieves BD-PSNR gains of 3.30 dB, 3.83 dB, and 3.80 dB than JPEG baseline on Kodak, Tecnick, and CLIC2020, respectively. Better qualitative quality of FLM can be achieved in improving semantically high fidelity and suppressing blocking artifacts. FLM is also validated to be applicable to the lossless recompression task with competitive performance.

I. INTRODUCTION

FLM addresses the fidelity limitations of generative compression and the bitrate limitations of conventional codecs by modeling quantized DCT coefficients with an LLM. Across multiple datasets, it reports superior rate-distortion performance and competitive lossless JPEG recompression.

  • Motivation: Generative compression can produce plausible textures and semantic details that differ from the source, making faithful reconstruction difficult.The paper identifies fidelity, rather than perceptual realism alone, as a central challenge.
  • Motivation: Existing LLM-based image compression mainly codes pixels or residuals losslessly, requiring high bitrates for barely perceptible noise and textures.Frequency-domain representations offer sparse, structured sequences through transform and quantization.
  • Approach: FLM combines JPEG-style DCT transform coding with LLM-based next-coefficient prediction for frequency-domain probabilistic modeling.The framework directly models conditional distributions of quantized DCT coefficient tokens.
  • Approach: Macroblock tokenization, range-constrained probability modeling, and task-specific two-stage fine-tuning support coefficient modeling across multiple bitrates.These components are designed to improve the LLM’s understanding and coding of frequency-domain coefficients.
  • Results: FLM achieves overall superior rate-distortion and qualitative quality against traditional and generative codecs, while also delivering competitive lossless JPEG recompression.Evaluations cover Kodak, Tecnick, and CLIC2020.

A. GAN-based image compression

The paper situates FLM among generative and LLM-based compression methods, including GAN, diffusion, and autoregressive approaches. These methods improve perceptual reconstruction or entropy modeling, but motivate FLM’s frequency-domain alternative.

  • GAN-based image compression: GAN-based compression uses adversarial training to improve perceptual quality and fine-detail reconstruction of decoded images.Early GAN compression results were unsatisfactory, while later work used more sophisticated encoder and decoder architectures.
  • Diffusion-based image compression: Diffusion-based compression encodes image representations in a diffusion model’s semantic space and reconstructs them through iterative denoising.The decoder entropy-decodes and dequantizes the latent representation before diffusion-based reconstruction.
  • LLM-based image compression: LLM-based lossless compression predicts pixels and supplies their probabilities to an arithmetic encoder, establishing a general entropy-modeling paradigm.Subsequent work extended this paradigm to image-trained models, residual coding, and medical images.

III. PROPOSED METHOD

FLM converts images into quantized DCT coefficient sequences and uses an LLM to predict their conditional distributions for arithmetic coding. Its tokenization and vocabulary design make frequency-domain coefficients suitable for autoregressive modeling.

  • Encoding framework: FLM applies a JPEG-style frontend that converts RGB images into quantized DCT coefficients before organizing them into one-dimensional sequences.The frontend includes color conversion, chroma subsampling, DCT, quantization, and zigzag scanning.
  • Decoding framework: The decoder uses matching model weights, tokenizer, and prompt template to autoregressively recover coefficient tokens from arithmetic-coded data.Side information constructs the task prompt used to predict the first and subsequent coefficient tokens.
  • Tokenization Strategy: The method uses macroblock-based coefficient tokenization and range-constrained probability modeling to organize sequences and restrict predictions to valid coefficient candidates.The resulting conditional distributions are passed to arithmetic coding for bitstream generation.
  • Tokenization Strategy: FLM addresses the absence of dedicated frequency-domain tokens by assigning one independent vocabulary token to each quantized DCT coefficient in [-1024, 1016].The newly introduced embeddings are optimized during subsequent training.
  • Tokenization Strategy: Subword mean-pooling initializes each new DCT-token embedding by averaging pretrained embeddings for the coefficient’s numerical subwords.This transfers numerical representations from the pretrained vocabulary into the new frequency-domain tokens.

2) Macroblock Organization:

FLM groups neighboring DCT coefficient blocks into macroblocks to provide broader spatial context, while exploiting coefficient sparsity and prompt metadata. The organization shortens sequences and supplies side information for prediction.

  • Macroblock Organization: Each color channel is partitioned into non-overlapping 8 × 8 blocks, each producing 64 frequency-domain coefficients.Independent block modeling would provide only limited spatial context.
  • Macroblock Organization: The macroblock strategy organizes spatially adjacent coefficient sub-blocks into sequences so the LLM can model dependencies beyond a single block.The paper’s evaluated configurations compare different macroblock sizes.
  • Macroblock Organization: The ⟨DCT EOB⟩ token replaces trailing zero runs in zigzag-scanned AC coefficients, shortening sequences while preserving exact coefficient reconstruction.Quantized DCT coefficients are especially sparse in high-frequency regions.
  • Macroblock Organization: The prompt template includes channel, macroblock dimensions, quality factor, and minimum and maximum DC and AC coefficient values.Coefficient-range statistics are collected at the encoder and transmitted to the decoder as side information.

C. Range-Constrained Probability Modeling

FLM constrains next-token probability estimation to coefficient values valid at each prediction position, improving arithmetic coding of frequency-domain tokens. The strategy uses separate DC and AC ranges and supports the model’s frequency-domain training.

  • Quality-factor dependence: At low quality factors, stronger quantization produces many zero and narrowly ranged high-frequency coefficients, making constrained prediction especially relevant.The coefficient distribution becomes easier to model as quantization strengthens, while unconstrained softmax still includes invalid candidates.
  • Range-constrained candidate sets: FLM dynamically restricts each prediction to a valid candidate-token set derived from macroblock-level coefficient ranges.Separate ranges are used for DC and AC coefficients because their numerical ranges and distributions differ.
  • Probability normalization: Tokens outside the valid set receive zero probability, while valid-token logits are normalized exclusively over that set before arithmetic coding.This prevents invalid vocabulary entries from consuming probability mass assigned to feasible coefficients.
  • Training support: A dedicated frequency-domain coefficient dataset and two-stage fine-tuning strategy improve the LLM’s ability to model these coefficient distributions.The training design complements range-constrained probability modeling within FLM.

1) Dataset:

FLM is trained with frequency-domain coefficient sequences generated from JPEG-style processing, using quality-factor-aware sampling and two-stage fine-tuning. The design targets one model that operates across multiple bitrates.

  • Dataset construction: The fine-tuning dataset is built from DIV2K images processed through FLM’s JPEG-style transform and macroblock-based DCT tokenization pipeline.The pipeline converts two-dimensional frequency coefficients into sequences suitable for language-model training.
  • Quality-factor sampling: Training samples are grouped into low-Q, medium-Q, and high-Q ranges to reflect quality-factor-dependent coefficient distributions and prediction difficulty.Higher quality factors produce more complex coefficient distributions.
  • Quality-factor sampling: The sampling allocation assigns 25%, 25%, and 50% of samples to low-Q, medium-Q, and high-Q groups, respectively.The larger high-Q share emphasizes the more difficult distributions.
  • Two-stage fine-tuning: Two-stage fine-tuning first adapts newly introduced DCT-token embeddings and output mappings before further model training.The first stage freezes the model backbone while training the token embedding and output projection layers.

1) Dataset:

FLM is evaluated on Kodak, Tecnick, and CLIC2020 against conventional and generative compression methods using objective rate-distortion and lossless-recompression measures. Its rate-distortion results are strongest overall against JPEG.

  • Datasets: The evaluation uses Kodak, Tecnick, and CLIC2020, with full-resolution Kodak images and standardized resizing and cropping for the other datasets.The datasets contain 24, 140, and 428 images, respectively.
  • Baselines: FLM is compared with JPEG, JPEG-XL, and generative methods including DiffEIC, Control-GIC, PerCo, Diff-ICMH, and OSCAR.The comparison covers both traditional and generative image compression approaches.
  • Metrics: PSNR, BD-Rate, and BD-PSNR assess reconstructed-image quality and overall rate-distortion performance, while BPP evaluates lossless JPEG recompression.Lossless recompression comparisons use the same quality factor Q and preserve the original quantized DCT coefficients.
  • Quantitative results: 52.82%, 53.47%, and 55.51% BD-Rate reductions versus JPEG are reported on Kodak, Tecnick, and CLIC2020, respectively.FLM’s rate-distortion curve generally lies above and to the left of competing methods on Kodak and CLIC2020, and remains competitive on Tecnick.

2) Qualitative Comparison:

FLM preserves fine semantic details more faithfully than competing reconstructions at comparable bitrates, but its overall subjective perceptual quality is lower than generative methods. The paper attributes this trade-off partly to fixed JPEG quantization tables.

  • Subjective comparison: At comparable BPP levels, FLM better preserves fine semantic details and numerical information than the other compared methods.The qualitative comparison specifically reports clearer recovery of numerical information on the sailboat.
  • Subjective comparison: Other methods show blurred character boundaries and missing numerical semantics in the enlarged local details.These differences are reported in the comparisons for Kodak reconstructions.
  • Quality trade-off: FLM’s overall subjective perceptual quality is inferior to generative compression methods despite its stronger semantic-detail fidelity.The paper distinguishes semantic fidelity of fine details from overall perceptual quality.
  • Future improvements: Future work may use learnable quantization tables or end-to-end rate-distortion optimization to improve reconstructed-image subjective quality.The proposed directions address the limitation that FLM directly uses standard JPEG quantization tables without jointly optimizing quantization parameters.

C. Lossless Recompression Performance Comparison

FLM is evaluated for lossless JPEG recompression across multiple quality factors and preserves identical reconstructions while comparing bitrate efficiency. It outperforms several existing methods and remains comparable to the strongest cited approach.

  • FLM outperforms Lepton, LLJPEG, and CMIX in lossless JPEG recompression while achieving performance comparable to Xiang et al.All methods preserve the original quantized DCT coefficients, so comparisons use BPP under identical reconstruction results.
  • The lossless recompression evaluation compares methods at quality factors Q ∈ {65, 75, 85, 95}.
  • The ablation study evaluates coding performance on Kodak at Q ∈ {5, 35, 55, 75}.It examines fine-tuning strategy, macroblock size, and functional modules.
  • The common ablation baseline is the non-fine-tuned model with a 16 × 8 macroblock size.This baseline is repeated in the first row of each experimental group for consistent comparison.

1) Fine-Tuning:

Fine-tuning substantially improves FLM’s coding efficiency, while macroblock size and module choices materially affect bitrate. The method combines frequency-domain tokenization, next-coefficient prediction, arithmetic coding, and a two-stage fine-tuning strategy.

  • Fine-Tuning: 57.603% average bitrate saving is achieved after fine-tuning across all quality factors.The improvement is attributed to better adaptation to DCT coefficient distributions and more accurate target-token probability estimation.
  • Macroblock Size: The 8×4 and 1×1 macroblocks incur average bitrate overheads of 5.181% and 93.512%, respectively, versus the 16 × 8 baseline.The results suggest that larger macroblocks provide richer spatial context for modeling adjacent coefficient sub-blocks.
  • Module Design: Removing the prompt causes a 3.338% average bitrate penalty, while removing Range Constraint causes a 31.590% penalty.The prompt supplies channel, quality-factor, and coefficient-range information; Range Constraint excludes invalid candidate tokens.
  • FLM combines DCT-based frequency transformation, macroblock coefficient tokenization, LLM next-coefficient prediction, arithmetic coding, and two-stage fine-tuning.The decoder autoregressively recovers DCT coefficient sequences to reconstruct images.
Loading 2608.28687v1…