PaperlayerView exact revision on arXiv

Source-linked AI summary

Neural Image Compression with Text-guided Encoding for both Pixel-level and Perceptual Fidelity

Hagyeong Lee, Minkyu Kim, Jun-Hyuk Kim, Seungeon Kim, Dokwan Oh, Jaeho Lee

arXiv:2403.02944v2cs.CVcs.LG

TL;DR

Text-guided image compression can improve perceptual quality but commonly degrades pixel-wise fidelity, leaving open whether that trade-off can be mitigated. TACO uses text-adaptive encoding and joint image-text training; it outperforms image-compression baselines on LPIPS while maintaining competitive pixel-level fidelity.

  • Problem

    Text-guided compression methods commonly degrade PSNR, and it remains unknown whether perceptual gains can be achieved with minimal pixel-wise distortion.

    Source: §1 · p. 2, §2 · p. 4

  • Method

    TACO adds a text adapter to a conventional compression encoder and trains from scratch with a joint image-text loss.

    Source: §1 · p. 2, §3 · p. 4

  • Results

    Across datasets, TACO outperforms all image-compression baselines on LPIPS while keeping competitive pixel-level fidelity; realism gains occur on MS-COCO and Kodak, but not CLIC.

    Source: §1 · p. 2, §4.2 · p. 7

  • Takeaways & Limitations

    The authors conclude that using text to generate better codes is effective for high PSNR and perceptual quality, while text-guided decoding may be more challenging.

    Source: §5 · p. 9

    Text-handling encoding cost scales quadratically with sequence length and can become prohibitively long for very long inputs such as video transcripts.

    Source: §5 · p. 9

Abstract

from arXiv · show

Recent advances in text-guided image compression have shown great potential to enhance the perceptual quality of reconstructed images. These methods, however, tend to have significantly degraded pixel-wise fidelity, limiting their practicality. To fill this gap, we develop a new text-guided image compression algorithm that achieves both high perceptual and pixel-wise fidelity. In particular, we propose a compression framework that leverages text information mainly by text-adaptive encoding and training with joint image-text loss. By doing so, we avoid decoding based on text-guided generative models -- known for high generative diversity -- and effectively utilize the semantic information of text at a global level. Experimental results on various datasets show that our method can achieve high pixel-level and perceptual quality, with either human- or machine-generated captions. In particular, our method outperforms all baselines in terms of LPIPS, with some room for even more improvements when we use more carefully generated captions.

1. Introduction

Text-guided compression can improve perceptual quality, but commonly degrades pixel-wise fidelity, leaving open whether both can be achieved together. TACO uses text-adaptive encoding and joint image-text training, and reports stronger LPIPS performance while remaining competitive in pixel fidelity.

1. Introduction

  • Text-guided codecs can improve perceptual quality but commonly degrade PSNR, and whether that loss can be mitigated remains unknown.Diffusion-based codecs are reported to achieve 3–5 dB less PSNR than standard neural image-compression codecs.Source: §1 · p. 2
  • TACO uses text only for encoding, aiming to preserve perceptually meaningful information and inject semantics globally without a text-guided generative decoder.The authors suggest text's value at standard bitrates may lie more in its relationship to human perception than in efficient information storage.Source: §1 · p. 2
  • TACO injects text into a PSNR-oriented ELIC encoder through a CLIP-based adapter with bidirectional attention, then trains from scratch using joint image-text loss.The loss encourages reconstructions to be semantically aligned with the supplied text.Source: §1 · p. 2
  • TACO outperforms all image-compression baselines in LPIPS while remaining competitive in pixel-level fidelity and realism across human- and machine-captioned datasets.The reported pattern holds at standard rates of at least 0.1 bpp, and TACO also outperforms conventional LPIPS-focused text-guided baselines.Source: §1 · p. 2

2. Related Work

Image-compression research evaluates pixel distortion, perceptual similarity, and realism, with neural codecs and generative methods pursuing different quality tradeoffs. Text-guided methods improve realism or perceptual similarity, but their pixel-fidelity cost remains an open issue.

2. Related Work

  • Pixel distortion is commonly measured with MSE or PSNR, while perceptual metrics such as MS-SSIM, LPIPS, and PieAPP assess human-perceived similarity.Realism metrics such as FID and KID compare reconstructed-image feature distributions with those of natural images; CMMD is another proposed metric.Source: §2 · p. 3
  • Neural autoencoder codecs advanced pixel and perceptual fidelity, while GAN- and diffusion-based codecs target realism but tend to incur greater distortion.The rate-distortion-realism tradeoff has also been theoretically studied, with results suggesting both goals may not be achieved simultaneously at a fixed rate.Source: §2 · p. 3
  • Text-guided decoder methods target realism, especially at extremely low bitrates, while caption-conditioned codecs at conventional bitrates target perceptual distortion.Decoder approaches include pretrained diffusion models and scratch-trained GAN-based models that insert text through architectural modules.Source: §2 · p. 3
  • Text-guided compression methods have improved realism or perceptual similarity, but whether they can do so with only minimal PSNR degradation remains unanswered.Both text-guided decoder approaches and from-scratch codecs have been reported to trade pixel-level fidelity for perceptual quality.Source: §2 · p. 4

3. Method

TACO turns a PSNR-oriented ELIC codec into a text-guided codec by adding encoder-side text adaptation and training the full model with a joint image-text loss.

3.1. Background: Autoencoder-based Codec

  • ELIC is the PSNR-oriented backbone: its encoder maps an image to latent features, which are quantized and entropy-coded before the decoder reconstructs the image.The encoder where TACO injects text uses three residual bottleneck blocks, interleaved with convolutional layers and attention modules.Source: §3.1 · p. 4

3.2. TACO: Text-Adaptive Compression

  • The adapter embeds captions with frozen CLIP features, then uses three linear and three cross-attention layers to inject text into the encoder, including downsampled attention for more global influence.The text adapter first embeds the caption, then gradually processes and injects text features through multiple layers.Source: Figure 4 · p. 4, §3.2 · pp. 4–5
  • Training combines rate, MSE, LPIPS, and a CLIP-based joint loss aligning the reconstruction with its caption and original image; TACO omits adversarial loss for better PSNR.The joint loss combines text–reconstruction relevance with the CLIP embedding distance between original and reconstructed images.Source: §3.2 · p. 5

4. Experiments

The experiments compare TACO with conventional and text-guided codecs, then examine its components, caption inputs, and computational and memory costs. Across datasets, TACO leads the baselines in LPIPS while retaining competitive pixel-level fidelity.

4.1. Experimental Setup

  • Training uses 82,783 MS-COCO images with five human captions each; evaluation spans MS-COCO, CLIC, and Kodak, using OFA captions for captionless datasets.The evaluation uses 30,504 MS-COCO images, 428 CLIC images, and 24 Kodak images.Source: §4.1 · pp. 5–6

4.2. Results: vs. Image Compression Codecs

  • 0.025 LPIPS at 0.226 bpp beats HiFiC's 0.053 at 0.233 bpp; across datasets, TACO beats all LPIPS baselines and stays within 1 dB of ELIC/LIC-TCM PSNR.Realism results vary by dataset: TACO outperforms baselines on MS-COCO and Kodak, but not CLIC, where the gap tends to be small.Source: Figure 6 · p. 6, Figure 7 · p. 6, §4.2 · p. 7

4.3. Results: vs. Text-guided Decoding

  • On MS-COCO 40k, TACO outperforms Qin et al. (2023), including on LPIPS and FID; the comparison also reports TGIC on MS-COCO 1k.The baselines lack public code or checkpoints and use different evaluation setups, limiting direct comparison.Source: §4.3 · p. 7, Figure 8 · p. 7

4.4. Ablation Studies

  • Removing the text adapter worsens LPIPS by more than 4×; removing joint image-text loss degrades perceptual fidelity and realism with only a tiny PSNR degradation.The ablation results are shown in Figures 10 and 11.Source: §4.4 · p. 8, Figure 10 · p. 8, Figure 11 · p. 8

4.5. Further Analyses

  • TACO preserves caption semantics better than ELIC and LIC-TCM, but its comparison with MS-ILLM is inconclusive and slightly worse at slightly lower bpp.Text similarity is measured between machine-generated captions for original and reconstructed images.Source: §4.5 · p. 8, Table 1 · p. 9
  • TACO's encoding time is 7 ms longer than vanilla ELIC, but it remains faster than LIC-TCM and comparable with MS-ILLM.For high-resolution images, the table reports attention's increasing cost falling from +10.2% to +4.8%.Source: §4.5 · p. 9, Table 2 · p. 9
  • TACO adds 64.82M parameters to ELIC, reaching 101.75M total, about 80% fewer than MS-ILLM and HiFiC.The added parameters comprise 63.17M from CLIP and 1.65M from the text adapter; total parameters require about 400 MB of memory.Source: §4.5 · p. 9, Table 3 · p. 9

Q5. Can we use other captioning models?

5. Conclusion

The paper concludes that using text to generate better codes is a simple strategy for achieving high PSNR and perceptual quality, while text-guided decoding may be more challenging. TACO also has encoding-cost and training-data limitations.

5. Conclusion

  • Text-guided encoding is presented as a simple strategy for achieving high PSNR and perceptual quality, while text-guided decoding may be more challenging.The conclusion describes these as findings about utilizing auxiliary text when compressing images.Source: §5 · p. 9
  • Cross-attention makes text-handling encoding cost scale quadratically with sequence length, potentially making very long transcripts prohibitively slow.The paper gives video-like data with voice transcripts as an example.Source: §5 · p. 9
  • TACO training scalability is limited by the relative scarcity of image-text datasets compared with image-only or text-only datasets.The paper suggests generating image-text pairs with a captioning model as a possible baseline for future work.Source: §5 · p. 9

Impact Statement

The authors anticipate societal consequences from advancing machine learning and data compression but identify none for specific emphasis.

Impact Statement

  • The authors expect societal consequences from advancing machine learning and data compression, but identify no specific consequence to highlight.Source: Impact Statement · p. 10

A. Additional Experimental Details

The implementation uses an ELIC-based codec with a CLIP text encoder and 38-token captions. The reported baseline implementations include BPG 0.9.8 and VTM 19.2.

A. Additional Experimental Details

  • The TACO implementation uses an ELIC variant with intermediate and final feature dimensions of 192 and 320, respectively.Its CLIPTextModel uses openai/clip-vit-base-patch32; captions are set to 38 tokens, yielding embeddings of size R38x512.Source: A. Additional Experimental Details · p. 13
  • The experiments use BPG version 0.9.8 and VTM version 19.2; neural-codec baselines mostly use official checkpoints.Source: A. Additional Experimental Details · p. 13

B. Additional Metrics on Compression Results

The additional compression results report MS-SSIM, PieAPP, and CMMD on MS-COCO 30k, CLIC, and Kodak. Figure A1 organizes the metrics by dataset.

B. Additional Metrics on Compression Results

C. Additional Ablation Study

Freezing the fully image-trained ELIC base model was not beneficial in the tested setting.

C. Additional Ablation Study

D. Additional Qualitative Results

Additional visual comparisons show TACO's reconstructions alongside distinct artifacts in MS-ILLM and LIC-TCM.

D. Additional Qualitative Results

Loading 2403.02944v2…