Source-linked AI summary
Perceptual Image Compression with Cooperative Cross-Modal Side Information
Shiyu Qin, Bin Chen, Yujun Huang, Baoyi An, Tao Dai, Shu-Tao Xia
TL;DR
Image compression often transmits associated text separately, while prior distributed image-compression methods have not exploited cross-modal text to aid perceptual compression. The paper adds decoder-only text guidance using CLIP and semantic-spatial fusion. Experiments report improved perceptual quality and semantic relevance.
Problem
Prior distributed image-compression methods use image side information but do not exploit cross-modal text, despite text’s contextual value for image understanding and compression.
Source: §1 · pp. 1–2Method
The method uses decoder-only text side information with a CLIP text encoder and Semantic-Spatial Aware masks to guide pixel-level image enhancement.
Source: §1 · p. 2Results
Experiments report that textual side information enhances reconstructed-image perceptual quality and semantic relevance.
Source: §5 · p. 9Takeaways & Limitations
The paper presents text side information as an alternative way to compensate for semantic loss in deep image compression.
Source: §1 · p. 2
Abstract
from arXiv · showhide
The explosion of data has resulted in more and more associated text being transmitted along with images. Inspired by from distributed source coding, many works utilize image side information to enhance image compression. However, existing methods generally do not consider using text as side information to enhance perceptual compression of images, even though the benefits of multimodal synergy have been widely demonstrated in research. This begs the following question: How can we effectively transfer text-level semantic dependencies to help image compression, which is only available to the decoder? In this work, we propose a novel deep image compression method with text-guided side information to achieve a better rate-perception-distortion tradeoff. Specifically, we employ the CLIP text encoder and an effective Semantic-Spatial Aware block to fuse the text and image features. This is done by predicting a semantic mask to guide the learned text-adaptive affine transformation at the pixel level. Furthermore, we design a text-conditional generative adversarial networks to improve the perceptual quality of reconstructed images. Extensive experiments involving four datasets and ten image quality assessment metrics demonstrate that the proposed approach achieves superior results in terms of rate-perception trade-off and semantic distortion.
1. Introduction
Text is often transmitted separately from images despite its contextual value, and prior image-side-information methods do not exploit cross-modal text for perceptual compression. The paper proposes decoder-side text guidance and reports improvements in perceptual quality and semantic relevance.
1. Introduction
- Most practical scenarios encode images and text separately, although text provides contextual and descriptive information that complements image content.The introduction notes that text can aid image understanding and compression efficiency.Source: §1 · p. 1
- Existing distributed image-compression work uses correlated image side information, while the cited text-based approach incorporates text at both encoding and decoding stages.The introduction identifies the opportunity to exploit text’s semantic information as cross-modal side information for perceptual compression.Source: §1 · pp. 1–2
- Text-guided distributed image compression uses text only at the decoder, where it guides decompression and can compensate for semantic loss.The paper frames text-based side information as an alternative way to compensate for semantic loss in deep image compression.Source: §1 · p. 2
- The Semantic-Spatial Aware block uses semantic masks to identify image regions requiring text-based enhancement at the pixel level.The framework also incorporates the CLIP text encoder and a text-conditioned discriminator loss to preserve perceptual quality and semantic relevance.Source: §1 · p. 2
- Textual side information improves reconstructed-image perceptual quality and semantic relevance, with experiments also exploring varied text inputs and compression models.The paper reports evaluation from ten perspectives and describes applying different text inputs to the same image and across various compression models.Source: §1 · p. 2
2. Related Work
Related work traces advances in deep image compression and learned distributed source coding, then contrasts prior text-conditioned coding with this paper’s decoder-only setting. It identifies practical use of concise text side information for perceptual compression as an open need.
2. Related Work
- Deep image compression progressed from end-to-end rate-distortion optimization to hyperprior code-rate estimation and decoder-side GAN artifact reduction [1].The review also cites generative compression for high-fidelity reconstruction [21].Source: §2 · p. 2
- Learned distributed source coding uses aligned image side information [2] and multi-scale feature-domain patch matching [15], while Figure 3 contrasts joint image-text coding [16] with the proposed distributed setup.The figure caption identifies the two compared approaches as joint image-text coding and distributed coding with text side information.Source: §2 · p. 2, Figure 3 · p. 3
- Prior text-description coding places text in the encoder, entropy coding, and decoder, whereas this paper considers text available only at the decoder.The related-work passage motivates the decoder-only setting by citing distributed source coding’s theoretical equivalence in compression efficiency and its practical reduction of encoder complexity.Source: §2 · p. 3
3. Proposed Method
The method uses text available only at the decoder to guide image decompression, with an adversarial discriminator during training. Its SSA blocks apply text-conditioned, spatially localized feature transformations, while the training objective combines rate, distortion, and adversarial terms.
3.1. Overview
- The decoder combines compressed image features with text side information, while training uses a discriminator and decompression uses only the generator.The model also includes an encoder, entropy prior, and text feature extractor.Source: §3.1 · p. 3, Figure 4 · p. 4
3.2. Joint Training with Text Side Information
- Each SSA block uses text and previous-stage reconstruction features to predict a spatial mask that guides text-conditioned affine transformations.The mask indicates where features should be enhanced with textual information.Source: §3.2 · pp. 3–4
3.3. Objective Functions
- The encoder, prior, and generator objective combines code rate, distortion, and generator adversarial loss.The distortion term combines MSE and LPIPS; the rate terms measure the bit-rate of compressed features.Source: §3.3 · p. 4
- The discriminator is trained to assess both whether reconstructed images look real and whether images match their text descriptions.Its inputs include the reconstructed image, encoded feature representation, and text description.Source: §3.3 · p. 4
4. Experiments
Across four datasets and ten evaluation perspectives, the experiments report improvements in perceptual quality, fidelity, and semantic relevance, alongside analyses of text-guided bit allocation, text robustness, and a variant backbone.
4.1. Experimental Setup
- Experiments cover MS COCO, Kodak, CLIC2020, and CUB, using ten metrics spanning perceptual quality, fidelity, text relevance, diversity, human assessment, distortion, and classification.Source: §4.1 · pp. 4–5
4.2. Comparison with Other Methods
- Across MS COCO, Kodak, and CLIC2020, the method outperforms baselines on perceptual, fidelity, and semantic metrics while reducing distortion.Source: Figure 6 · p. 5, Figure 7 · p. 5, §4.2 · p. 6
- The method also improves classification accuracy and restores fine textures with less blocky blur in visual comparisons.Source: §4.2 · p. 6, Figure 9 · p. 6, Figure 11 · p. 8
4.3. Contributions of Text Side Information
- Text-guided masks steer bits toward emphasized ears, nose, and textures, while ablations indicate decoder-side text uses fewer bits than discriminator-side text.Source: §4.3 · pp. 6–8, Table 1 · p. 7
4.4. Stability Evaluation
- Different matched texts yield similar performance without changing bitrate, whereas mismatched text degrades perceptual metrics more than distortion metrics.Source: §4.4 · p. 8
4.5. Variant Backbones
- On Cheng2020, adding text improves fidelity, perceptual quality, semantic relevance, and classification accuracy.Source: §4.5 · p. 8, Figure 12 · p. 8
5. Conclusion
The conclusion reports that textual side information improves reconstructed-image perceptual quality and semantic relevance, and points to other signal types as a future direction.