PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Semantic Communications: Deep Learning-Based Image Semantic Coding

Danlan Huang, Feifei Gao, Xiaoming Tao, Qiyuan Du, Jianhua Lu

arXiv:2208.04094v1cs.CVcs.AI

TL;DR

Image semantic communications must preserve meaningful visual content while reducing the data burden of bandwidth-sensitive image transmission, beyond the pixel-level focus of existing approaches. The paper proposes RL-ASC, which represents images as semantic concepts, allocates bits adaptively using rate-semantic-perceptual criteria, and decodes them with an attention-based generative model. The method is reported to produce semantically consistent, natural reconstructions with reduced transmission data and robustness to AWGN noise.

  • Problem

    Existing image communication methods lack interpretation of image content, motivating semantic image coding that preserves semantic fidelity while reducing transmitted data.

  • Method

    RL-ASC extracts semantic concepts with a convolutional encoder, adaptively allocates quantization levels through reinforcement learning, and reconstructs images using an attention-based generative decoder.

  • Results

    RL-ASC produces reconstructions with high semantic similarity and naturalness, remarkably reduced transmission data, and robustness to noise in an AWGN channel.

  • Takeaways & Limitations

    The approach extends semantic coding beyond pixel-level image representation and can facilitate multiple vision tasks in semantic communication scenarios.

Abstract

from arXiv · show

Semantic communications has received growing interest since it can remarkably reduce the amount of data to be transmitted without missing critical information. Most existing works explore the semantic encoding and transmission for text and apply techniques in Natural Language Processing (NLP) to interpret the meaning of the text. In this paper, we conceive the semantic communications for image data that is much more richer in semantics and bandwidth sensitive. We propose an reinforcement learning based adaptive semantic coding (RL-ASC) approach that encodes images beyond pixel level. Firstly, we define the semantic concept of image data that includes the category, spatial arrangement, and visual feature as the representation unit, and propose a convolutional semantic encoder to extract semantic concepts. Secondly, we propose the image reconstruction criterion that evolves from the traditional pixel similarity to semantic similarity and perceptual performance. Thirdly, we design a novel RL-based semantic bit allocation model, whose reward is the increase in rate-semantic-perceptual performance after encoding a certain semantic concept with adaptive quantization level. Thus, the task-related information is preserved and reconstructed properly while less important data is discarded. Finally, we propose the Generative Adversarial Nets (GANs) based semantic decoder that fuses both locally and globally features via an attention module. Experimental results demonstrate that the proposed RL-ASC is noise robust and could reconstruct visually pleasant and semantic consistent image, and saves times of bit cost compared to standard codecs and other deep learning-based image codecs.

I. INTRODUCTION

Image semantic communications aims to reduce transmission data while preserving semantic fidelity, addressing the limits of pixel-level coding for bandwidth-sensitive visual data. The proposed RL-ASC approach encodes semantic concepts adaptively and reconstructs images using semantic and perceptual criteria.

  • Motivation: Image and video traffic creates substantial bandwidth pressure, motivating image semantic communications that reduce transmitted data without sacrificing semantic fidelity.The paper frames visual data as highly informative and identifies delay and congestion as transmission challenges.
  • Limitations of Existing Methods: Existing semantic image methods improve transmission under difficult channel conditions but generally lack interpretation of image content or remain tied to pixel-level reconstruction.The cited approaches include JSCC, semantic bit allocation, semantically structured coding, and direct semantic analysis from bitstreams.
  • Limitations of Existing Coding: Pixel-level codecs optimize rate-distortion and can introduce artifacts that degrade human perception and semantic analysis tasks.Reported artifacts include blocking, ringing, blurring, and checkerboard effects, with impacts on classification, detection, and segmentation.
  • Research Gap: Prior semantic similarity approaches tolerate pixel errors but are limited in supported tasks, while existing methods may fail to jointly satisfy semantic and perceptual performance at low bit rates.The paper highlights face-recognition limitation, pixel-level predictive reconstruction, and the need to encode regions according to differing semantic importance.
  • Proposed Approach: RL-ASC represents images with semantic concepts, optimizes rate-semantic-perceptual loss, allocates quantization adaptively, and uses an attention-based generative decoder.Semantic concepts contain category, visual feature, and spatial relation; the decoder combines global and category-wise local features.
  • Reported Outcomes: The proposed method is reported to reconstruct images with high semantic similarity and perceptual quality, reduce transmission data, support multiple vision tasks, and remain robust to AWGN noise.The decoder targets high perceptual quality under aggressive compression, while the conclusion reports reduced transmission data and noise robustness.

II. THE RL-BASED ADAPTIVE SEMANTIC CODING

RL-ASC represents images as semantic concepts rather than individual pixels, combining category masks and category-specific visual features. Its encoder extracts and classifies these concepts before adaptive coding and semantic reconstruction.

  • II. THE RL-BASED ADAPTIVE SEMANTIC CODING: The system contains a semantic encoder, a reinforcement-learning agent for adaptive quantization, and a semantic decoder.During inference, these modules are pretrained and fixed for a downstream task.
  • II. THE RL-BASED ADAPTIVE SEMANTIC CODING: The encoder extracts each object class, assigns it an optimal quantization level, and transmits the resulting semantic information at an aggressive bit rate.The decoder reconstructs the received semantic representation rather than reproducing the original pixels exactly.
  • A. Representation Unit: From Pixel to Semantic Concept: The semantic encoder obtains label maps with a segmentation network and extracts downscaled convolutional feature maps for the input image.The feature map is projected from N channels to n channels, with n < N, to discard perceptually redundant features.
  • A. Representation Unit: From Pixel to Semantic Concept: Category-specific feature maps are filtered using semantic masks and trained with a classification loss to preserve feature distinctiveness.Masked feature maps are pooled and classified through shared fully connected parameters.
  • A. Representation Unit: From Pixel to Semantic Concept: Semantic concepts combine category-specific features and spatial masks as the representation units for image coding.Each concept is formed from a category-specific feature map and its semantic mask.

B. Reconstruction Metric: From Pixel Loss to Semantic-Perceptual Loss

The paper replaces pixel-based reconstruction objectives with a rate-semantic-perceptual criterion. This criterion measures task-level semantic fidelity, perceptual quality, and transmitted bit rate together.

  • B. Reconstruction Metric: From Pixel Loss to Semantic-Perceptual Loss: The proposed reconstruction criterion jointly balances semantic fidelity, perceptual quality, and bit rate.Weighting parameters control the trade-off among the three terms, and changing λ produces different operating points.
  • B. Reconstruction Metric: From Pixel Loss to Semantic-Perceptual Loss: Semantic loss measures degradation in a downstream task's precision between the original and reconstructed images.The formulation can generalize to arbitrary user-defined semantic tasks through an accuracy-degradation function.
  • B. Reconstruction Metric: From Pixel Loss to Semantic-Perceptual Loss: For semantic segmentation, semantic loss is based on mean IoU, which averages intersection-over-union consistency across classes.The segmentation network produces predictions whose object-region overlap is evaluated using IoU.
  • B. Reconstruction Metric: From Pixel Loss to Semantic-Perceptual Loss: For classification and person re-identification, semantic loss uses cross-entropy-based objectives, with label smoothing applied for re-identification.The classification formulation compares predicted probabilities with ground-truth labels.
  • B. Reconstruction Metric: From Pixel Loss to Semantic-Perceptual Loss: Perceptual loss compares original and reconstructed images in a pretrained model's high-dimensional feature space to assess visual similarity.The paper motivates this metric by the importance of naturalness and clarity in reconstructed images.

C. The RL-based Semantic Bit Allocation Model

The RL-based bit allocation model treats sequential semantic coding as a Markov decision process. It assigns finer quantization and higher bit rates to more informative concepts while reducing allocation to less relevant regions.

  • C. The RL-based Semantic Bit Allocation Model: More informative and visually salient objects receive higher bit rates, while irrelevant regions receive lower bit rates.The design reflects the observation that downstream tasks emphasize only task-related objects.
  • C. The RL-based Semantic Bit Allocation Model: The RL agent sequentially selects quantization levels for semantic concepts by balancing bit cost against reconstruction performance.Each concept is encoded and decoded in label-index order within the MDP.
  • C. The RL-based Semantic Bit Allocation Model: The agent state contains the current feature map, semantic mask, concept index, and preceding reconstructed image.The action is a discrete quantization level, with finer levels producing higher bit rates and finer details.
  • C. The RL-based Semantic Bit Allocation Model: The immediate reward is the improvement in semantic similarity and perceptual quality together with a reduction in bitrate after updating one concept.Only the region corresponding to the current concept changes while other regions remain unchanged.
  • C. The RL-based Semantic Bit Allocation Model: The process initializes the whole image at the coarsest quantization level, then evaluates successive agent decisions using downstream and perceptual networks.The reconstructed image is updated sequentially and its rate-semantic-perceptual loss is recomputed after each concept.
  • C. The RL-based Semantic Bit Allocation Model: The policy maximizes discounted cumulative reward over complete trajectories using sampled rollouts and gradient ascent.The state transition is deterministic, and α denotes the learning rate in the update procedure.

III. SEMANTIC DECODER AND TRAINING DETAILS

The paper introduces soft quantization, entropy coding, and an attention-based semantic decoder as part of the decoder and training design.

  • III. SEMANTIC DECODER AND TRAINING DETAILS: The decoder and training section covers soft quantization, entropy coding, and the semantic decoder architecture.These components are presented as the remaining implementation and training details of RL-ASC.

A. Soft Quantization and Entropy Coding

RL-ASC adaptively quantizes semantic concepts and entropy-codes their representations, then uses a generative decoder to reconstruct images from semantic information rather than minimizing pixel error alone.

  • Soft Quantization: The RL agent selects quantization actions for each feature map to allocate bits adaptively and concentrate them on semantically important regions.
  • Soft Quantization: Nearest-neighbor quantization assigns each feature entry to a quantization center from the available levels.
  • Soft Quantization: Differentiable soft quantization addresses the zero-gradient problem that otherwise makes end-to-end gradient optimization ineffective.
  • Entropy Coding: Huffman coding converts quantized features into variable-length codes using predicted codeword frequencies, while the semantic label map is coded losslessly.
  • Generative Semantic Decoder: The generative semantic decoder combines class-specific local reconstructions with globally structured features through an attention-weighted sum.
  • Generative Semantic Decoder: Local and global generators preserve object details and spatial layout, while adversarial and perceptual objectives encourage natural, visually similar reconstructions.

C. Training Algorithm

Training proceeds in stages: first learning semantic encoding and decoding, then learning reinforcement-based bit allocation, and finally jointly fine-tuning the complete model.

  • Stage I: Stage I trains the semantic encoder and decoder end-to-end while initially assigning every semantic concept the highest quantization precision.
  • Stage I: The first stage alternates discriminator and generator updates while minimizing adversarial, perceptual, and feature classification losses.
  • Stage II: Stage II fixes the pretrained encoder and decoder, trains a randomly initialized RL agent, and rewards actions using changes in perceptual loss, semantic loss, and rate.
  • Stage II: The RL agent is optimized with a policy-gradient algorithm to maximize discounted cumulative rewards.
  • Stage III: Stage III jointly fine-tunes the semantic encoder, both generators, and the RL agent.

A. Simulations Setup

The evaluation uses Cityscapes urban street scenes and compares RL-ASC with engineered and learned image codecs across controlled bitrate ranges.

  • Dataset: RL-ASC is trained and evaluated on Cityscapes scene parsing and instance segmentation data from urban street scenes.
  • Baselines: The comparison includes JPEG2000, JPEG, BPG, DSSLIC, DSSLIC-finetuned, and HiFiC.

3) Evaluation Metrics:

The evaluation measures semantic fidelity, perceptual quality, and coding efficiency across bitrate settings, with results reported for downstream semantic segmentation.

  • Evaluation Metrics: PSNR and SSIM are described as shallow metrics that fail to capture many nuances of human perception, motivating semantic and perceptual evaluation.
  • Evaluation Metrics: Semantic performance is measured with downstream-task metrics, using mIoU as the objective metric for semantic segmentation.
  • Evaluation Metrics: FID and KID measure divergence between reconstructed and real-image distributions in Inception feature space, with lower KID indicating better performance.
  • Limitations: The Cityscapes dataset lacks ground-truth labels for object detection and image classification, preventing objective mAP measurement for object detection.
  • Results: RL-ASC outperforms JPEG, J2K, BPG, DSSLIC, and DSSLIC-finetuned in mIoU, while HiFiC approximates its performance but does not reach below 0.1 bpp.
  • Coding Efficiency: BD-mIoU measures average mIoU differences between rate-semantic curves, while BD-rate measures bitrate reduction at equivalent task-related accuracy.
  • Results: More than 60% average bitrate savings are achieved at the same mIoU compared with deep learning-based methods and DSSLIC-finetuned.

2) Subjective Quality:

At comparable bitrates, RL-ASC preserves semantic content and produces more natural reconstructions than conventional codecs, while adaptive bit allocation improves the rate-semantic-perceptual trade-off.

  • Codec comparison: Deep learning-based codecs DSSLIC and HiFiC outperform classic engineered codecs in semantic reconstruction quality.The comparison is made through reconstructed-image semantic labels and downstream visual tasks.
  • Downstream vision tasks: Classic JPEG, J2K, and BPG reconstructions lose semantic information at low bitrate, causing object misinterpretation and missed detections.The failures are associated with blocky, blurring, and ringing artifacts.
  • Downstream vision tasks: RL-ASC preserves major objects and semantic concepts, enabling downstream detection and segmentation to recognize and localize them accurately.Small objects such as traffic lights and overlapping cars remain detectable, and segmentation results resemble the ground truth.
  • Perceptual quality: RL-ASC is comparable with HiFiC and outperforms other baselines in perceptual performance.Its GAN-based decoder and adversarial loss are described as encouraging natural and realistic reconstructions.
  • Rate-semantic-perceptual performance: Adaptive semantic bit allocation improves perceptual performance over simplified RL-ASC at the same bitrate.The full method allocates precision selectively across semantic concepts rather than encoding all concepts equally.
  • Rate-semantic-perceptual performance: RL-ASC achieves lower semantic and perceptual loss than baselines at equal bitrate by a large margin.Semantic loss is defined as 1-mIoU, while perceptual loss combines weighted FID and KID values.

2) Subjective Quality:

RL-ASC prioritizes salient semantic concepts to retain visually and semantically meaningful content at low bitrate, accepting lower pixel-based quality while remaining robust to channel noise.

  • Visual reconstruction: At 0.16 bpp, RL-ASC produces a reconstruction almost indistinguishable from the input image.Its bits concentrate on salient semantic concepts compared with simplified RL-ASC.
  • Visual reconstruction: At low bitrate, baseline reconstructions exhibit blur, ringing, and blocky artifacts.These artifacts are visible in the comparative reconstruction examples.
  • Objective quality: Pixel-oriented baselines achieve better PSNR and SSIM, whereas RL-ASC tolerates pixel errors and does not enforce local consistency.The difference reflects optimization toward semantic rather than pixel-level fidelity.
  • Objective quality: RL-ASC outperforms simplified RL-ASC because complex objects receive higher precision while simpler image regions are encoded more economically.This comparison supports adaptive rather than equal-precision semantic coding.
  • Conclusion: The paper concludes that RL-ASC reconstructs images with high semantic similarity and naturalness, reduces transmitted data, and remains robust to AWGN noise.FID, KID, and mIoU are identified as useful tools for assessing perceptual preference and semantic exchange efficiency.
Loading 2208.04094v1…