PaperlayerView exact revision on arXiv

Source-linked AI summary

GAN-Based Semantic Communication for Image Transmission in IoV

Ruixing Ren, Shan Chen, Junhui Zhao, Xiaoke Sun

arXiv:2608.27989v1cs.CVeess.SY

TL;DR

IoV cooperative perception needs efficient, reliable visual transmission under limited bandwidth and dynamic channels while preserving safety-relevant semantics. The paper proposes a priority-aware GAN semantic-communication framework with semantic encoding and coarse-to-fine reconstruction, achieving improved segmentation and low-rate image quality with channel robustness on Cityscapes.

  • Problem

    Low bit rates, high noise, changing scenes, and latency constraints make semantic fidelity and task-compliant image recovery difficult for visual semantic communication in IoV.

  • Method

    The framework extracts semantic label maps, prioritizes safety-critical categories during encoding, and reconstructs images with a two-stage GAN decoder using multi-scale discrimination and additional temporal and spatial mechanisms.

  • Results

    The proposed method outperformed UNet, FCN, and other baselines in mIoU, produced visually pleasing low-rate images, and showed robustness across AWGN and Rayleigh channels.

  • Takeaways & Limitations

    Priority-aware semantic transmission and GAN reconstruction support compact, safety-focused visual communication for IoV cooperative perception under constrained channels.

Abstract

from arXiv · show

For cooperative perception in the internet of vehicles, this paper proposes a generative adversarial network-based semantic communication framework to address the efficiency and fidelity bottlenecks of traditional communication systems in visual data transmission under limited bandwidth and dynamic channel conditions. At the transmitter, the framework adopts a pyramid attention network to extract semantic label maps and introduces a semantic priority preservation mechanism. It assigns differentiated weights to distinct semantic categories based on driving safety, guiding bit allocation and loss function design. At the receiver, an image reconstruction module integrating a coarse to-fine multi-resolution generator and multi-scale discriminator is designed. Combined with the temporal consistency branch, spatial pyramid pooling and class-aware convolutional layers, it achieves high-fidelity reconstruction of high-quality images from corrupted semantic labels. The model is trained with combined adversarial, feature matching and perceptual losses, effectively improving semantic consistency and visual realism of generated images. Experimental results on the Cityscapes dataset show that the proposed method outperforms existing counterparts in both semantic segmentation accuracy and reconstructed image quality, and maintains stable reconstruction performance under AWGN and Rayleigh channels.

I. INTRODUCTION

IoV visual communication must transmit task-relevant information efficiently and reliably under limited bandwidth, congestion, latency constraints, and dynamic channels. The paper addresses these challenges with semantic prioritization and GAN-based reconstruction.

  • Motivation: IoV cooperative perception requires real-time sharing of massive visual information for tasks including safe driving.
  • Motivation: Semantic communication transmits inherent meaning rather than complete waveforms, reducing transmission volume while preserving semantic fidelity.
  • Motivation: Low bit rates and high noise make preserving image semantic fidelity and recovering task-compliant images a central visual semantic-communication bottleneck.
  • IoV Challenges: IoV requires compact representations because rapidly changing scenes and strict latency constraints limit pixel-level transmission and retransmission.
  • Proposed Response: The framework uses semantic label maps, priority preservation, temporal consistency, and GAN decoding to address compactness, safety, inter-frame variation, and harsh channels.
  • Proposed Response: The encoder assigns smaller quantization steps, greater coding redundancy, and higher segmentation penalties to high-priority safety-critical regions.
  • Proposed Response: A two-stage cascaded generator restores global layout and texture details progressively, supported by multi-scale discrimination and IoV-specific reconstruction modules.
  • Evaluation: Cityscapes experiments evaluate segmentation accuracy, reconstructed image quality, and robustness under AWGN and Rayleigh channels.

II. SYSTEM MODEL

The system encodes camera images into semantic labels, transmits their latent representation through a wireless channel, decodes the labels, and generates a reconstructed image. Its objective is semantic rather than pixel-level distortion, with simulations modeling AWGN and Rayleigh fading.

  • System Pipeline: The IoV system has an encoder that extracts and transmits semantic label maps and a decoder that recovers labels and reconstructs images.
  • Transmitter: The semantic encoder P maps image I to label map S, while Wenc maps S to an n-dimensional latent representation z.
  • Wireless Channel: The wireless channel transforms z into received signal z̃; AWGN adds Gaussian noise, whereas Rayleigh fading scales z elementwise before noise.
  • Receiver: The decoder Wdec recovers semantic label map Ŝ, and generator G maps Ŝ to reconstructed image Î.
  • Simulation Assumptions: The simulated chain uses priority-aware allocation, coding, modulation, all-bit-per-pixel accounting, i.i.d. Rayleigh fading, perfect CSI, and receiver equalization.
  • Objective: The objective is minimizing semantic distortion between the original and reconstructed images rather than conventional pixel-level distortion.

III. PROPOSED SCHEME

The proposed scheme combines a semantic segmentation encoder with a semantics-aware image reconstruction decoder.

  • III. PROPOSED SCHEME: The proposed scheme consists of a semantic segmentation encoder at the transmitter and a semantics-aware image reconstruction decoder at the receiver.

A. Semantic Segmentation Encoder

The encoder converts images into discrete semantic label maps and protects categories according to their importance for driving safety. Priority weighting guides both transmission protection and semantic-loss optimization, with empirical gains in safety-critical reconstruction fidelity.

  • Semantic representation: Semantic label maps provide a discrete, low-dimensional, interpretable representation that retains task-relevant scene structure while omitting pixel-level details.Their low entropy supports transmission with few bits, and their discrete form permits unequal protection through classical channel codes.
  • Segmentation encoder: The pyramid attention network uses ResNet-50 with FPA and GAU modules to extract semantic label maps from raw images.
  • Priority preservation: Safety-critical categories receive more bits and enhanced protection during encoding and transmission than less consequential scene categories.The mechanism reflects the greater driving-safety impact of errors involving pedestrians and vehicles compared with degraded sky or building reconstruction.
  • Priority preservation: Priority levels assign category-dependent weights that directly guide quantization steps and channel-coding redundancy.The implementation uses three levels: α(1) = 1.0 for safety-critical categories, α(2) = 0.8 for navigation-relevant categories, and α(3) = 0.5 for background categories.
  • Priority-weighted training: The weighted semantic loss applies priority-weighted cross-entropy to emphasize classification accuracy in high-priority regions.The loss uses the difference between original and reconstructed semantic labels, with D defined as cross-entropy.
  • Priority-weighted training: Empirical results verify that the specified priority weighting improves reconstruction fidelity in safety-critical regions.

B. Image Reconstruction Decoder

The decoder reconstructs photorealistic images from recovered semantic label maps using a coarse-to-fine generator and multi-scale discrimination. Temporal consistency, scale-adaptive pooling, class-aware convolutions, and auxiliary losses target fidelity, realism, and safety-critical details.

  • Generator design: The coarse-to-fine generator restores global scene layout at low resolution before refining local textures and edges at full resolution.The two cascaded subnetworks balance receptive field, resolution, reconstruction quality, and computational cost.
  • Generator design: The global generator G1 processes a 2× downsampled semantic map into a 1024 × 512 image using larger 5 × 5 kernels.Its larger receptive field targets overall scene layout.
  • Generator design: The local enhancer G2 combines G1 output with front-end features to produce a 2048×1024 image focused on texture and edge reconstruction.It uses smaller 3×3 kernels and can be extended to 4096 × 2048 after enhancement.
  • Discriminator design: Two discriminators evaluate original- and downsampled-resolution images, jointly providing fine-detail and global-consistency feedback.D1 emphasizes details such as wheel spokes and traffic signal text, while D2 evaluates road continuity and vehicle proportion.
  • Discriminator design: The discriminator is adapted for video scenes through temporal consistency, multi-scale pooling, and class-aware weighting of pedestrians and vehicles.Temporal consistency suppresses flicker in high-correlation regions, while low-correlation regions rely on the current frame to avoid motion blur.
  • Loss design: Feature matching and perceptual losses supplement adversarial training by aligning discriminator features and high-level VGG features between real and generated images.The perceptual loss emphasizes semantic content rather than exact pixel-level matching.

IV. SIMULATION EXPERIMENTS AND ANALYSIS

Experiments on Cityscapes evaluate segmentation, reconstruction quality, rate-distortion behavior, loss convergence, and channel robustness. The proposed system improves safety-critical segmentation and low-rate semantic reconstruction, while trading off pixel-level PSNR against semantic and perceptual fidelity.

  • The evaluation uses 2975 Cityscapes training images and 1525 test images, with mIoU, key-category accuracy, PSNR, and SSIM.
  • Semantic segmentation and priority ablation: The priority mechanism improves vulnerable-class mIoU from 0.181 to 0.208 (+2.7 percentage points), while overall mIoU decreases from 0.266 to 0.257.The safety-class mIoU remains comparable, changing from 0.259 to 0.255.
  • Semantic segmentation and priority ablation: PAN produces semantic label maps more consistent with the real scene and supports stronger object recognition and localization than the compared models.Other models show block, blurring, and ringing artifacts under low bit rates.
  • Image reconstruction: The proposed reconstructions remain highly similar to inputs with favorable quality consistency, whereas other models are susceptible to blurring and block artifacts at low bit rates.The discriminator receives 39 conditional input channels: 3 RGB, 35 one-hot semantic-label, and 1 instance-map channel.
  • Training behavior: The adversarial loss declines gradually, feature matching and perceptual losses converge rapidly, and discriminator loss remains stable during training.These trends indicate optimization at both feature and semantic levels while maintaining discrimination between real and generated samples.
  • Rate-distortion and channel robustness: In the 0.1–0.3 bpp range, the proposed method achieves higher PSNR than JPEG, J2K, and CRN, while BPG performs better across the evaluated range.The method’s advantage is reported as semantic fidelity and robustness at low bit rates and under channel impairments, not maximum pixel-level PSNR.
  • Rate-distortion and channel robustness: Under AWGN, reconstructed-image PSNR improves with SNR; under Rayleigh fading, PSNR is lower overall but reconstruction quality remains more stable.The proposed scheme produces visually acceptable images even at low SNR by compensating for missing information.

V. CONCLUSION

The paper concludes that its GAN-based semantic communication framework combines priority-aware semantic segmentation with coarse-to-fine reconstruction for IoV cooperative perception. Cityscapes experiments report stronger segmentation, visually pleasing low-rate images, and stable performance across channel conditions.

  • The framework uses PAN semantic segmentation, semantic priority preservation, a multi-resolution generator, a multi-scale discriminator, and adversarial, feature matching, and perceptual losses.
  • On Cityscapes, PAN outperforms UNet, FCN, and other baselines in mIoU, while the GAN reconstruction method produces visually pleasing images at low bit rates.
  • Robustness tests under various channel conditions show that the proposed scheme maintains stable reconstruction performance.
Loading 2608.27989v1…