PaperlayerView exact revision on arXiv

Source-linked AI summary

Wireless Deep Video Semantic Transmission

Sixian Wang, Jincheng Dai, Zijian Liang, Kai Niu, Zhongwei Si, Chao Dong, Xiaoqi Qin, Ping Zhang

arXiv:2205.13129v2cs.CVcs.IT

TL;DR

Wireless video transmission requires efficient end-to-end source-channel processing that accounts for source semantics and imperfect channels. DVST combines nonlinear transforms, conditional coding, temporal context, and variable-length deep JSCC to adapt transmission, achieving stronger rate-distortion performance and up to 50% channel-bandwidth savings versus classical coded schemes.

  • Problem

    Conventional JSCC schemes rely on handcrafted probabilistic designs with intractable optimization for complex sources and ignore the semantic aspects of source messages.

  • Method

    DVST combines nonlinear transforms and conditional coding with temporal semantic-feature context, using variable-length deep JSCC codecs to allocate channel bandwidth within and across video frames.

  • Results

    DVST generally surpasses traditional wireless video coded transmission schemes, saving up to 50% channel bandwidth at identical end-to-end performance and requiring 40% to 80% of H.264 + LDPC bandwidth at equal PSNR quality.

  • Takeaways & Limitations

    DVST supports both perceptual video reconstruction and machine vision tasks, while maintaining higher perceptual fidelity for human vision.

Abstract

from arXiv · show

In this paper, we design a new class of high-efficiency deep joint source-channel coding methods to achieve end-to-end video transmission over wireless channels. The proposed methods exploit nonlinear transform and conditional coding architecture to adaptively extract semantic features across video frames, and transmit semantic feature domain representations over wireless channels via deep joint source-channel coding. Our framework is collected under the name deep video semantic transmission (DVST). In particular, benefiting from the strong temporal prior provided by the feature domain context, the learned nonlinear transform function becomes temporally adaptive, resulting in a richer and more accurate entropy model guiding the transmission of current frame. Accordingly, a novel rate adaptive transmission mechanism is developed to customize deep joint source-channel coding for video sources. It learns to allocate the limited channel bandwidth within and among video frames to maximize the overall transmission performance. The whole DVST design is formulated as an optimization problem whose goal is to minimize the end-to-end transmission rate-distortion performance under perceptual quality metrics or machine vision task performance metrics. Across standard video source test sequences and various communication scenarios, experiments show that our DVST can generally surpass traditional wireless video coded transmission schemes. The proposed DVST framework can well support future semantic communications due to its video content-aware and machine vision task integration abilities.

I. INTRODUCTION

DVST addresses weaknesses of separated and existing deep JSCC video transmission by combining nonlinear transforms, conditional coding, and deep JSCC with temporal context. It adaptively models and transmits semantic features while optimizing end-to-end rate-distortion performance for perceptual and machine-vision objectives.

  • Motivation: Separated video transmission can suffer cliff effects under time-varying channels, while entropy coding is sensitive to transmission errors.These limitations motivate integrating source and channel processing for wireless video delivery.
  • Motivation: Existing deep JSCC performance degrades as source dimension increases and provides weaker coding gain than classical separated schemes.The paper attributes this poor coding gain to naively designed codec networks using highly integrated encoders for dimension reduction.
  • DVST framework: DVST combines nonlinear transform, conditional coding, and deep JSCC to extract and transmit video semantic features end to end.The framework uses temporal correlations among frames to encode the current frame through contextual processing.
  • Context-driven modeling: Temporal semantic-feature context makes the nonlinear transform temporally adaptive and produces a richer entropy model for allocating bandwidth to the current frame.The same context also provides prior information to the deep JSCC codec for reconstructing semantic feature maps.
  • Rate-adaptive transmission: Variable-length transmission lets DVST allocate limited channel bandwidth within and among video frames to improve overall performance.This mechanism is developed from the temporally adaptive entropy model and improves the coding gain of video deep JSCC.
  • Performance validation: DVST achieves better coding gain and rate-distortion performance across standard sequences and can save up to 50% channel bandwidth versus classical H.264/H.265-based transmission.It also supports machine-vision tasks while maintaining higher perceptual fidelity for human vision.

II. THE PROPOSED METHOD

DVST integrates contextual nonlinear transforms with deep JSCC for frame-by-frame wireless video transmission. Temporal semantic-feature and codeword contexts adapt encoding, decoding, and channel-resource use.

  • The Framework of DVST: DVST encodes each frame into a lower-dimensional semantic latent representation before generating variable-length continuous-valued channel inputs.The receiver reconstructs the corrupted signal and applies a contextual synthesis transform to recover the frame.
  • System Model: The system transmits video frame-by-frame over noisy channels, with AWGN modeled as additive Gaussian noise and other channel models supported through the transfer function.Video is organized into GOPs containing an intra-coded reference frame followed by predictive frames.
  • The Framework of DVST: DVST uses semantic-feature and codeword-domain temporal priors to make nonlinear transform and deep JSCC modules temporally adaptive.A rate-control module supports variable-length encoding, while the primary-link architecture is illustrated in Fig. 2.
  • The Framework of DVST: Contextual analysis and synthesis transforms condition current-frame processing on semantic features derived from reference frames.The learned analysis transform replaces handcrafted residual subtraction and selectively extracts features using current and reference frames.
  • The Framework of DVST: Contextual deep JSCC encoder and decoder condition codeword processing on reference feature maps and reconstructed codewords at transmitter and receiver.Transmitter contexts are simulated from reference data, while receiver contexts use buffered reconstructed frames and feature maps.

C. Entropy Model for Rate-Adaptive Transmission

DVST estimates latent-feature entropy using hierarchical, spatial, autoregressive, and temporal priors, then converts estimated entropy into adaptive channel bandwidth costs.

  • Entropy Model: The entropy model estimates each latent embedding’s distribution using hierarchical, spatial, and temporal context priors.The latent representation is variationally modeled with per-embedding Laplace distribution parameters.
  • Entropy Model: The per-embedding entropy is computed as the negative logarithm of its conditional probability given context, hyperprior, and preceding quantized embeddings.Backward adaptation is used because the model includes autoregressive computation over preceding dimensions.
  • Entropy Model: Uniform noise replaces quantization during training so the entropy model remains compatible with gradient-based optimization.The noisy latent uses offsets sampled uniformly from U(−1/2, 1/2), while testing uses quantized latent values.
  • Rate-Adaptive Transmission: The allocated channel bandwidth for embedding yt,i is kt,i = −ηt log P̄yt,i|čt,z̄t,ȳt,<i(ȳt,i|čt,z̄t,ȳt,<i).The scaling factor ηt maps embedding entropy to channel symbols and depends on channel capacity and codec efficiency.
  • Rate-Adaptive Transmission: η′t = 1 represents an ideal JSCC codec matching entropy-achieving source coding combined with capacity-achieving channel coding.The primary-link bandwidth cost is obtained by aggregating costs for transmitting semantic features.

D. Motion Transmission and Context Learning

DVST transmits motion information through a separate motion link and learns contexts in semantic-feature and deep-JSCC-codeword domains using motion estimation and compensation.

  • Context Learning: Motion estimation and compensation generate contexts in semantic-feature and deep-JSCC-codeword domains rather than the conventional source-pixel domain.The approach uses feature/codeword information density to form contexts for the primary link.
  • Motion Transmission: The motion link estimates motion vectors between the current frame and reconstructed reference frame, then transmits their semantic features without contextual conditioning.Its process copies the primary link while omitting context.
  • Motion Transmission: Motion-link bandwidth costs are allocated from a learned entropy model for motion-vector semantic features.The formulation parallels the primary-link entropy-based allocation.
  • Context Learning: At the transmitter, feature contexts are formed by extracting reference-frame features, warping them with reference motion vectors, and refining spatial discontinuities.The codeword context applies precoding to the reference semantic feature map before warping and refinement.
  • Context Learning: At the receiver, analogous context procedures use realistic received motion vectors and buffered reconstructed frames and feature maps.This preserves matching transmitter and receiver context construction around the wireless channel.

E. Optimization Goal

DVST optimizes channel bandwidth against end-to-end distortion while accounting for temporal dependencies within a group of pictures. Its distortion objective can combine reconstruction quality with downstream machine-vision task loss.

  • DVST minimizes a rate-distortion objective that uses channel bandwidth cost and end-to-end distortion.The loss at time t is formulated using the total bandwidth cost and distortion.
  • The trade-off between total channel bandwidth cost and distortion is controlled by λ.
  • Temporal correlations within a GOP let DVST allocate bandwidth both within individual frames and across frames.Conditional coding makes previous-frame performance affect subsequent frames during training.
  • For machine-type semantic communications, distortion combines reconstruction loss with downstream-task loss as Dt = Dt,rec + βDt,task.This supports video transmission for both human vision and machine analytics.

III. ARCHITECTURES AND IMPLEMENTATIONS

The implementation section details the DVST network architectures and introduces progressive training to support stable model learning.

  • DVST uses specified network architectures and a progressive training strategy for stable model learning.

A. Network Architectures

DVST combines contextual nonlinear transforms with contextual deep JSCC modules in its primary link, using motion vectors and temporal feature contexts to adapt transmission.

  • The primary link contains contextual nonlinear transform modules and contextual deep JSCC modules.The motion link has nearly the same architecture, except that motion vectors have two channels and contextual operations are removed.
  • Network Architectures: The semantic-domain context network extracts context through feature extraction, warping, and refinement before analysis transformation.The analysis transform concatenates the context with the current frame to learn a compact latent representation.
  • Network Architectures: The entropy model estimates spatially varying means and standard deviations using hierarchical, spatial, and temporal priors.The contextual synthesis transform mirrors the analysis transform while using a reconstructed context.
  • Contextual Deep JSCC: The contextual deep JSCC module uses codeword contexts to transmit the current latent representation at a variable rate based on estimated entropy.It assigns bandwidth costs to spatial positions, partitions fused features into patch embeddings, and quantizes their practical costs.
  • Contextual Deep JSCC: A shared backbone and on-demand fully connected layers implement variable-rate transmission without training 2^q separate deep JSCC networks.Different output dimensions are selected through a group of 2^q fully connected layers.

B. Progressive Training Strategy

DVST is trained progressively, first preparing the motion and primary links before jointly optimizing the complete framework. The final training integrates bandwidth allocation across links and frames.

  • Progressive Training Strategy: Training begins by pretraining nonlinear-transform components of the motion link, including motion estimation, gmv, and the entropy model.The lossless previous frame is used as the reference during this stage.
  • Progressive Training Strategy: The motion-link deep JSCC codec and rate-adaptation module are then jointly trained with nonlinear-transform components under wireless transmission error.Pretraining distortion terms are added to stabilize training.
  • Progressive Training Strategy: The primary link and context-generation networks are trained while the motion link remains frozen, using an ideal lossless previous frame as reference.Bit-cost terms are temporarily removed and later reintroduced to accelerate convergence.
  • Progressive Training Strategy: The whole framework is subsequently trained so the current frame is transmitted with help from context in JSCC codeword space.The primary-link bandwidth cost and end-to-end distortion are included in this stage.
  • Progressive Training Strategy: After unfreezing the motion link, final GOP training optimizes the bandwidth trade-off between primary and motion links.The integrated GOP loss also optimizes rate allocation among frames, while pretraining distortion acts as a regularizer for stability.

1) Datasets:

DVST is evaluated on standard video datasets and compared with classical coded transmission across reconstruction quality, channel conditions, bandwidth cost, and coding gain. Results generally favor DVST, with performance depending on resolution and scene texture.

  • Datasets: DVST is trained on Vimeo-90k and evaluated on HEVC and UVG sequences spanning varied resolutions, content, and frame rates.Training uses 89,800 Vimeo-90k clips and seven unrolled frames per GOP; evaluation covers HEVC Classes A–E and seven 1920 × 1080 UVG videos.
  • Reconstruction Results: At SNR = 10dB, DVST (PSNR) outperforms H.264 + LDPC across most sequences and CBRs, while approaching H.265 + Capacity on some sequences.The performance gap over H.264 + LDPC increases with CBR, indicating stronger coding gain.
  • Content Dependence: DVST’s coding gain generally increases with video resolution but is slightly worse than H.265 + LDPC on HEVC Classes C and E with complex textures.It performs better on Classes A and D, which have relatively flat foregrounds and simpler textures.
  • Reconstruction Results: DVST outperforms classical schemes on MS-SSIM, with larger improvements for high-resolution videos and high-CBR regions.MS-SSIM values are converted to dB for legibility, and the metric is described as more aligned with human perception.
  • Channel Robustness: DVST avoids catastrophic degradation when SNRtest < SNRtrain, whereas H.265 + LDPC exhibits a significant cliff effect.DVST also improves as SNRtest increases beyond SNRtrain under the evaluated CBR constraints.
  • Bandwidth Efficiency: DVST uses fewer channel resources at equal reconstruction quality, requiring only 40% to 80% of H.264 + LDPC bandwidth and saving bandwidth versus H.265 + LDPC in 21/25 videos.The H.264 comparison corresponds to PSNR at AWGN SNR = 10dB; savings are 60% to 20% relative to H.264 + LDPC.

C. Downstream Machine Vision Task Results

DVST is adapted for semantic segmentation while retaining signal-level reconstruction. On CamVid, it achieves better reconstruction and segmentation performance than separated coding schemes across evaluated CBRs.

  • Task-Oriented Design: DVST is extended for machine vision by concatenating an analytics model after transmission and optimizing reconstruction together with semantic segmentation.HyperSeg is used for segmentation, and a weighted bootstrapped cross-entropy term is added to the objective.
  • Experimental Setup: The CamVid evaluation uses 468 annotated training images and 233 test images resized to 256 × 192.CamVid provides four driving-scene video clips with densely annotated subsets.
  • Quantitative Results: DVST achieves better reconstruction and segmentation performance than separated coding schemes across various CBRs.The reported gap increases with the evaluated CBRs.
  • Qualitative Results: DVST reconstructions preserve more semantic information for machine recognition, yielding more accurate segmentation results.This qualitative comparison is shown against H.264 + LDPC and H.265 + LDPC reconstructions.

V. CONCLUSION

The paper concludes that DVST combines nonlinear transform, conditional coding, and variable-length deep JSCC to optimize wireless video transmission. Experiments show generally better performance than traditional coded transmission across perceptual and downstream-task metrics.

  • Conclusion: DVST adaptively extracts semantic features across frames and transmits them through learned variable-length deep JSCC codecs.Its optimization targets end-to-end rate-distortion performance under perceptual-quality or downstream-task metrics.
  • Conclusion: DVST generally surpasses traditional wireless video coded transmission schemes in extensive numerical experiments.The conclusion frames the method as a customized learning-based approach for end-to-end semantic communications.
Loading 2205.13129v2…