PaperlayerView exact revision on arXiv

Source-linked AI summary

Semantic-Aware Joint Source-Channel Optimization for Encoder-Agnostic Digital Video Communication

Xiangben Zhu, Caili Guo, Yang Yang, Chuanhong Liu, Meiyi Zhu

arXiv:2609.39296v1cs.LG

TL;DR

Deep-learning-based video semantic communication can demand substantial computation, while codec-compatible methods leave temporal semantics underexplored. This paper proposes a lightweight semantic-aware joint source–channel scheme using inter-frame importance and CSI to choose coding parameters. It reports BD-rate reductions of 34.86% and 18.01% over traditional and deep video encoders, respectively, and validates the scheme on a hardware testbed.

Abstract

from arXiv · show

Video semantic communication has attracted increasing attention as a promising approach to improving video transmission efficiency. However, most existing approaches rely on computationally intensive deep learning-based video encoders and decoders, which hinders their deployment in resource-constrained scenarios. To address this issue, we propose a lightweight semantic-aware joint source-channel optimization (SAJSCO) scheme that can be integrated into existing digital video communication systems as a plug-in module. Specifically, we develop a video communication system model in which the transmitter jointly optimizes source and channel coding parameters based on the inter-frame semantic importance of the input video and estimated channel state information. On this basis, we formulate an optimization problem that maximizes semantic importance weighted video reconstruction quality under a maximum bitrate constraint. To solve it, we first quantify inter-frame semantic importance using a cosine similarity-based metric with a shifted window mechanism. We then develop a multi-actor proximal policy optimization (MPPO) algorithm to solve the formulated problem by jointly adapting the source compression rate and channel coding rate. The learned policy can be directly applied to different video encoders without encoder-specific retraining or fine-tuning. SAJSCO achieves Bj{\o}ntegaard Delta rate reductions of 34.86\% and 18.01\% when integrated with H.265, a conventional video encoder, and DCVC-RT, a deep learning-based video encoder. Over-the-air experiments on a hardware testbed further demonstrate a PSNR gain of up to 1.448 dB with H.265 and an LPIPS reduction of up to 0.033 with DCVC-RT compared with the respective best-performing fixed-parameter baselines.

I. INTRODUCTION

Video semantic communication faces a deployment gap: deep-learning approaches are computationally demanding, while codec-compatible methods often emphasize spatial rather than temporal semantics. This paper proposes a lightweight scheme that uses inter-frame importance to guide joint source–channel coding.

A. Related Work

  • Existing deep-learning approaches jointly optimize source and channel coding but impose heavy computational burdens and often lack explicit adaptation to time-varying channels.Codec-compatible semantic methods offer greater infrastructure compatibility but mostly focus on spatial semantics, leaving inter-frame semantics underexplored.Source: §I · p. 1
  • Temporal semantics capture motion patterns and inter-frame dependencies relevant to prediction and redundancy elimination, but remain underexplored in codec-based semantic communication.Prior ROI-based work mainly prioritizes intra-frame semantic importance.Source: §I · p. 1, A. Related Work · p. 2

B. Challenges and Contributions

  • The paper identifies three challenges: real-time operation on existing infrastructure, measuring inter-frame importance under computational constraints, and combining that importance with real-time CSI for joint optimization.The motivation also notes that fluctuating SNR makes joint consideration of video semantics and channel state important.Source: B. Challenges and Contributions · p. 2
  • The proposed framework combines lightweight semantic analysis with existing digital video systems to optimize semantic-weighted reconstruction quality under a maximum bitrate constraint.Its stated goal is to address real-time deployment constraints while using inter-frame importance to guide coding.Source: B. Challenges and Contributions · p. 2
  • The authors propose a cosine-similarity method with shifted windows for local inter-GOP dependencies and a multi-actor reinforcement-learning algorithm for separate source- and channel-coding decisions.The framework is described as compatible with existing digital video communication systems.Source: B. Challenges and Contributions · p. 2

II. SYSTEM MODEL AND PROBLEM FORMULATION

The system extracts semantic features and GOP importance, combines importance with CSI to select coding parameters, and reconstructs the video at the receiver. It evaluates semantic-weighted quality while accounting for bitrate and channel conditions.

A. System model

  • A pretrained network extracts semantic features from GOPs, which the semantic-importance model uses to assign importance values.The input video is divided into GOPs before feature extraction.Source: A. System model · p. 3
  • The transmitter uses semantic importance and CSI to select a source-coding parameter controlling compression and a channel-coding parameter controlling error protection.The policy receives importance W and CSI C, and the selected parameters are applied GOP by GOP.Source: A. System model · pp. 3–4
  • The selected parameters are used to encode GOPs sequentially, and the video encoder may be a conventional H.26X codec or a deep neural network-based encoder.The resulting bitstream is transmitted over a wireless channel, then decoded by a channel decoder and a matched video decoder.Source: A. System model · pp. 3–4
  • CSI is estimated once per GOP from pilot signals; the system uses the estimated SNR to obtain the CSI associated with each timeslot.The channel coefficient is estimated by least squares from known pilot symbols.Source: A. System model · p. 4
  • Reconstruction quality is assessed using GOP-level MSE and PSNR, with pixel values defined for the original and reconstructed frames.PSNR also uses the maximum possible pixel value.Source: A. System model · p. 4
  • WPSNR weights GOP-level PSNR by semantic importance so that GOPs with higher importance contribute more to the semantic-aware reconstruction-quality measure.The paper contrasts this with conventional metrics that treat all GOPs equally.Source: A. System model · p. 4

B. Problem formulation

The problem formulation seeks the highest semantic-importance-weighted reconstruction quality within a maximum bitrate, and uses learning to address the difficulty of direct optimization.

B. Problem formulation

  • The optimization objective is to maximize overall weighted reconstruction quality subject to a maximum allowable bitrate.The source- and channel-encoding parameters are selected from their feasible sets.Source: B. Problem formulation · p. 4
  • Joint source–channel parameter selection is non-convex, with a rate-distortion tradeoff that depends on video importance and time-varying channel conditions.The mapping from coding parameters to reconstruction quality is highly nonlinear and has no explicit closed-form expression.Source: B. Problem formulation · p. 4
  • The authors use deep reinforcement learning to learn a joint source–channel optimization policy under varying video semantics and channel states.The approach is presented as a heuristic for the difficult optimization problem.Source: B. Problem formulation · p. 5

III. INTER-FRAME SEMANTIC IMPORTANCE EVALUATION

The method evaluates temporal semantic importance by comparing semantic features across adjacent GOPs.

III. INTER-FRAME SEMANTIC IMPORTANCE EVALUATION

  • The method evaluates each GOP’s temporal semantic importance using an inter-frame approach.It first extracts semantic features from video GOPs.Source: §III · p. 5
  • Semantic features are extracted from video GOPs before temporal importance is modeled.Source: §III · p. 5
  • Cosine similarity between adjacent GOP features is modeled to capture inter-frame relationships.Source: §III · p. 5

A. Shifted-Window Feature Construction

The method uses shifted local windows to capture temporal semantic dependencies among adjacent GOPs without redundant global computations. It measures GOP importance by comparing semantic features with neighboring GOPs, then uses that importance in joint source–channel optimization.

A. Shifted-Window Feature Construction

  • The shifted-window strategy centers a constant-size window around each target GOP to capture adjacent GOP dependencies while avoiding redundant global computation.The target GOP is positioned as close to the window center as possible.Source: A. Shifted-Window Feature Construction · p. 5

B. Cosine-Similarity-Based Importance Metric

IV. MPPO ALGORITHM FOR JOINT SOURCE–CHANNEL OPTIMIZATION

  • MPPO addresses joint source–channel optimization based on inter-frame semantic importance.Source: §IV · pp. 5–6

A. DRL-based Parameter Selection Algorithm

The method formulates sequential GOP encoding as an MDP and uses MPPO to choose source and channel coding parameters. Separate policies address the two action types, while the reward balances reconstruction quality and transmission cost.

A. DRL-based Parameter Selection Algorithm

1) Components of the MPPO Algorithm:

  • The state combines GOP semantic weight and normalized channel state, with binary flags indicating whether each exceeds its threshold.Source: 1) Components of the MPPO Algorithm · p. 6
  • The reward promotes reconstruction quality while penalizing transmission cost, encouraging actions that keep total bitrate within the constraint.The reward terms include penalties for decoding failure and transmitted data; its coefficients balance the terms and are reported as insensitive to dataset choice in practice.Source: 1) Components of the MPPO Algorithm · p. 6
  • Two actor networks learn separate source and channel policies, and the shared critic evaluates decisions through reward feedback.Source: 1) Components of the MPPO Algorithm · p. 6

2) Network Architecture:

The MPPO architecture uses two actors to select source and channel actions and a critic to evaluate decisions. Its training loop transmits and reconstructs video, evaluates WPSNR and reward, and updates the policies from sampled trajectories.

2) Network Architecture:

  • Two actor networks generate distinct source-coding and channel-coding actions, while one critic estimates the value of the current state.The actors share a structure, and the critic outputs a state value used to estimate decision quality.Source: 2) Network Architecture · pp. 6–7

3) Training Procedure:

  • Training applies the selected source and channel parameters, transmits and reconstructs the video, then uses WPSNR and reward feedback to update the policies.Source: 3) Training Procedure · p. 7
  • The policy update uses clipped probability ratios to constrain update size, alongside value-function loss and an entropy bonus.The clipping is described as a way to avoid excessively large policy updates and improve training stability.Source: 3) Training Procedure · p. 7
  • Iterative policy updates continue until convergence, learning a mapping from coding parameters to reconstructed video quality for the rate-distortion tradeoff.Source: 3) Training Procedure · pp. 7–8

B. Complexity Analysis

The complexity analysis separates semantic feature extraction, inter-frame importance evaluation, and MPPO decision-making. It gives the importance-evaluation cost in terms of window size, feature dimension, and GOP count.

B. Complexity Analysis

  • The framework’s total computational complexity comprises semantic feature extraction, inter-frame importance evaluation, and MPPO decision-making.Source: B. Complexity Analysis · p. 8
  • The analysis also accounts for feature extraction and MPPO decision costs, with MPPO cost parameterized by state and action dimensions and network-layer sizes.Source: B. Complexity Analysis · p. 8
  • For N GOPs, importance evaluation scales with the number of other GOPs in the window and semantic-feature dimension, with per-GOP cost O((M −1)df).The window size is M and the semantic-feature dimension is df.Source: B. Complexity Analysis · p. 8

V. SIMULATION RESULTS AND ANALYSIS

The section presents simulations to validate the SAJSCO scheme and MPPO algorithm.

V. SIMULATION RESULTS AND ANALYSIS

  • The simulations are intended to validate the efficacy of both SAJSCO and MPPO.The passage introduces the section's simulation studies.Source: §V · p. 8

A. Simulation Setup

The simulations use ActivityNet for MPPO training and testing, with HEVC Class D sequences also used for evaluation; RadioML2016.10a models time-varying channel conditions. In ablations, MPPO converges faster and achieves higher, more stable reward than the tested variants.

A. Simulation Setup

  • ActivityNet videos are split into training and test subsets at a 5:1 ratio; additional testing uses Class D HEVC sequences at 416 × 240 resolution.ActivityNet covers 203 activity categories and has variable spatial resolutions.Source: 1) Video Dataset · p. 8

2) Channel Environment:

  • RadioML2016.10a supplies 64 normalized SNR values remapped to 0–20 dB to model channel variation, with LDPC used for channel coding.The selected sequence's first 64 SNR values are used.Source: 2) Channel Environment · p. 8

3) Semantic Extraction Module:

  • The video semantic extractor uses MobileNetV2 features to represent frames, with inverted residual blocks, linear bottlenecks, and depthwise separable convolutions to reduce complexity.The extracted features are described as discriminative frame-level semantic features.Source: 3) Semantic Extraction Module · p. 8

4) Baselines:

  • The baselines include fixed-parameter H.265 and DCVC-RT, spatial-semantic SwinJSCC, and a single MPPO policy used with both encoders without encoder-specific retraining.SwinJSCC is evaluated on each GOP's first frame, with channel bandwidth ratio matched to SAJSCO.Source: 4) Baselines · p. 8
  • The action spaces use five source compression-rate levels and three LDPC rates; quality is measured with PSNR, WPSNR, LPIPS, and semantic-weighted WLPIPS.The three channel coding rates are 1/3, 1/2, and 2/3; modulation uses 16QAM.Source: 4) Baselines · pp. 8–9

B. Training Convergence and Ablation Study

  • MPPO converges faster and achieves higher, more stable reward than the ablated variants; removing the semantic-weight flag degrades training, and MPPO outperforms single-actor PPO.The authors attribute MPPO's advantage over SPPO to separate actors specializing in source compression and channel reliability while optimizing a shared reward.Source: B. Training Convergence and Ablation… · p. 9, Figure 4 · p. 9

C. Video Transmission Under Fixed CSI

Across fixed and time-varying channel settings, SAJSCO adapts source and channel coding parameters and improves performance over fixed-parameter baselines. On HEVC Class D, it achieves BD-rate reductions with both H.265 and DCVC-RT, while adding 4.36 ms latency per frame.

C. Video Transmission Under Fixed CSI

D. Video Transmission Under Time-Varying CSI

  • On ActivityNet with H.265 and RadioML, SAJSCO reduces PSNR BD-rate by 22.74%, 28.92%, and 52.87%, and LPIPS BD-rate by 17.69%, 15.73%, and 36.09% versus three non-semantic baselines.It also improves average PSNR over the overlapping bitrate range while reducing LPIPS.Source: D. Video Transmission Under… · p. 9, Figure 6 · p. 10
  • On HEVC Class D, SAJSCO reduces BD-rate versus the best non-semantic baseline by 34.86%, 37.58%, 4.12%, and 9.17% for H.265 (PSNR, WPSNR, LPIPS, WLPIPS).For DCVC-RT, the respective BD-rate reductions are 18.01%, 21.77%, 10.24%, and 16.41%.Source: D. Video Transmission Under… · p. 10, Figure 7 · p. 11

E. Complexity and Visualization

  • In the illustrated time-varying channel GOP, SAJSCO preserves more visual detail and improves reconstructed-frame quality for both H.265 and DCVC-RT.At 13.7 dB, DCVC-RT without SAJSCO has more severe visual distortion than H.265 under the stated bitrate setting.Source: E. Complexity and Visualization · p. 10, Figure 8 · p. 11
  • SAJSCO adds 4.36 ms latency per frame while maintaining real-time encoding capability, with a negligible increase in parameters compared with non-semantic baselines.The comparison covers MACs, parameters, and 1080p encoding and decoding speed.Source: E. Complexity and Visualization · pp. 10–11, Table II · p. 10
  • The policy selects more robust coding for semantically important GOPs or poor channels and more efficient compression when semantic importance is lower or channel conditions are favorable.Its state includes SNR and inter-frame semantic importance, while actions select source compression and channel coding parameters.Source: E. Complexity and Visualization · pp. 11–12, Figure 9 · p. 12

VI. PROTOTYPE VALIDATION

A two-USRP testbed evaluates H.265 and DCVC-RT over a wireless link. SAJSCO improves reconstruction metrics against the corresponding fixed-parameter baselines, supporting the simulation gains in physical transmission.

VI. PROTOTYPE VALIDATION

  • The prototype validation evaluates whether the proposed algorithm's gains extend beyond simulation to real-world communication hardware.The testbed uses two USRP devices as transmitter and receiver.Source: §VI · p. 12

A. Testbed Architecture

B. Experimental Results and Analysis

  • The over-the-air experiments use packet-based single-carrier BPSK at 5.5 GHz over a line-of-sight link, with transmit gain varied across four equal-duration segments.The first and last segments use 50 dB transmit gain, and the two intermediate segments use 60 dB; receive gain remains 56 dB.Source: B. Experimental Results and Analysis · p. 12
  • H.265+SAJSCO gains 1.448 dB PSNR and 4.695% decoding success rate over the best fixed H.265 baseline; DCVC+SAJSCO improves best LPIPS by 0.033 over its best fixed baseline.Across the experiments, SAJSCO improves reconstruction quality and decoding success rate for both encoder cases.Source: B. Experimental Results and Analysis · p. 12, Table IV · p. 13

VII. CONCLUSION

  • The conclusion reports BD-rate reductions of 34.86% with H.265 and 18.01% with DCVC-RT, alongside testbed results consistent with simulation improvements.SAJSCO jointly considers inter-frame semantic importance and channel state information and uses multi-actor PPO to adapt source and channel parameters.Source: §VII · pp. 12–13

Related papers

Loading 2609.39296v1…