Source-linked AI summary
DeepWiVe: Deep-Learning-Aided Wireless Video Transmission
Tze-Yang Tung, Deniz Gündüz
TL;DR
Wireless video transmission must handle unpredictable channel conditions and the cliff-effect of separation-based designs. DeepWiVe uses end-to-end DNN-based JSCC with residual prediction and RL-based dynamic bandwidth allocation, and reports superior performance to H.264 + LDPC and H.265 + LDPC under supported metrics and conditions.
Problem
Separation-based wireless video systems suffer cliff-effect degradation under channel mismatch, while prior deep-learning video compression treats communication as a perfect bit pipe.
Method
DeepWiVe directly maps video signals to channel symbols with an end-to-end DNN transform, predicts residuals without distortion feedback, and uses RL to allocate bandwidth among frames.
Results
DeepWiVe beats H.264 with LDPC codes in all tested channel conditions and outperforms H.265 under MS-SSIM evaluation, while achieving graceful degradation with channel quality.
Takeaways & Limitations
The results support end-to-end JSCC as a wireless video transmission design that avoids the cliff-effect and dynamically allocates constrained bandwidth among frames.
Abstract
from arXiv · showhide
We present DeepWiVe, the first-ever end-to-end joint source-channel coding (JSCC) video transmission scheme that leverages the power of deep neural networks (DNNs) to directly map video signals to channel symbols, combining video compression, channel coding, and modulation steps into a single neural transform. Our DNN decoder predicts residuals without distortion feedback, which improves video quality by accounting for occlusion/disocclusion and camera movements. We simultaneously train different bandwidth allocation networks for the frames to allow variable bandwidth transmission. Then, we train a bandwidth allocation network using reinforcement learning (RL) that optimizes the allocation of limited available channel bandwidth among video frames to maximize overall visual quality. Our results show that DeepWiVe can overcome the cliff-effect, which is prevalent in conventional separation-based digital communication schemes, and achieve graceful degradation with the mismatch between the estimated and actual channel qualities. DeepWiVe outperforms H.264 video compression followed by low-density parity check (LDPC) codes in all channel conditions by up to 0.0462 on average in terms of the multi-scale structural similarity index measure (MS-SSIM), while beating H.265 + LDPC by up to 0.0058 on average. We also illustrate the importance of optimizing bandwidth allocation in JSCC video transmission by showing that our optimal bandwidth allocation policy is superior to the naïve uniform allocation. We believe this is an important step towards fulfilling the potential of an end-to-end optimized JSCC wireless video transmission system that is superior to the current separation-based designs.
I. INTRODUCTION
Separation-based wireless video systems face cliff-effect failures under channel degradation, motivating end-to-end JSCC. DeepWiVe directly maps video to channel symbols, predicts residuals without distortion feedback, and dynamically allocates bandwidth with RL.
- Separation-based video delivery divides transmission into source compression and channel coding, preserving modularity but limiting joint optimization.
- Unpredictable channels, ultra-low latency requirements, and energy-limited devices expose limitations of separation-based designs in applications such as wireless VR and drone surveillance.
- The separation-based cliff-effect makes source information irrecoverable when channel conditions fall below the anticipated level, causing abrupt performance deterioration.
- A DNN predicts residuals without distortion feedback, addressing residual uncertainty caused by channel conditions during JSCC transmission.
- DeepWiVe directly maps video sequences to channel vectors under a GoP bandwidth constraint, jointly performing compression and communication in an end-to-end JSCC design.
- RL trains a bandwidth allocation network to optimize bandwidth among video frames, while different allocation networks enable variable bandwidth transmission.
- DeepWiVe beats H.264 with LDPC codes in tested channel conditions and outperforms H.265 under MS-SSIM evaluation while avoiding the cliff-effect.
II. RELATED WORK
Prior JSCC video approaches include layered, hybrid digital-analog, and scalable coding designs, but reported limitations include weaker video quality, limited bandwidth exploitation, and insufficient dynamic adaptation. DeepWiVe addresses video-specific challenges involving inter-frame redundancy and resource allocation.
- Earlier JSCC video work used hierarchical layers, scalable video coding, and unequal error protection to serve receivers with different channel capacities.
- SoftCast directly maps pixels to channel symbols using a hybrid digital-analog design that exploits frequency-domain sparsity.
- Although prior hybrid digital-analog methods overcome the cliff-effect, they are not competitive with separation-based schemes in video quality and cannot dynamically exploit bandwidth or adapt to channel and network conditions.
- Deep-learning video-compression studies reported competitive or superior compression against H.264/5 but treated communication as a perfect bit pipe and did not address wireless delivery.
- Video JSCC differs from image transmission because it must exploit inter-frame redundancies and optimize resource allocation across frames.
III. PROBLEM FORMULATION
The paper formulates wireless video transmission as reconstructing GoPs from noise-corrupted channel symbols under bandwidth and average-power constraints. DeepWiVe seeks to maximize reconstruction quality using PSNR or MS-SSIM.
- Encoding and decoding: The encoder E maps a video sequence to k complex channel symbols, while the decoder D reconstructs the video from the noisy received symbols.The channel output is modeled as y = z + n, producing an approximate reconstruction X̂.
- Bandwidth constraint: The system restricts transmission to k channel uses per GoP, defining the bandwidth constraint through the bandwidth compression ratio ρ.The ratio is introduced to characterize how many channel uses represent the video content.
- Channel model: The channel is modeled as AWGN with complex Gaussian noise n ∼ CN(0, σ^2I), under an average transmitter-power constraint.The identity matrix I is the noise covariance structure, and σ^2 denotes the noise-power parameter.
- Optimization objective: The design maximizes PSNR or MS-SSIM between the input video and its reconstruction under bandwidth-ratio ρ and average-power P constraints.MS-SSIM is used alongside PSNR to measure reconstructed-video quality.
A. Joint Source-Channel Video Coding
DeepWiVe uses DNN-based encoders and decoders for key frames and interpolated frames, combining channel transmission with motion and residual processing. The design also adapts bandwidth allocation to channel conditions and frame content.
- Joint Source-Channel Video Coding: DeepWiVe decomposes video coding into key-frame networks, interpolation networks, and a bandwidth-allocation function, all represented by DNNs.The components are fθ/fθ′, gφ/gφ′, and qψ.
- Bandwidth allocation: Bandwidth is dynamically allocated among frames using a reinforcement-learning-trained network qψ rather than assuming identical allocation for every frame.The allocation network is trained to optimize bandwidth utilization within a GoP.
- Frame coding: The key frame is compressed and transmitted through fθ, while remaining frames encode motion and residual information using gφ.GoPs are encoded and decoded sequentially, with previously decoded frames available as references.
- Frame interpolation: DeepWiVe uses channel emulation at the transmitter to approximate the receiver’s reconstructed reference frames when forming interpolation inputs.This addresses the dependence of residual information on channel conditions in JSCC transmission.
- Motion estimation: Scaled space flow generalizes pixel warping by blurring difficult-to-model motion regions and compensating them with residual information.The SSF estimator hη produces the flow used for interpolation.
- Frame interpolation: The decoder reconstructs interpolated frames by estimating scaled space flow, residuals, and a mask from noisy channel representations.The mask combines two motion-compensated predictions and the predicted residual as convex weights.
B. Bandwidth Allocation
DeepWiVe formulates per-GoP bandwidth allocation as an MDP and uses reinforcement learning to assign channel uses among frames according to their information needs. Successive latent refinement enables frames to receive different bandwidth amounts while remaining decodable.
- MDP formulation: Bandwidth allocation is formulated as an MDP whose actions assign the available channel uses k across the N frames in a GoP.The state includes frame-related tensors, prior reference information, and estimated channel noise power; the reward reflects final video quality.
- MDP formulation: Frames with larger differences from their reference frames require more transmitted information, while similar frames can use less bandwidth.The previous GoP’s last frame becomes the next GoP’s reference frame.
- Variable bandwidth transmission: Latent vectors are divided into V equal-sized blocks, allowing successive refinement as increasingly more information is transmitted.Randomly varying the number of transmitted blocks trains models to support different bandwidth amounts.
- Variable bandwidth transmission: The action set contains all ways to assign V blocks among N frames, producing a combinatorial allocation space.The paper gives the number of assignments as |A| = (V + N −1)! / [V !(N −1)!].
- RL optimization: The reward uses either lPSNR or lMS-SSIM, depending on the video-quality metric being optimized.The bandwidth allocation network qψ selects actions from the allocation space.
- RL optimization: Deep Q-learning approximates the state-action Q-function to learn the allocation policy that maximizes discounted visual-quality rewards.Replay buffers, target networks, and ϵ-greedy exploration support Q-function learning and stabilize updates.
A. Training Details
The models are trained on UCF101 with separate stages for JSCC networks and the bandwidth allocation network. Training samples channel SNR uniformly across a broad range while using a discretized allocation space for tractable Q-learning.
- Model training: The models use UCF101, PyTorch, Adam with learning rate 1e−5, and an 0.8:0.1:0.1 training-validation-evaluation split.The JSCC networks are trained before the bandwidth allocation network.
- Channel and allocation settings: During training, the transmitter and receiver are assumed to estimate the channel SNR accurately, so ˆσ2 = σ2.This assumption is made during bandwidth allocation network training.
- RL training: Bandwidth allocation training uses DQN hyperparameters γ = 0.99, τ = 0.005, and replay buffer size |R| = 1000.The allocation network assigns bandwidth within one video sequence per episode.
- Channel and allocation settings: Training SNR is sampled uniformly from [−5, 20]dB, with N = 4 frames and V = 20 latent blocks.Choosing V = k would provide finer control but create an action space that makes Q-learning difficult.
B. Simulation Results
DeepWiVe is evaluated against separation-based H.264/H.265 plus LDPC schemes across channel conditions, channel-estimation mismatch, and bandwidth settings. The experiments also test whether reinforcement-learning bandwidth allocation improves quality over uniform allocation.
- Channel estimation mismatch: DeepWiVe gracefully degrades as SNR decreases despite fixed SNREst, unlike the cliff-edge deterioration of separation-based transmission.The comparison specifically examines channel-estimation error and the H.264-plus-LDPC cliff-effect regime.
- Visual comparison: Visual examples show the clearest DeepWiVe-versus-H.264 quality difference in the three people standing at the back of the scene.The encircled background subjects are the highlighted comparison region.
- Codec comparisons: DeepWiVe outperforms H.264 with LDPC at all tested SNRs for ρ = 0.031 and ρ = 0.018 in the reported comparisons.The ρ = 0.018 comparison reports gains of 2.05 dB PSNR and 0.0129 MS-SSIM at SNR ∈[13, 20]dB, and 4.72 dB PSNR and 0.0462 MS-SSIM at SNR ∈[3, 6]dB.
- Codec comparisons: DeepWiVe outperforms H.265 in MS-SSIM, although H.265 performs better in PSNR at higher SNRs.At very low SNR, H.265 does not produce results because it cannot meet the required compression rate.
- Bandwidth allocation: Optimized allocation improves over uniform allocation by 0.67 dB PSNR and 0.0008 MS-SSIM at ρ = 0.031.For ρ = 0.018, the gains increase to 0.99 dB PSNR and 0.0023 MS-SSIM, with larger gains under more limited bandwidth.
V. CONCLUSION
The conclusion presents DeepWiVe as a DNN-aided JSCC video transmission system with dynamic bandwidth allocation and residual estimation without distortion feedback. It reports graceful degradation, strong performance against H.264-based separation schemes, perceptual gains over H.265, and benefits from optimized allocation.
- Contribution: DeepWiVe is presented as the first DNN-aided joint source-channel wireless video transmission scheme in the literature.Its architecture supports dynamic bandwidth allocation and residual estimation without distortion feedback.
- Robustness: DeepWiVe overcomes the cliff-effect and achieves graceful degradation with channel quality.The conclusion attributes this behavior to the end-to-end JSCC system under changing channel conditions.
- Bandwidth allocation: DeepWiVe improves over uniform bandwidth allocation by up to 0.87 dB in PSNR.The allocation network dynamically distributes available bandwidth within a GoP to maximize visual quality under the bandwidth constraint.
- Comparative performance: DeepWiVe is better than H.264 with LDPC in all considered channel conditions by up to 4.72 dB.H.265 performs better in PSNR, while DeepWiVe outperforms H.265 in MS-SSIM, a perceptually aligned metric.