Semantic-Aware Neural Video Codec for Error-Resilient Low-Latency Transmission

Matin Mortaheb; Homa Esfahanizadeh; Jinfeng Du; Harish Viswanathan

Source-linked AI summary

Semantic-Aware Neural Video Codec for Error-Resilient Low-Latency Transmission

Matin Mortaheb, Homa Esfahanizadeh, Jinfeng Du, Harish Viswanathan

arXiv:2609.16279v1Sep 14, 2026eess.IVcs.ITcs.LGcs.MM

TL;DR

Low-latency physical AI video transmission must remain task-relevant despite packet erasures, while existing codecs suffer from sequential decoding dependencies and uniform protection. The paper extends DCVC-RT with semantic- and feature-aware packetization, independent per-packet entropy coding, and end-to-end training over multi-level erasure channels. Experiments report higher robustness, graceful degradation in less important regions, and better preservation of task-relevant content.

Problem
Existing neural video codecs are optimized mainly for pixel-level reconstruction, assume error-free transmission, and use sequential entropy estimation that makes packet erasures propagate errors.
Method
The framework partitions latent representations across spatial regions and feature slices, assigns packets to protection levels, decodes packets independently, and trains end-to-end over multi-level erasure channels.
Results
The proposed framework significantly improves robustness over DCVC-RT under packet erasures, preserving task-relevant regions while gracefully degrading less important regions.
Takeaways & Limitations
Semantic-aware unequal error protection makes low-latency video transmission more task-oriented by prioritizing important content under unreliable channels.

Abstract

from arXiv · show

Emerging physical AI systems require low-latency, task-oriented video communication over unreliable channels. We propose a semantic-aware multi-level neural video coding method for robust low-latency video transmission over unreliable channels that are abstracted as multi-level packet erasure channels. Built upon the real-time DCVC-RT neural video codec, the proposed framework introduces a semantic- and feature-aware coding strategy that partitions encoded representations into packets carrying different levels of semantic and latent-feature importance and assigns these packets to different streams, each associated with a priority level when transmitted over unreliable communication channels. We also developed an error-resilient entropy model that removes inter-packet dependencies, allowing each packet to be decoded independently under packet losses. The complete system is trained end-to-end over the abstracted multi-level packet erasure channels, enabling learning of channel-aware representations together with importance-aware packet assignment while facilitating the network for differentiated packet prioritization. Experiments show that the proposed framework significantly improves robustness over baseline DCVC-RT under packet erasures, achieving graceful degradation in less important regions while better preserving task-relevant visual content.

I. INTRODUCTION AND PRELIMINARIES

Physical AI needs low-latency video transmission that remains reliable over packet-erasure channels, but conventional and neural codecs are vulnerable to losses and error propagation. The proposed framework combines semantic prioritization, unequal packet protection, independent decoding, and end-to-end channel-aware training.

I. INTRODUCTION AND PRELIMINARIES: Packet losses can catastrophically degrade entropy-coded video because conventional systems assume reliable transmission and neural codecs often use sequential entropy dependencies.Low-latency settings also make retransmissions and feedback-based rate adaptation infeasible.
I. INTRODUCTION AND PRELIMINARIES: Existing approaches generally optimize pixel-level quality or feature-domain robustness without jointly aligning region semantics, feature importance, and heterogeneous channel reliability.Related block-erasure methods protect important latent features but do not explicitly protect task-relevant spatial regions.
I. INTRODUCTION AND PRELIMINARIES: The framework jointly partitions representations across feature and spatial domains so packets containing task-relevant regions and important latent features receive preferential protection.Packets are assigned to streams with potentially different reliability levels.
I. INTRODUCTION AND PRELIMINARIES: An independently decodable entropy model removes inter-packet dependencies, preventing packet losses from propagating decoding errors to subsequent packets.This redesign targets the sequential entropy dependency that makes existing neural codecs fragile under erasures.
I. INTRODUCTION AND PRELIMINARIES: End-to-end training over stochastic multi-level block-erasure channels enables channel-aware representations, importance-aware assignment, and graceful degradation under impairments.The design is compatible with transport architectures in which protocol data units carry different priority levels.

A. Preliminaries

DCVC-RT provides real-time neural video compression but assumes error-free communication and uses sequential packet dependencies. The proposed system augments it with packet-independent entropy coding and semantic- and feature-aware protection.

A. Preliminaries: DCVC-RT compresses frames into compact latent representations using learned encoder, decoder, and entropy-coding components for real-time transmission.Compression quality is controlled by a quantization parameter that trades compactness against reconstruction fidelity.
A. Preliminaries: Loss of one packet in DCVC-RT can corrupt probability estimates for subsequent packets, causing severe reconstruction degradation.Its original entropy model therefore assumes reliable, error-free communication.
A. Preliminaries: The proposed system addresses DCVC-RT’s limitations with packet-independent entropy decoding, semantic- and feature-aware packetization, and importance-aware packet-level assignment.These additions target both error propagation and uniform treatment of semantically relevant regions.

II. METHOD

The method extends DCVC-RT into a task-oriented codec that partitions latent representations by spatial and feature importance, assigns packets to protection levels, and independently decodes received packets under erasures.

II. METHOD: The framework combines semantic analysis, hybrid packetization, importance assignment, multi-level erasure transmission, independent decoding, and end-to-end stochastic-erasure training.Missing packets are replaced by zeros before frame reconstruction.
II. METHOD: A semantic module uses a frame and textual query to produce a pixel-level importance map whose larger values identify regions more relevant to the communication task.Continuous pre-threshold confidence values are retained for unequal error protection and quantized into four semantic levels in the experiments.
II. METHOD: The pixel-level map is converted into patch-level importance by dividing each frame into a 4 × 4 grid of 16 spatial patches.Patch importance levels are then consumed by the importance assignment module.
II. METHOD: Training uses randomly generated and spatially smoothed importance maps instead of segmentation-model outputs to reduce overfitting and support generalization to unseen tasks.This exposes the codec and assignment module to diverse spatial importance distributions.

B. Semantic-Aware Packetization

The proposed hybrid packetization jointly partitions latent representations by spatial region and feature slice, enabling protection to reflect both semantic relevance and latent-feature importance.

B. Semantic-Aware Packetization: In original DCVC-RT packetization, each packet combines subsets of latent channels and spatial locations, so losing one packet removes information from both a feature subset and a region.This coupling can cause severe reconstruction degradation.
B. Semantic-Aware Packetization: Feature-only partitioning can preserve globally important latent information but cannot explicitly protect task-relevant spatial regions because every packet spans the full frame.The hybrid strategy addresses this spatial protection limitation.
B. Semantic-Aware Packetization: The hybrid design divides each frame into 16 spatial patches and the latent representation into 8 feature slices, pairing one patch with one slice per packet.This produces 128 packets per frame for both I-frames and P-frames.
B. Semantic-Aware Packetization: Each hybrid packet receives semantic and feature importance indicators, allowing packets from important features and semantically relevant regions to receive preferential protection.The assignment module uses these indicators to allocate packets to protection levels.

C. Semantic-Aware Packet Importance Assignment

The module assigns packets to protection levels using semantic importance, feature significance, learned packet representations, and balanced utilization constraints.

C. Semantic-Aware Packet Importance Assignment: Packets combine semantic importance and latent feature-slice importance to guide their assignment to protection levels.Semantic importance is derived from a patch-level map, while feature importance comes from the corresponding latent feature slice index.
C. Semantic-Aware Packet Importance Assignment: The learned assignment module jointly optimizes packet-to-level allocation with the encoder and decoder while aligning important packets with stronger protection.The objective also maintains balanced utilization across importance levels despite the discrete nature of packet allocation.
C. Semantic-Aware Packet Importance Assignment: Packet descriptors are embedded and combined with level reliability, semantic importance, and feature-slice index to produce assignment logits.Descriptors include mean value, standard deviation, and packet energy; level reliability is r_k = 1−ϵ_k.
C. Semantic-Aware Packet Importance Assignment: The balanced Sinkhorn normalizer produces a soft packet-to-level transport matrix while enforcing balanced level utilization.During training, hard assignments drive the forward pass and soft assignments preserve differentiability in backpropagation.
C. Semantic-Aware Packet Importance Assignment: For Npkt = 128, transmitting the packet-to-level assignment metadata requires 384 bits per frame, described as negligible relative to the compressed video stream.The assignment matrix must accompany the compressed bitstream because the receiver cannot deterministically recover the learned assignments.

D. Multi-Level Erasure Channel Abstraction

The system models unreliable transmission as balanced multi-level packet erasure channels, training packet assignments to match importance with protection levels.

D. Multi-Level Erasure Channel Abstraction: The abstraction uses L levels ordered by erasure probability, with level 1 providing the greatest protection and level L the least.Erasure probabilities satisfy ϵ1 ≤ ϵ2 ≤ ··· ≤ ϵL.
D. Multi-Level Erasure Channel Abstraction: Each importance level carries an equal number of packets to preserve balanced utilization.This constraint complements the balanced assignment mechanism used by the importance module.
D. Multi-Level Erasure Channel Abstraction: Training over the abstraction teaches the framework to place important features and spatial regions on higher-protection levels.The model is intended as a general erroneous-channel abstraction rather than requiring a physical multi-channel infrastructure.
D. Multi-Level Erasure Channel Abstraction: The abstraction aligns with QoS structures in which PDU blocks carry differentiated importance levels.A PDU set can contain multiple PDU blocks corresponding to packets assigned to one importance level.

E. Error-Resilient Entropy Coding and Decoding

The codec replaces sequential packet entropy modeling with independent packet decoding based on shared hyperprior-derived parameters, preventing loss propagation and supporting graceful degradation.

E. Error-Resilient Entropy Coding and Decoding: Original DCVC-RT estimates packet distributions sequentially using previously decoded packets, creating inter-packet dependencies under reliable transmission.The sequential strategy improves compression efficiency but makes later probability estimates vulnerable when earlier packets are lost.
E. Error-Resilient Entropy Coding and Decoding: The redesigned entropy model estimates each packet distribution from shared hyperprior-derived per-packet parameters without using other packets.A binary spatial mask extracts (µ_i, σ_i) for packet Y_i from the global (µ, σ) map.
E. Error-Resilient Entropy Coding and Decoding: The redesigned model retains the hyper-encoder, hyper-decoder, and prior-fusion network while removing cross-packet dependencies.The shared hyperprior is expanded to the full latent size at both encoder and decoder.
E. Error-Resilient Entropy Coding and Decoding: Independent packet decoding prevents a lost packet from corrupting probability estimation for subsequent packets, enabling graceful degradation under erasures.Successfully received packets are decoded into their latent positions, while missing packets are replaced with zeros before reconstruction.

F. End-to-End Semantic and Erasure-Aware Learning

The framework jointly trains compression, semantic weighting, packet assignment, entropy modeling, and decoding under stochastic erasures. Its objective penalizes semantic distortion and guides packets from important regions and feature slices toward stronger protection levels.

F. End-to-End Semantic and Erasure-Aware Learning: The objective combines bitrate, assignment entropy, patch-priority, and feature-priority terms to optimize compression, deterministic assignment, and importance-aware protection.The weighting coefficients control the relative contributions of these four components.
F. End-to-End Semantic and Erasure-Aware Learning: Semantic-weighted MSE assigns larger penalties to reconstruction errors in important regions, encouraging higher-fidelity preservation of task-relevant content.The loss uses a pixel-level importance map and the reconstructed frame.
F. End-to-End Semantic and Erasure-Aware Learning: End-to-end training exposes the encoder, importance assignment module, and decoder to randomly simulated packet erasures before decoding and reconstruction.This trains the system under the multi-level erasure abstraction rather than only under error-free compression.

III. EXPERIMENTAL RESULTS

The evaluation uses Vimeo-90K for training and five varied-resolution video sequences for testing. Results are reported at the highest QP value of 63.

III. EXPERIMENTAL RESULTS: Training uses 89,800 Vimeo-90K clips, with the first frame for the I-frame model and six remaining frames for the P-frame model.P-frame training follows DCVC-RT’s implicit temporal modeling design using decoded previous-frame features.
III. EXPERIMENTAL RESULTS: Evaluation covers five video sequences totaling 1,262 frames at 1280 × 720 and 1920 × 1080 resolutions, with all results measured at QP 63.The test sequences vary in resolution.

A. Experiment 1: Robustness and Graceful Degradation

The framework remains more robust than DCVC-RT across packet-erasure conditions and allocates reconstruction quality toward semantically important regions. Its independent entropy model supports graceful degradation, while less important regions degrade more readily.

A. Experiment 1: Robustness and Graceful Degradation: The proposed framework consistently achieves higher PSNR than DCVC-RT across a wide range of erasure probabilities.It also remains superior under channel conditions more severe than those used during training.
A. Experiment 1: Robustness and Graceful Degradation: Semantically important regions receive significantly higher reconstruction quality, while less important regions degrade gracefully instead of degrading uniformly.DCVC-RT exhibits nearly uniform degradation across regions.
A. Experiment 1: Robustness and Graceful Degradation: The independent entropy model enables graceful quality reduction, whereas DCVC-RT suffers catastrophic degradation even at low erasure rates.The comparison attributes the baseline failure to its sequential entropy dependency.

B. Experiment 2: Semantic-Aware Unequal Error Protection

The framework aligns packet protection with semantic and latent-feature importance over multi-level erasure channels. It preserves semantically important regions while allowing less important regions to degrade gracefully, unlike DCVC-RT.

B. Experiment 2: Semantic-Aware Unequal Error Protection: Inference evaluates semantic-aware protection under channel erasure probabilities ranging from 0.1 to 0.8 after training with non-uniform erasure probabilities from 0.10 to 0.30.A semantic segmentation module produces importance maps used by the packet routing module.
B. Experiment 2: Semantic-Aware Unequal Error Protection: Semantically important regions achieve higher reconstruction quality while less important regions degrade gracefully under packet erasures, unlike DCVC-RT’s nearly uniform degradation.This demonstrates semantic-aware unequal error protection and alignment between transmission reliability and semantic importance.
B. Experiment 2: Semantic-Aware Unequal Error Protection: The proposed codec combines independent per-packet decoding, hybrid semantic- and feature-aware packetization, and learned importance assignment across protection levels.End-to-end training over multi-level erasure channels supports channel-aware representations and preferential protection of task-relevant content.