PaperlayerView exact revision on arXiv

Source-linked AI summary

ReLViC: Loss-Resilient Learned Video Coding with Dispersed Packetization and Controllable Packet Dependencies

Xuyang Chen, Daquan Feng, Xianfu Chen, Xiang-Gen Xia

arXiv:2608.30202v1eess.IVcs.MM

TL;DR

Packet loss disrupts learned video coding through missing latent tokens and temporal dependencies. ReLViC combines dispersed packetization, a dual-purpose Transformer, controllable dependencies, and progressive training for recovery. Under burst-loss traces, it provides more stable reconstruction and outperforms H.265+FEC and GRACE under severe packet loss.

  • Problem

    Packet loss can compromise spatial reconstruction and temporal prediction in learned video coding.

  • Method

    ReLViC jointly codes and recovers latent representations using dispersed packetization, a dual-purpose Transformer, controllable packet dependencies, and progressive training.

  • Results

    ReLViC provides more stable reconstruction and outperforms H.265+FEC and GRACE under severe packet loss in burst-loss-trace experiments.

  • Takeaways & Limitations

    Selecting segment length g tunes the trade-off between compression efficiency and error-propagation range without retraining.

Abstract

from arXiv · show

Packet loss can severely impair learned video coding because missing latent tokens compromise both spatial reconstruction and temporal prediction. We present ReLViC, a loss-resilient learned video coding framework that jointly addresses latent coding and packet-loss recovery. ReLViC disperses spatially adjacent latent tokens across packets and employs a dual-purpose Transformer to estimate entropy-model parameters during coding and reconstruct missing latent tokens at the receiver. It controls packet dependencies through a periodic-reset packet-context topology parameterized by the segment length, thereby tuning the trade-off between compression efficiency and error-propagation range without retraining. A three-stage progressive training procedure establishes single-frame coding, learns temporal context for entropy modeling, and then optimizes the recovery of masked latent tokens under simulated packet loss. Experiments using burst-loss traces evaluate ReLViC against H.265 protected by Reed--Solomon forward error correction (FEC) and GRACE, a loss-resilient learned video codec. ReLViC delivers more stable reconstruction and outperforms both baselines under severe packet loss.

I. INTRODUCTION

ReLViC addresses packet-loss sensitivity in learned video coding by jointly designing latent coding, packetization, dependency control, and recovery. It uses dispersed packetization, a shared Transformer, and progressive training, with experiments showing stable reconstruction and stronger severe-loss performance than the baselines.

  • Packet loss can cause decoding failures, freezes, and persistent error propagation because learned and conventional codecs rely on spatial, temporal, and entropy-model dependencies.
  • Existing loss-resilient approaches do not fully resolve temporal error propagation, compression–resilience trade-offs, changing network conditions, or sustained burst-loss recovery.
  • ReLViC jointly optimizes latent coding, dispersed packetization, entropy modeling, and packet-loss recovery for digital packet networks.
  • Its dual-purpose Transformer estimates entropy-model distributions and reconstructs missing latent tokens from received spatial and temporal evidence.
  • Dispersed packetization distributes adjacent latent tokens across packets, while segment length g tunes compression efficiency against error-propagation range without retraining.
  • Experiments with real-world network traces show stable reconstruction across loss rates and outperform H.265+FEC and GRACE under severe packet loss.

B. Loss-Resilient Video Transmission

Loss-resilient transmission combines conventional feedback or redundancy mechanisms with ReLViC’s source-coding strategy. ReLViC disperses latent tokens, synchronizes states after loss, and controls dependency propagation through segment length g.

  • ARQ and HARQ depend on feedback and sufficient time before playback deadlines, while FEC must choose redundancy before the loss pattern is known.
  • The learned codec maps frames to quantized latent maps, entropy-codes them, and reconstructs frames through analysis and synthesis transforms.
  • Window-based token partitioning restricts attention to shorter sequences, reducing computational complexity while preserving latent-token structure.
  • ReLViC disperses spatially adjacent latent tokens across packets so burst losses produce separated missing-token regions that the Transformer can reconstruct.
  • Segment length g defines periodic-reset packet dependencies and controls the maximum error-propagation span without retraining.
  • Receiver feedback uses loss bitmaps to synchronize encoder and decoder temporal reference states before the next frame.

B. Spatiotemporal Context Modeling for Latent Tokens

ReLViC builds historical context from preceding latent maps and combines it with current-frame tokens. Asymmetric windowing gives current tokens access to broader spatial and temporal evidence for prediction and recovery.

  • The context model summarizes the two preceding latent maps as joint spatiotemporal historical context for the current frame.
  • Preceding maps use larger overlapping windows, while the current map uses smaller nonoverlapping windows to provide each current token with a broader receptive field.
  • Spatial and temporal Transformers independently process historical frames before combining their outputs into zhist.
  • The current latent map is tokenized for Tcur, which estimates latent-token distributions and reconstructs missing tokens.

3) Dual-Purpose Transformer:

The dual-purpose Transformer shares spatiotemporal context extraction between entropy modeling and latent-token recovery. Its density and recovery heads support coding received tokens and predicting missing ones.

  • Current latent tokens and historical context are processed with self-attention and cross-attention to estimate conditional distributions.
  • The density head outputs Gaussian mean and scale parameters for the entropy model, whose continuous distribution is integrated over quantization intervals for discrete coding probabilities.
  • The recovery head predicts missing latent tokens from the same spatiotemporal evidence used for distribution estimation.
  • Sharing the Transformer backbone reduces parameter count and computational cost despite the recovery and distribution tasks using different visibility patterns.
  • The final latent representation combines predicted tokens with received tokens before reconstruction.

C. Three-Stage Progressive Training

ReLViC uses three progressive training stages to build single-frame coding, temporal entropy modeling, and packet-loss recovery. The final stage jointly fine-tunes the model with randomized masking and balances intact reconstruction against recovery quality.

  • Stage I: Single-Frame Compression: Stage I trains the spatial codec for single-frame rate–distortion coding while including both main-latent and hyperlatent rates.The hyperprior is used only in Stage I and discarded afterward.
  • Stage II: Temporal Entropy Modeling: Stage II freezes the image codec and trains temporal context modules to predict current-frame token distributions from historical latent context, reducing entropy-coding rate.A learnable reference frame supports the first frame when no historical frame is available.
  • Stage III: Packet-Loss Recovery: Stage III jointly fine-tunes all parameters through masked latent modeling, replacing masked tokens with a learned latent vector and recovering them with bidirectional interactions.The masking ratio is sampled uniformly between 0.05 and 0.99.
  • Stage III: Packet-Loss Recovery: The two-branch distortion objective balances reconstruction quality without packet loss against recovery quality through a hyperparameter α.A larger α assigns greater importance to masked reconstruction.
  • Stage III: Packet-Loss Recovery: Randomized mask locations expose different tokens to optimization across iterations, enabling global joint optimization while unmasked tokens provide conditional context for masked-token coding.The masked-token rate is optimized explicitly, while unmasked tokens serve as context.

1) Dispersed Latent Packetization:

Dispersed latent packetization assigns nearby latent tokens to different packets, converting concentrated erasures into spatially distributed missing tokens. ReLViC uses deterministic QLDS mapping as its default because it avoids random-seed dependence and regular interleaving periodicity.

  • Packetization principle: Dispersed assignment places neighboring latent tokens in different packets, so a single erasure produces spatially distributed missing tokens rather than a contiguous region.The received tokens consequently provide local evidence for recovery.
  • Packet assignment: The packet assignment maps the two-dimensional latent tokens to K packets, with Mpack(x, y) denoting the packet index at spatial location (x, y).Fig. 3 illustrates QLDS, random, and spatial-interleaving assignments.
  • Alternative assignments: Random packetization uses a fixed permutation and can disperse tokens favorably, but it may form local clusters and depends on a random seed.A deterministic mapping provides more controlled spatial coverage.
  • Alternative assignments: Spatial interleaving separates neighboring tokens on a regular two-dimensional grid, providing strictly regular spacing.Its assignment is defined by Mgrid(x, y) = (x mod Kh) Kw + (y mod Kw).
  • QLDS mapping: QLDS deterministically disperses tokens through a low-discrepancy two-dimensional traversal, avoiding both random assignment’s seed dependence and regular interleaving’s periodicity.The paper adopts QLDS as the default mapping because it is deterministic, reproducible, and requires no layout search.

2) Packet-Context Modes:

ReLViC partitions packet dependencies into segments of length g, using richer context for compression while limiting how far packet losses can propagate. Increasing g expands both the available context and the possible error-propagation path.

  • Context modes: The segment length selects distinct dependency modes: g = 1 removes interpacket references, intermediate g periodically resets the chain, and g = K permits all preceding packets as context.These modes tune the compression–propagation trade-off without changing the model’s training.
  • Entropy modeling: The shared Transformer Tcur estimates conditional distributions for packet tokens, with conditional entropy decreasing as more preceding packet context is available.Let Y_i denote the tokens in packet i.
  • Context trade-off: Additional visible context can improve compression efficiency but lengthens error-propagation paths, motivating finite packet-context segments of length g.The packets are divided into segments S_m of up to g packets.
  • Segment topology: Each segment begins with an independent packet, later packets depend on earlier packets within that segment, and different segments remain independent.The dependency matrix is strictly lower triangular, so packet i uses neither the current nor future packets.
  • Error propagation: The maximum error-propagation span is g−1 packets because a lost referenced packet can affect only later dependent packets in its segment.Physically lost and rendered-undecodable packets are recovered by fPLR.
  • Context modes: As g increases, dependency blocks become larger, providing richer cross-packet context while extending the possible error-propagation path.Fig. 5 visualizes configurations for g = 1, 4, 7, and K.

IV. EXPERIMENTAL SETUP AND RESULTS

The evaluation uses Vimeo-90k-trained models, UVG 1080p videos, PSNR, ten packets per frame, and burst-loss simulation based on a three-state Markov model. ReLViC is compared with H.265 protected by Reed–Solomon FEC and GRACE.

  • Training: Models are trained on Vimeo-90k septuplets using three progressive stages, 256 × 256 crops, AdamW, and two NVIDIA A100 GPUs.Stage I uses λ1 = 0.01 with a warm-up; Stage III varies λ3 across five values.
  • Baselines: ReLViC is compared with H.265 protected by Reed–Solomon FEC and the mask-trained learned codec GRACE.H.265 redundancy ratios are evaluated at 0%, 10%, 30%, 50%, and 70%.
  • Evaluation: Evaluation uses seven 1080p UVG videos, PSNR, ten packets per frame, and an effective H.265+FEC payload budget of 1460 bytes per packet.FEC redundancy is allocated non-uniformly to improve H.265 decodability, and failed frames receive a PSNR penalty.
  • Loss simulation: Burst losses are simulated with a three-state Markov model whose states represent loss-free transmission, packet loss, and intermittent availability.Four loss patterns are derived from mobile Internet traces, with transition parameters and average loss statistics listed in Table II.

B. Results

ReLViC maintains more stable reconstruction under burst losses by combining dispersed packetization, controllable packet dependencies, and context-based latent recovery. Its performance advantages are strongest under severe loss, while packet-context selection trades compression efficiency against error propagation.

  • Visual reconstruction: At 50% packet loss, ReLViC retains more visible structure than GRACE by reconstructing missing latents from current-frame tokens and temporal context.The comparison reports that GRACE accumulates distortions and artifacts along its reference path.
  • Visual reconstruction: At 90% packet loss, later frames improve because received current-frame tokens and recovered historical context provide additional reconstruction evidence.The first frame lacks preceding-frame context and shows larger smeared regions, while subsequent frames reduce highlighted error regions.
  • Burst-loss performance: Across four burst-loss traces, ReLViC outperforms GRACE and H.265 + FEC under higher-loss conditions, while FEC still suffers numerous decoding errors with 70% redundancy.Under EP3 and EP4, ReLViC has a clear advantage; under EP2, it performs better than both cited baselines as loss probability increases.
  • Packet-context modes: Intra-slice coding is more resilient at high loss rates, whereas layered and segmented modes outperform it at lower loss rates by using interpacket references.Longer reference chains degrade more rapidly as packet loss increases.
  • Packet-context modes: Selecting segment length g directly controls the trade-off between compression efficiency and error-propagation range without retraining.The context modes provide different performance profiles because interpacket dependencies extend propagation paths.
  • Packetization ablations: QLDS-based packetization performs strongly across coding modes because its low-discrepancy dispersion avoids local clustering and periodic interleaving structure.Under intra-slice coding, QLDS, random, and spatial-interleaving packetization are comparable when missing tokens are sufficiently dispersed.
  • Training trade-offs: With α = 1, the intra-slice mode shows a 3.16 dB smaller PSNR drop than α = 0.01 as packet loss rises from 0.1 to 0.9.The parameter α balances compression performance against loss resilience; smaller α emphasizes loss-free reconstruction quality.
  • Loss-free performance and complexity: Under loss-free conditions, ReLViC has compression performance comparable to or slightly better than H.265 while offering greater packet-loss resilience than GRACE.The model contains 231.155 M parameters and costs approximately 32.8 TFLOPs for encoding and decoding.

V. CONCLUSION

ReLViC couples latent recovery with packet-level organization through a dual-purpose Transformer and dispersed packetization. Experiments show strong performance under heavy packet loss, while segment-length control enables adaptation between compression efficiency and transmission resilience without retraining.

  • V. CONCLUSION: A dual-purpose Transformer uses entropy-modeling context to infer missing latent tokens, coupling compression and recovery in one framework.The progressive training procedure establishes single-frame coding, learns temporal entropy modeling, and then trains masked-token recovery.
  • V. CONCLUSION: Dispersed packetization prevents one erasure from creating a large contiguous missing region, while segment length g sets the maximum propagation span to g−1 packets.These packet-level choices provide different compression-efficiency and error-propagation trade-offs.
  • V. CONCLUSION: ReLViC substantially outperforms H.265+FEC and GRACE under heavy packet loss and maintains favorable reconstruction quality at a 90% packet-loss rate.Switching packet-context modes balances compression efficiency and transmission resilience under different network conditions.
Loading 2608.30202v1…