PaperlayerView exact revision on arXiv

Source-linked AI summary

Tokenized Joint Source-Network Coding for Loss-Resilient Video Streaming

Ke Yang, Sixian Wang, Kailin Tan, Guo Lu, Jincheng Dai, Wenjun Zhang, Ping Zhang

arXiv:2610.21166v1eess.IVcs.MM

TL;DR

Real-time video streaming must handle packet losses under strict deadlines, while conventional protection and physical-layer JSCC pose distinct tradeoffs. TokJSNC jointly adapts token transmission and recovery to video content and network conditions. At 70% packet loss, it maintains above 22 fps, while the strongest baseline remains below 12 fps.

  • Problem

    Separated compression and loss protection make it difficult to achieve both high compression efficiency and loss-resilient reconstruction under strict deadlines and dynamic packet losses.

    Source: §I · p. 2

  • Method

    TokJSNC jointly adapts token allocation, packet organization, transport representations, and latent-token recovery using video content and packet-loss conditions.

    Source: §I · p. 2

  • Results

    At a 70% packet-loss ratio, TokJSNC maintains an average playback frame rate above 22 fps, whereas the strongest baseline remains below 12 fps.

    Source: §V · p. 13

  • Takeaways & Limitations

    TokJSNC demonstrates adaptive token transmission for robust real-time video streaming over diverse packet-loss conditions without FEC or retransmission.

    Source: §V · p. 13

Abstract

from arXiv · show

For decades, conventional video streaming systems have relied on packet-level forward error correction (FEC) codes and retransmissions to ensure reliable video delivery. However, in real-time communications, FEC is inherently inefficient under unknown and time-varying packet-loss conditions, while retransmission can incur excessive latency and fail to meet stringent playout deadlines. Emerging deep joint source-channel coding (DeepJSCC) provides an alternative by learning channel-noise-resilient representations, yet its tight coupling with the physical layer makes it difficult to integrate into today's layered network architectures and protocols. In this paper, we propose TokJSNC, a simple-but-effective tokenized joint source-network coding paradigm for packet-loss-resilient video streaming. Rather than first compressing video into a compact bitstream and then protecting it with packet-level FEC, TokJSNC directly encodes video into variable-length discrete token-stream, whose transmission rate is jointly adapted to video content complexity and network conditions. By exploiting the spatiotemporal correlations among learned video tokens, TokJSNC enables robust video reconstruction over a wide range of packet-loss rates without explicit FEC or retransmissions. Consequently, TokJSNC brings the JSCC-like graceful degradation property to the network layer while remaining compatible with existing network architectures and transport protocols. Experimental results show that TokJSNC maintains an average playback frame rate above 22 fps even at a 70% packet-loss rate, whereas the best-performing baseline drops below 12 fps. Overall, TokJSNC establishes a practical pathway for repurposing modern generative video tokenizers for loss-resilient streaming, while providing a fundamental framework for video token communications.

I. INTRODUCTION

Conventional streaming separates compression from loss protection, leaving dependent bitstreams vulnerable to packet loss while FEC and retransmissions strain bandwidth or latency. TokJSNC instead coordinates token allocation, packet organization, and recovery around video content and network conditions.

I. INTRODUCTION

  • Compact conventional and neural video bitstreams contain dependencies that can cause decoding failures and error propagation after packet loss, while FEC adds bandwidth overhead and retransmissions can miss strict deadlines.This separation makes it difficult to achieve both high compression efficiency and loss-resilient reconstruction under stringent deadlines and dynamic losses.Source: §I · pp. 1–2
  • JSCC offers graceful degradation as channel SNR varies, but its end-to-end physical-layer optimization is difficult to integrate with packet-based network architectures.The paper proposes extending joint design to source representation, packetization, and packet-loss concealment while treating the lossy network as a non-trainable layer.Source: Figure 1 · p. 1, §I · p. 2
  • TokJSNC uses discrete tokens that remain usable after partial packet reception, with clip-level token budgets and informative positions guided by reconstruction proxies.It also disperses neighboring spatiotemporal tokens across packets and adapts encoding and decoding using packet-loss statistics and observed token availability.Source: §I · p. 2
  • TokJSNC is presented as a tokenized joint source-network coding framework that reconstructs from partial packets while remaining compatible with layered network architectures.Its design combines adaptive token routing, network-state-conditioned encoding and decoding, and spatiotemporal packetization without FEC or retransmissions.Source: §I · p. 2

II. RELATED WORK

Prior video codecs compress efficiently but generally depend on reliable delivery; tokenizers offer modular representations, while existing loss-resilient methods leave system-level coordination open. TokJSNC coordinates token allocation, transmission, packetization, and recovery for packet-loss streaming.

A. Conventional and Neural Video Coding

  • Conventional and neural video codecs exploit spatial and temporal dependencies for compression, but packet loss can trigger decoding failures or severe reconstruction degradation.Neural codecs improve compression through learned motion, context, and feature representations, yet generally do not explicitly optimize reconstruction from incomplete packet reception.Source: A. Conventional and Neural Video Coding · p. 3

B. Neural Video Tokenization

  • Discrete video tokens support partial packet reception because individual tokens can remain available after other tokens are lost, but existing tokenizers do not adapt allocation, packet organization, or recovery to heterogeneous networks.MAGVIT, Open-MAGVIT2, Cosmos Tokenizer, LFQ, and OmniTokenizer are described as advances in tokenization primarily for generation or representation learning.Source: B. Neural Video Tokenization · p. 3

C. Loss-Resilient Neural Video Coding

  • GRACE reconstructs video under packet loss using randommask training and packet-level entropy coding, whereas ResiComp addresses loss-resilient image compression without video temporal dependencies or playback continuity.These methods focus mainly on recoverability of a given representation; TokJSNC instead formulates allocation, network conditioning, packet organization, and receiver recovery as a unified source-network coding problem.Source: C. Loss-Resilient Neural Video Coding · p. 3

A. Overview

  • TokJSNC maps video to a spatiotemporal latent-token grid, routes selected tokens using reconstruction-derived clip complexity, and conditions transport representations on shared network statistics.At the receiver, a state-aware decoder reconstructs incomplete latent tokens before the detokenizer produces the video clip.Source: A. Overview · pp. 3–4
  • TokJSNC groups quantized transport tokens into packets and models packet loss through the availability states of selected tokens.The packet set contains token groups; the decoder uses token-state information and the shared network prior to recover the complete latent representation.Source: A. Overview · p. 4

B. Adaptive Token Router

The adaptive router addresses fixed token allocation by assigning representation capacity according to clip complexity and prioritizing informative positions. Its motivation is that more complex video clips should receive larger token budgets, although the true data distribution is difficult to estimate directly.

B. Adaptive Token Router

  • The reconstruction-proxy-guided router sets both a clip-level token budget and content-priority token positions, addressing fixed allocation that can waste resources on simple clips and underserve complex ones.The router estimates complexity from distortion between the original clip and a proxy reconstruction generated by a frozen detokenizer.Source: B. Adaptive Token Router · p. 4

1) Motivation for Adaptive Routing:

  • Higher representation complexity motivates assigning more tokens, but the underlying video data distribution is unknown and difficult to estimate directly.The paper uses the negative ELBO as a variational surrogate for representation complexity; the routing policy normalizes the estimate around a nominal budget βN and maps it to an admissible integer budget.Source: 1) Motivation for Adaptive Routing · p. 4

2) Reconstruction-Proxy-Based Token Budgeting:

TokJSNC uses a frozen reconstruction proxy to estimate clip complexity and allocate an adaptive token budget. The same proxy also supplies localized errors for selecting informative token positions.

2) Reconstruction-Proxy-Based Token Budgeting:

3) Token-Level Reconstruction-Guided Selection:

TokJSNC selects tokens using localized reconstruction difficulty and adapts transmission and recovery to packet-loss conditions through a shared network-state prior.

3) Token-Level Reconstruction-Guided Selection:

  • Token-aligned reconstruction error is pooled to the latent-token grid, with larger scores marking positions of greater reconstruction difficulty and content priority.The selected set I contains the L highest-scoring positions.Source: 3) Token-Level Reconstruction-Guided… · p. 5
  • The reconstruction error for each token is measured as regional MSE between the input clip and its proxy reconstruction.Each region corresponds to the spatiotemporal area aligned with a latent token.Source: 3) Token-Level Reconstruction-Guided… · p. 5

C. Network-State-Conditioned Token Encoder and Decoder

  • The encoder conditions transport representations on token ratio η and shared network prior ϕ, while the decoder combines that prior with observed token states to recover incomplete latents.The prior provides slow-timescale information about packet-loss conditions.Source: C. Network-State-Conditioned Token… · p. 5

1) Network State Prior Representation:

TokJSNC represents packet-loss conditions with a shared prior available before transmission, then uses that prior to condition the encoder's transport representations.

1) Network State Prior Representation:

  • The framework considers random, burst, and structured packet-loss models, representing memoryless, temporally correlated, and trace-driven or periodic losses, respectively.Source: 1) Network State Prior Representation · p. 5
  • The network state prior ϕ summarizes packet-loss statistics shared by transmitter and receiver before transmission, without revealing future loss realizations.The profile descriptor captures average loss rate, burst-length characteristics, and temporal regularity.Source: 1) Network State Prior Representation · pp. 5–6
  • The encoder and decoder share the network-state prior, with the decoder also using token-state information to recover incomplete latent tokens.Source: Figure 4 · p. 6

2) TokJSNC Encoder:

  • The TokJSNC encoder conditions transport representations on token ratio η and prior ϕ, then extracts and quantizes transport tokens for packetized transmission.Its Transformer blocks capture token dependencies under the source and network conditions.Source: 2) TokJSNC Encoder · p. 6

3) TokJSNC Decoder:

The TokJSNC decoder distinguishes received, dropped, and unselected latent positions, then uses their states and the shared network prior to reconstruct the latent representation.

3) TokJSNC Decoder:

  • Each latent position is marked received, dropped, or unselected, distinguishing transmission loss from source-side removal during routing.Source: 3) TokJSNC Decoder · p. 6
  • The decoder restores received tokens to their latent positions and pads dropped and unselected positions with separate learnable embeddings.Source: 3) TokJSNC Decoder · p. 6
  • The token-state vector τ and network prior ϕ condition the decoder, which reconstructs the complete latent representation from the state-aware input.The decoder comprises stacked Transformer blocks.Source: 3) TokJSNC Decoder · p. 6

D. Spatiotemporal Token Packetization

Packetization determines which spatiotemporal tokens disappear together, so TokJSNC distributes tokens to avoid concentrated gaps. Training proceeds from packet-loss-free initialization to loss-aware learning and receiver refinement.

D. Spatiotemporal Token Packetization

  • Packet losses erase all tokens assigned to a packet, so packetization should disperse tokens across space and time to avoid concentrated missing regions.The resulting availability patterns affect how difficult token recovery is at the receiver.Source: D. Spatiotemporal Token Packetization · pp. 6–7
  • Spatial grouping can erase concentrated regions, temporal grouping can remove complete trajectories, and QLDS-3D spreads packet tokens across the full spatiotemporal grid.QLDS-2D distributes tokens spatially; QLDS-3D additionally disperses them over time.Source: D. Spatiotemporal Token Packetization · p. 7, Figure 5 · p. 7

E. Multi-Stage Training Strategy

  • Stage I trains the encoder, decoder, and detokenizer without packet loss using pixel, perceptual, and LFQ auxiliary objectives.Training uses a full-token warm-up before gradually reducing the token ratio to its target.Source: E. Multi-Stage Training Strategy · p. 7
  • Stage II jointly trains the network-state encoder and TokJSNC modules under simulated random, burst, and structured packet losses; Stage III refines the receiver adversarially.Stage II conditions transmission and token recovery on sampled loss-process statistics; Stage III updates the decoder, detokenizer, and discriminator while other modules remain fixed.Source: E. Multi-Stage Training Strategy · p. 7

IV. EXPERIMENTS

The experiments evaluate offline reconstruction and end-to-end streaming across UVG videos, comparing WebRTC-integrated codecs with loss-resilient methods under shared network conditions. They measure perceptual quality and playback continuity using a common deadline-based stall criterion.

Evaluation Datasets

  • Streaming evaluation uses seven UVG sequences at 30 fps, with a 300-frame warm-up followed by 153 measured frames per run.Offline codec evaluation uses the first 153 frames of each video.Source: Evaluation Datasets · p. 8

Compared Methods

  • The comparison includes WebRTC-based H.264, H.265, Cosmos, and DCVC-RT, alongside native-pipeline GRACE and TokJSNC.TokJSNC and Cosmos packetize quantized token symbols without entropy coding; TokJSNC operates at approximately 270 kbps at the application layer.Source: Compared Methods · p. 8

Evaluation Metrics

  • A frame counts as recovered only when available within 200 ms and at least 18 dB RGB PSNR; a continuous failure interval of 200 ms or longer counts as a stall.The evaluation reports stall ratio, average stall duration, stall-event count, and average playback frame rate.Source: Evaluation Metrics · p. 8

Network Setup

  • End-to-end tests use a 1000 kbps bottleneck, 275 kbps application budget, 20 ms one-way delay, 1200-byte payloads, and a 200 ms frame deadline.The source rate is 30 fps; TokJSNC is paced within the same application budget, and its measured wire rate averages approximately 284 kbps.Source: Network Setup · p. 9

B. Codec Performance under Loss-Free Delivery

Under loss-free delivery, TokJSNC offers rate-controllable reconstruction with stronger perceptual and distributional metrics than H.264 and H.265 at comparable medium and high rates, but lower MS-SSIM. A representative playback timeline distinguishes displayed from stalled intervals.

B. Codec Performance under Loss-Free Delivery

  • At comparable medium and high rates, TokJSNC has lower LPIPS, DISTS, and FID than H.264 and H.265, while its MS-SSIM remains lower.Increasing TokJSNC's nominal token ratio β improves reconstruction quality across all four metrics; the evaluation excludes packetization, transport, and recovery.Source: B. Codec Performance under Loss-Free… · p. 9
  • The codec comparison measures loss-free reconstruction without packetization, network transport, or packet-recovery mechanisms.TokJSNC's operating points vary by nominal token ratio β.Source: B. Codec Performance under Loss-Free… · p. 9
  • The representative Beauty playback timeline marks displayed intervals in blue and stalls in orange under 25% random and burst packet loss.H.264, H.265, DCVC-RT, and Cosmos use WebRTC pipelines; GRACE uses its native loss-resilient pipeline.Source: Figure 10 · p. 10

C. End-to-End Performance under Packet Loss

TokJSNC sustains playback and degrades smoothly under random and burst packet loss, with QLDS-3D dispersing missing tokens across space and time. It also produces coherent reconstructions and avoids reliance on FEC or retransmission.

Reconstruction Quality

  • Under random loss, TokJSNC with QLDS-3D has comparable MS-SSIM and generally lower LPIPS, DISTS, and FID than GRACE over most of the evaluated loss range.Under burst loss, it maintains smooth quality degradation, with MS-SSIM competitive with or better than baselines and generally lower LPIPS, DISTS, and FID.Source: Reconstruction Quality · pp. 10–11
  • QLDS-3D provides the strongest robustness among TokJSNC packetization variants by dispersing tokens across all three spatiotemporal dimensions.The evaluation reports smooth degradation up to 70% loss even though training sampled packet-loss ratios only from [0, 0.3].Source: Reconstruction Quality · p. 10

Playback Continuity

  • Across random and burst losses, TokJSNC achieves the lowest or tied-lowest stall ratio and relatively short, infrequent interruptions without FEC or retransmission.At the same average loss ratio, burst loss generally causes higher stall ratios and lower average playback frame rates than random loss.Source: Playback Continuity · p. 11
  • At 70% packet loss, TokJSNC sustains 23 fps under random loss and 22 fps under burst loss, while the strongest baseline reaches 12 and 8 fps, respectively.TokJSNC maintains low stall ratio and high playback frame rate without achieving continuity through aggressive frame skipping.Source: Playback Continuity · p. 12

Visual Quality Comparison

  • At 25% random packet loss, TokJSNC preserves more coherent structures and fine details, with the lowest LPIPS in all illustrated examples at a lower transmission rate than most baselines.The frame comparison annotates bitrate, relative bitrate difference, and LPIPS.Source: Figure 11 · p. 11, Visual Quality Comparison · p. 12
  • Across consecutive Jockey frames under 40% random loss, TokJSNC maintains stable color, scene structure, and motion without visible error propagation or frame freezing.The comparison contrasts persistent color shifts, mosaic artifacts, frame freezing, and blurring in other methods.Source: Figure 12 · p. 11, Visual Quality Comparison · p. 12

D. Model Complexity and Processing Latency

TokJSNC sustains stronger playback continuity under lossy conditions while incurring added model and computational overhead; ablations attribute benefits to content-adaptive budgeting and network-state conditioning.

D. Model Complexity and Processing Latency

  • TokJSNC incurs greater model and computational overhead than conventional and lightweight neural baselines, primarily because of loss-resilient token recovery.The paper identifies this as a computational tradeoff against improved reconstruction robustness and playback continuity.Source: D. Model Complexity and Processing… · p. 12

E. Ablation Studies

  • Under rate-matched conditions, network-state conditioning consistently improves reconstruction quality across random, burst, structured, and measured loss patterns.The gain is particularly evident on the measured Mahimahi trace, whose loss characteristics deviate from simple synthetic models.Source: Network State Prior · p. 12
  • At matched average transmission rates, adaptive token budgeting improves all four reconstruction metrics over assigning every clip a fixed budget.The largest improvement is in FID.Source: Adaptive Token Budgeting · p. 12
  • The ablations identify complementary roles: content-adaptive budgets improve representation efficiency, while network-state conditioning improves robustness across packet-loss conditions.Source: Adaptive Token Budgeting · p. 13

V. CONCLUSION

  • Above 22 fps at 70% packet loss, TokJSNC exceeds the strongest baseline, which remains below 12 fps.The conclusion also reports improved perceptual reconstruction quality across diverse packet-loss conditions and substantially better continuity under random and burst loss.Source: §V · p. 13

Related papers

Loading 2610.21166v1…