PaperlayerView exact revision on arXiv

Source-linked AI summary

Toward Generative Video Communication: A Dual-Stream Digital Transmission Framework

Bingyan Xie, Longyu Zhou, Tianhao Liang, Yongpeng Wu, Zehui Xiong, Wenjun Zhang, Tony Q. S. Quek

arXiv:2609.36565v1cs.MM

TL;DR

Wireless video communication must balance bitrate efficiency, robustness, and perceptual quality, while existing semantic methods are mostly reconstruction-oriented. DGVC combines a source-grounded traditional link with a generative link, coordinating them for receiver-side generation. In a case study, it achieves favorable LPIPS and FVD performance across a wide range of CBRs on UVG and QST compared with Promptus, GLC, VVC, and H.265.

  • Problem

    Traditional video communication struggles to jointly balance bitrate efficiency, robustness, and perceptual quality, while most video semantic communication remains reconstruction-oriented.

    Source: §I · p. 1

  • Method

    DGVC independently transmits a source-grounded TL reference and compact GL information, with the reference constraining diffusion-based receiver generation through ControlNet.

    Source: §I · pp. 1–2

  • Results

    Across a wide range of CBRs on UVG and QST, DGVC achieves favorable LPIPS and FVD performance compared with Promptus, GLC, VVC, and H.265.

    Source: C. Results Analysis · p. 7

  • Takeaways & Limitations

    DGVC provides a bitrate-adaptive framework combining source-grounded reference delivery with diffusion-based generation for perceptually efficient video communication.

    Source: §VI · p. 8

    The case study demonstrates the implemented ControlNet-based fusion, but does not isolate ControlNet’s gain from that of the dual-stream architecture.

    Source: C. Results Analysis · p. 7

Abstract

from arXiv · show

Generative video communication has shown promise for bandwidth-constrained wireless transmission and has the potential to support personalized content delivery. In this article, we propose a dual-stream digital generative video communication (DGVC) framework that integrates a traditional digital link with a generative link. The traditional link provides source-grounded visual references, while the generative link conveys compact semantic and perceptual information for receiver-side generation. We further discuss three bandwidth-dependent operating regimes and key technologies for dual-stream coordination, synchronization, reliability, and latency control. A practical case study demonstrates the perceptual and temporal-quality benefits of DGVC under wireless fading channels. Finally, we discuss open challenges and future research directions for generative video communication.

I. Introduction T

Traditional and semantic video communication face trade-offs in bitrate efficiency, robustness, and perceptual quality, while existing semantic methods are mostly reconstruction-oriented. DGVC jointly transmits a source-grounded reference and generative information, coordinating the two digital streams according to network conditions.

I. Introduction T

  • Traditional pixel-oriented links struggle to balance bitrate efficiency, robustness, and perceptual quality, while most video semantic communication remains reconstruction-oriented.Semantic communication transmits compact spatiotemporal representations, but receivers primarily decode features rather than generate visual content.Source: §I · p. 1
  • Prompt- and token-based generative methods support ultra-low-bitrate delivery, but single-stream methods may deviate from the source when transmitted conditions are insufficient or corrupted.Reference-guided diffusion can constrain generation, but typically assumes a locally available reference.Source: §I · p. 1
  • DGVC jointly transmits a low-bitrate source-grounded reference and compact generative information, using the reference to constrain receiver-side generation through ControlNet.The GL guides the diffusion backbone, while the TL reference constrains spatial structure and source-specific appearance.Source: §I · pp. 1–2
  • The TL and GL are independently source-coded, channel-protected digital streams coordinated according to transmission budget, channel condition, latency, and stream reliability.The proposed framework characterizes three operating regimes and coordinates the streams at the segment level.Source: §I · p. 2

II. Design and Applications of DGVC Framework

The section presents the DGVC architecture and its applications.

II. Design and Applications of DGVC

  • The section presents the architecture and applications of DGVC.Source: §II · p. 2

A. Architecture of DGVC

DGVC combines a low-bitrate traditional-link reference with compact generative-link conditioning for receiver-side video generation. Its link roles adapt to bandwidth, while receiver-side fusion and segment-level control account for reference reliability and system constraints.

A. Architecture of DGVC

  • The TL delivers a low-bitrate VVC/H.265 reference carrying source-grounded spatial, appearance, and motion cues, while the GL sends compact information for semantic and perceptual enhancement.The GL representation may be a text prompt, continuous embedding, discrete visual token, or hybrid, trading off interpretability, fidelity, bitrate, and robustness.Source: A. Architecture of DGVC · p. 2
  • The GL prompt guides Stable Diffusion semantically, while ControlNet converts the decoded TL reference into multi-scale structural features that constrain layout, geometry, appearance, and source identity.The reference-conditioning coefficient can be strengthened for reliable references or weakened to avoid propagating distortions or coding artifacts.Source: A. Architecture of DGVC · p. 2
  • The current case study uses only the reconstructed low-bpp TL frame; codec-side motion and reliability cues are proposed for future conditioning work.Candidate cues include motion vectors, reference indices, block partitions, quantization parameters, and residual-energy maps.Source: A. Architecture of DGVC · p. 2
  • A segment- or GOP-level controller selects bitrate allocation, protection, GL representation size, ControlNet strength, and generation settings under bandwidth, latency, decoding, and computation constraints.It uses throughput, channel and packet-error conditions, buffer state, playback deadlines, and stream reliability to balance fidelity, perceptual quality, semantic consistency, temporal stability, and cost.Source: A. Architecture of DGVC · p. 2

B. Applications of DGVC

III. Promising Technologies for DGVC

DGVC introduces coordination challenges because its two streams can overlap in information yet differ in reliability and timing. The section identifies four technology directions for managing this dual-stream setting.

III. Promising Technologies for DGVC

  • The TL and GL require joint coordination of information partitioning, channel protection, synchronization, scheduling, and latency, especially when overlapping information has different reliability and timing.Source: §III · p. 4
  • The four identified directions are adaptive coordination and semantic representation, reference-prompt synchronization, reliability-aware fusion and asymmetric protection, and latency coordination.Source: §III · p. 4
  • The coordination challenge is specific to DGVC’s two-stream setup, which must manage both overlapping information and mismatched stream reliability and timing.Source: §III · p. 4

A. Adaptive Dual-Stream Coordination and Semantic Representation Design

Adaptive coordination selects TL/GL operating modes and resource settings from network conditions and service targets. Control should operate at segment or GOP level to limit bitrate oscillation and temporal inconsistency.

A. Adaptive Dual-Stream Coordination and Semantic Representation Design

  • The controller selects operating mode, TL/GL bitrate allocation, stream protection, representation size, and generation settings using measured network state and service targets.Relevant inputs include throughput, channel quality, packet-error condition, buffer state, playback deadline, and stream reliability.Source: A. Adaptive Dual-Stream Coordination… · p. 4
  • An offline performance table can record candidate configurations, while an online controller selects among them using measured throughput, channel quality, buffer state, and service target.Source: A. Adaptive Dual-Stream Coordination… · p. 4
  • Coordination should occur by segment or GOP; GL-dominant operation pairs a lower-rate reference with stronger generative completion, while TL-dominant operation uses a higher-quality reference and GL enhancement.Frequent changes may cause bitrate oscillation and temporal inconsistency.Source: A. Adaptive Dual-Stream Coordination… · p. 4

B. Reference-Prompt Synchronization and Temporal Consistency

DGVC must synchronize TL references and GL representations despite different update granularities and processing delays. It coordinates stream reliability, protection, and generation latency to preserve temporal alignment while meeting playback deadlines.

B. Reference-Prompt Synchronization and Temporal Consistency

  • Different TL/GL update granularities and transmission, decoding, and generation delays can cause semantic misalignment, object displacement, or temporal flickering.Metadata can associate each GL representation with its TL frame or GOP, while codec-side cues can support correspondence and conditioning.Source: B. Reference-Prompt Synchronization… · pp. 4–5
  • Motion-aligned TL features and reused or interpolated GL representations can improve temporal consistency across adjacent frames.Reduce conditioning strength or refresh the TL reference when it becomes stale or unreliable.Source: B. Reference-Prompt Synchronization… · p. 5

C. Reliability-Aware Fusion and Asymmetric Error Protection

  • TL degradation can distort structure or motion cues, while GL errors can alter semantics, object identity, or appearance; separate reliability estimates can adapt each stream's contribution.The receiver can estimate TL reliability from decoding status, reconstruction quality, quantization parameters, and residual cues, and GL reliability from CRC results, representation distortion, or consistency with the decoded reference.Source: C. Reliability-Aware Fusion and… · p. 5
  • When both streams are unreliable, the receiver can use a conservative fallback; when one is reliable, it can suppress semantic drift or compensate for a degraded reference.Source: C. Reliability-Aware Fusion and… · p. 5
  • Stronger coding or retransmission priority should protect critical structural metadata, reference indices, prompt headers, quantization parameters, and high-importance embedding features.The streams' different error sensitivities motivate asymmetric protection.Source: C. Reliability-Aware Fusion and… · p. 5

D. Dual-Stream Latency Coordination and Mode Transition

  • The TL offers relatively predictable decoding latency, while GL adds prompt reconstruction and generative inference; GL enhancement is applied only if generation finishes before display.The TL reference can provide an immediate reconstruction, with TL output as fallback when generation misses the deadline.Source: D. Dual-Stream Latency Coordination… · p. 5
  • Segment-level smoothing, hysteresis, and minimum mode duration can stabilize transitions among GL-dominant, collaborative, and TL-dominant operation.Prompt reuse, reference refresh, and gradual ControlNet-strength adjustment can help maintain visual continuity; lightweight diffusion, model quantization, and accelerated inference can reduce latency mismatch.Source: D. Dual-Stream Latency Coordination… · p. 5
  • DGVC addresses resource allocation, reference-prompt synchronization, reliability-aware fusion and protection, and latency-aware mode coordination.Together, these issues are presented as enabling efficient and robust generative video delivery.Source: D. Dual-Stream Latency Coordination… · p. 5

IV. Case Study

The VOD case study implements DGVC with a low-bitrate TL reference and a compact GL prompt that condition receiver-side generation. The streams provide structural and semantic/perceptual guidance, respectively.

IV. Case Study

  • The case study evaluates the proposed DGVC framework for video-on-demand services.Source: §IV · p. 5
  • ControlNet strength is fixed in the current implementation; confidence-adaptive conditioning based on TL quality or channel reliability is left for future work.Source: §IV · p. 5

A. Network Structure

  • The TL sends the source frame through low-bpp VVC/H.265 compression and LDPC+QAM protection; its recovered reference supplies spatial and appearance cues to ControlNet.Source: A. Network Structure · p. 5
  • The implemented GL prompt is a continuous Stable Diffusion conditioning embedding, optimized against source frames with a frozen single-step diffusion backbone.Source: A. Network Structure · p. 5
  • The 77-by-1024 prompt embedding is represented by low-rank matrices U and V, with rank r selected from 1 to 32 according to the GL budget.For VOD, the matrices are optimized offline, only final matrices for selected key frames are transmitted, and intermediate-frame prompts are interpolated.Source: A. Network Structure · p. 5
  • The low-rank matrices use fitting-aware 8-bit quantization and error detection, LDPC coding, and QAM; after decoding, they reconstruct the prompt supplied to Stable Diffusion.The prompt supplies semantic and perceptual guidance, while the low-bpp TL frame provides structural constraints through ControlNet.Source: A. Network Structure · p. 6
  • ControlNet combines the low-bpp TL frame's structural guidance with the GL prompt's semantic and perceptual priors, while SD-Turbo provides single-step denoising.The current implementation does not add codec-side motion vectors, residuals, or block partitions as conditioning cues; these are described as possible future auxiliaries.Source: A. Network Structure · p. 6

B. Experimental Setups

The experimental setup evaluates DGVC on UVG and QST under Rayleigh fading, using digital wireless transmission components and perceptual and temporal-quality metrics. The setup also reports computation-cost and latency evaluation.

B. Experimental Setups

  • DGVC is evaluated on UVG and QST, with implementations in PyTorch 2.6.0 using RTX 5090 GPUs.Source: 1) Datasets · p. 6
  • The deployment uses VVC+LDPC for TL, Promptus for GL, Stable Diffusion conditioned by the recovered low-bpp TL frame, and Rayleigh fading channels.Offline low-rank prompt-factor optimization is reported separately from online communication and receiver-side generation latency; only final quantized factors count toward GL payload.Source: 2) Model Deployment Details · p. 6
  • The benchmarks are Promptus, GLC, and VVC/x265 with LDPC; 5G MCS, LDPC, QAM, channel bandwidth ratio, error detection, and retransmission model digital wireless delivery.Only correctly decoded bitstreams are forwarded to reconstruction, so the received GL payload matches the transmitted quantized representation.Source: 3) Comparison Benchmarks · p. 6
  • LPIPS measures frame-level perceptual quality and FVD assesses video-level temporal consistency; Fig. 5 compares DGVC with benchmarks on UVG and QST.Table I is identified as the evaluation of computation cost and latency; its caption supplies no individual values.Source: 4) Evaluation Metrics · pp. 6–7, Figure 5 · p. 7, Table I · p. 7

C. Results Analysis

DGVC performs favorably across tested bitrates and datasets, with visual improvements in source-dependent structure and local appearance. Its computation cost is comparable to Promptus+LDPC.

C. Results Analysis

  • Across UVG and QST and a wide range of CBRs, DGVC achieves favorable LPIPS and FVD performance versus Promptus, GLC, VVC, and H.265.Fig. 5 reports these comparisons; the supplied passage gives no metric values.Source: C. Results Analysis · p. 7
  • Visual examples show improved preservation of source-dependent structure and local appearance.The reported gains reflect the implemented ControlNet-based cross-stream fusion, not an isolated ControlNet effect.Source: C. Results Analysis · p. 7
  • With ControlNet-based fusion, DGVC has throughput and FLOPs comparable to Promptus+LDPC.The paper presents this comparison as evidence of DGVC's computational efficiency.Source: C. Results Analysis · p. 7

V. Challenges

Generative video communication must balance what is faithfully preserved against what may be generated, while protecting critical representation components against errors.

V. Challenges

  • DGVC faces paradigm-level challenges beyond the implementation technologies discussed in Sec. III.These challenges include deciding what to preserve versus generate and protecting critical representation components.Source: §V · p. 7

A. Communication Objective Beyond Pixel Reconstruction

  • The communication objective must balance fidelity preservation, semantic correctness, and perceptual realism under limited resources.A central open question is which content to preserve faithfully and which content may be flexibly generated.Source: A. Communication Objective Beyond… · p. 7

B. Error Propagation and Uncertainty Accumulation

  • Unequal error protection can prioritize critical components with lower code rates, more robust modulation, or higher HARQ priority.In the current retransmission-based DGVC setting, this maintains error-free delivery while reducing decoding failures and retransmission latency for critical information.Source: B. Error Propagation and Uncertainty… · p. 7

C. Lack of Unified Evaluation Metrics and Theory

Generative video communication still lacks unified evaluation principles and theoretical foundations, and generated content raises trustworthiness concerns. The paper concludes that DGVC combines reference delivery with diffusion-based generation for bitrate-adaptive, perceptually efficient communication.

C. Lack of Unified Evaluation Metrics and Theory

D. Trustworthiness and Responsible Deployment

  • Generated rather than directly reconstructed content heightens concerns about authenticity, accountability, explainability, and misuse, especially in safety-related applications.Source: D. Trustworthiness and Responsible… · p. 8

VI. Conclusion

  • DGVC combines a source-grounded reference link with diffusion-based generation for bitrate-adaptive, perceptually efficient video communication.The paper presents this as a practical transition toward content-oriented and intelligent video communication for 6G and beyond.Source: §VI · p. 8
Loading 2609.36565v1…