Generative-AI for XR Content Transmission in the Metaverse: Potential Approaches, Challenges, and a Generation-Driven Transmission Framework

I. INTRODUCTION

• Existing end devices have limited rendering capacity, edge servers have resource constraints, and cloud rendering adds unacceptable long-distance latency for real-time applications.The paper identifies the lack of a simple method to combine cloud and edge strengths for efficient XR transmission.Source: §I · p. 1
• GAI can generate complex content at edge servers instead of transmitting it from remote cloud servers, reducing network traffic and enhancing transmission efficiency.Source: §I · p. 1
• The paper proposes storing and rendering original XR content in the cloud while edge GAI generates essential content when network resources are insufficient.Generated content can include subsequent frames, objects, and background scenes.Source: §I · p. 1
• The paper discusses transmission challenges involving high-quality video and haptic feedback, and how GAI might address them alongside implementation issues.Source: §I · p. 1
• A DRL-based decision module is proposed to address decision-making challenges in the GAI-based XR content transmission framework.Source: §I · p. 1
• The case study evaluates subsequent-frame generation and reports higher normal frame ratio and lower end-to-end latency than baseline approaches.Normal frames meet XR quality and latency requirements.Source: §I · p. 1

A. Intelligent Adaptive Transmission for Large Volume of XR Content

• Volumetric video can require data rates up to 2.9 Gb/s, so current networks cannot transmit all XR content at full quality and require adaptive prioritization.The proposed prioritization factors include user behavior, content context, regions of interest, and network conditions.Source: A. Intelligent Adaptive Transmission… · p. 2

B. Seamless Interaction

• Haptic feedback can require 1 ms RTT, which is difficult to achieve when content travels from remote cloud servers.Prediction technologies can pre-render content outside the user's view and predict haptic content based on user behavior.Source: B. Seamless Interaction · p. 2

C. Consistently Stable Transmission in Dynamic Network for XR Content

• Fluctuating bandwidth, latency, and packet loss make stable XR transmission difficult, while buffering that helps traditional content cannot meet XR's need for uninterrupted connections.Source: C. Consistently Stable Transmission in… · p. 2

D. Challenges of Ubiquitous Access and Device Heterogeneity in XR Transmission

• XR transmission must support varied devices and access networks, adapting to differences in device computation and network performance.Examples include AR glasses, VR headsets, and haptic gloves, with Wi-Fi indoors and 5G outdoors.Source: D. Challenges of Ubiquitous Access and… · p. 2

III. POTENTIAL APPROACHES AND CHALLENGES OF UTILIZING GAI FOR XR CONTENT TRANSMISSION

• The paper reviews GAI's potential for XR transmission and discusses implementation challenges.Source: §III · p. 2

A. GAI Models

• GANs, Transformer Models, Diffusion Models, and LGAIMs [10] are reviewed, with model choice depending on task requirements, resources, and application needs.The paper says Table I summarizes their characteristics, advantages, limitations, and typical applications.Source: A. GAI Models · p. 2, Table I · p. 3

B. Potential Directions of Utilizing GAI for XR Content Transmission

• The paper identifies four potential GAI approaches for XR transmission that can be used independently or jointly according to application requirements and network conditions.Fig. 1 illustrates these approaches.Source: B. Potential Directions of Utilizing… · p. 2, Figure 1 · p. 3

1) Subsequent Frame Generation:

• Edge GAI can generate subsequent XR frames when network conditions cannot meet transmission requirements, using only text descriptions or base frames from the cloud.This approach is presented as a way to reduce bandwidth use and support continuity through network fluctuations.Source: 1) Subsequent Frame Generation · p. 2

2) Cross-modal Data Generation:

• Cross-modal generation can predict haptic feedback, such as texture and force from a video of interaction with a plastic ball, for instant edge delivery.Challenges include spatial and temporal coherence; timestamps in packet headers can improve synchronization.Source: 2) Cross-modal Data Generation · pp. 2–3

3) Outpainting for XR content:

• Cloud or edge servers can preemptively outpaint unseen virtual areas from the current scene, avoiding detailed user-behavior data transmission but requiring accurate prediction of what users will view next.Collecting and sending behavior information for rendering extended areas adds latency and can disrupt a seamless experience.Source: 3) Outpainting for XR content · p. 3

4) Selective Object Rendering:

• Selective object rendering prioritizes high-quality objects in the user's FoV while the cloud sends text descriptions for less important objects, which may be generated at lower fidelity.The paper states that this approach also applies in 6DoF, where users focus on certain objects.Source: 4) Selective Object Rendering · pp. 3–4

C. Challenge of Integrating GAI for XR Content Transmission

• Using GAI for XR content transmission offers promising solutions to overcome current network bottlenecks, but significant challenges must be addressed to fully realize its potential.Source: C. Challenge of Integrating GAI for XR… · p. 4

1) High Computational Demands Challenge:

• Limited edge resources make allocating computation across high-quality, long XR content generation tasks a central challenge.Generation demands increase with content quality and length.Source: 1) High Computational Demands Challenge · p. 4

2) Long Inference Latency Challenge:

• Significant GAI inference latency conflicts with strict real-time requirements in applications such as gaming and surgery.Even minor delays can degrade user experience.Source: 2) Long Inference Latency Challenge · p. 4

3) Decision-Making for Content Generation Over Dynamic Networks Challenge:

• Dynamic networks complicate decisions about cloud-edge cooperation, generated content quality and length, and when to activate GAI models.Source: 3) Decision-Making for Content… · p. 4

4) Prediction Accuracy:

• Probabilistic GAI outputs can differ from original content, degrading user experience and cross-modal spatial coherence when accuracy is low.Variations may occur in textures or colors.Source: 4) Prediction Accuracy · p. 4

5) Security and Privacy Challenge:

• XR inference inputs may expose biometric, preference, or behavioral data to cloud or edge servers, creating risks of unauthorized collection, analysis, or misuse [13].Encryption, anonymization, and strict access control are identified as needed protections.Source: 5) Security and Privacy Challenge · p. 4

IV. GAI-BASED CONTENT TRANSMISSION FRAMEWORK

• The proposed GAI-based framework links potential approaches and challenges, and comprises content captioning, content generation, and DRL-based decision modules.Source: §IV · p. 4

A. Cloud-Edge Collaboration Architecture

• Cloud and edge servers assign tasks according to network conditions, edge compute availability, and XR QoS requirements; edges generate essential content when direct cloud transmission cannot meet them.Otherwise, original XR content is transmitted directly from cloud servers to users.Source: A. Cloud-Edge Collaboration Architecture · p. 4

B. Content Generation Module

• Edge-based generation can supply subsequent frames, selected objects, and background scenes when remote-cloud transmission is inefficient; GANs suit less detailed content, while diffusion models suit realistic interactive objects [13].For subsequent frames, GANs are described as faster for low-latency scenarios, while diffusion models offer higher quality with more processing time.Source: B. Content Generation Module · p. 4

• GANs support faster subsequent-frame generation without iterative refinement, while diffusion models provide higher quality at increased processing time.Generated frames can maintain the immersive experience when network conditions cannot support transmission of original XR content from remote cloud servers.Source: • Long Inference Latency Challenge · p. 5
• Transformer-based models can generate one modality from another, reducing the need to transmit every modality when network resources are limited.This is described as supporting continuity and interaction in the XR experience.Source: • Long Inference Latency Challenge · p. 5
• GANs can quickly generate less-detailed background scenes for less important objects outside the region of interest, saving rendering resources.Source: • Long Inference Latency Challenge · p. 5
• Outpainting can prepare extended areas in advance while cloud servers transmit only region-of-interest content; diffusion models suit this detailed generation because user movement is typically on the order of seconds.Source: • Long Inference Latency Challenge · pp. 5–6

C. Content Caption Module

• The content caption module converts XR content and user actions into text descriptions to improve the similarity and accuracy of generated content.It extracts visual elements, actions, and context from frames and uses a transformer to capture temporal dependencies and produce coherent descriptions.Source: C. Content Caption Module · p. 6

D. DRL-based Decision Module

• The DRL-based transmission mode module models decision-making as an MDP and uses multiple agents to make different decisions dynamically.The agents support decisions for different GAI-based transmission tasks.Source: D. DRL-based Decision Module · p. 6
• Agents generally observe available bandwidth and compute, traffic load, XR content type, and user behavior; cross-modal generation also observes the current modality.The state set is S = {Ba, Ca, L, T, U}, with current modality Mc additionally considered for cross-modal generation.Source: D. DRL-based Decision Module · p. 6
• Task-specific actions select generated-frame start, length, and quality; objects to generate; whether to outpaint; or cross-modal output modality and quality.The subsequent-frame agent chooses frame range and quality, while the outpainting agent makes a binary perform-or-not choice.Source: D. DRL-based Decision Module · p. 6
• The DRL module proactively activates GAI before network conditions deteriorate, mitigating long inference latency while its limited state and action dimensions enable real-time inference.The figure reports performance evolution across frame distributions, traffic load, latency, and frame rate.Source: D. DRL-based Decision Module · pp. 6–7, Figure 5 · p. 7

V. A CASE STUDY

• The case study focuses on subsequent-frame generation, using GTA V gameplay and Runway GEN-2 when original content cannot be transmitted; D3QN is the selected DRL model.The simulation runs on a Linux server with dual Intel Xeon Platinum 8352V CPUs and four RTX 4090 GPUs.Source: §V · p. 7, A. Experiment Settings · p. 7

A. Experiment Settings

• Evaluation measures normal-frame ratio, end-to-end latency, and frame rate against CBWFQ, Pred-only, and GAN-based approaches.Frames are degraded if resolution falls because of insufficient bandwidth or if their SSIM relative to the original is below 0.9.Source: A. Experiment Settings · p. 8
• The simulation models a 100-second session over a 26 Mbps link, with 2K 60-FPS scenes requiring at least 16 Mbps and randomly added 5-to-25-Mb traffic flows.H.266 is used for compression, and the simulated network places original content in the cloud and GAI models at the edge.Source: A. Experiment Settings · p. 8

B. Experiment Results

• Multiple simulation runs produced consistent outcomes; Fig. 5(b) reports averages, while the other panels show randomly selected runs.Source: B. Experiment Results · p. 8
• 84% normal frame ratio: the proposed approach consistently exceeded GAN-based and Pred-only approaches, which produced more degraded frames.The paper attributes the result to the DRL-based decision module and the high quality of frames generated by DMs.Source: B. Experiment Results · p. 8
• Above 90% normal frame ratio at around 20 Mb traffic load, versus below 65% for baseline approaches, with DRL adjusting generated-frame quality and count to available bandwidth.Source: B. Experiment Results · p. 8
• Around 37 ms end-to-end latency with minimal fluctuations; GAI-based approaches exceeded 50 FPS, while CBWFQ fell below 27 FPS in the worst case.Among the approaches, GAN-based delivery achieved the best frame rate, attributed to faster generation than DMs.Source: B. Experiment Results · p. 8

1) Scalability:

• Scalability is constrained by the computational resources GAI models need for inference, although the framework generates only affected content when transmission requirements are unmet.Source: 1) Scalability · p. 8

2) Limitations:

• The approach depends on GAI prediction accuracy, requires content pre-generation, and needs network-change detection or prediction faster than millisecond-level inference.Source: 2) Limitations · p. 8
• Edge generation may involve sensitive user data, including text descriptions and original-content first frames, creating privacy risks for which encryption, access control, and anonymization are needed.Source: 2) Limitations · p. 8

3) Potential Impacts:

• The framework addresses XR transmission challenges including large data volumes, seamless interaction, dynamic-condition stability, and access across diverse devices.Source: 3) Potential Impacts · p. 8

VI. CONCLUSION AND FUTURE DIRECTIONS

• The case study validates the framework’s effectiveness for XR transmission from the content-generation perspective, including generating subsequent frames.Source: §VI · p. 9
• Future research should reduce generated-original content discrepancies, improve intelligent real-time low-latency network decisions, and explore lightweight GAI models to lower computational costs.Source: §VI · p. 9