Diffusion-based Reinforcement Learning for Edge-enabled AI-Generated Content Services

Hongyang Du; Zonghang Li; Dusit Niyato; Jiawen Kang; Zehui Xiong; Huawei Huang; Shiwen Mao

Source-linked AI summary

Diffusion-based Reinforcement Learning for Edge-enabled AI-Generated Content Services

Hongyang Du, Zonghang Li, Dusit Niyato, Jiawen Kang, Zehui Xiong, Huawei Huang, Shiwen Mao

arXiv:2303.13052v3Mar 23, 2023cs.NI

TL;DR

The paper addresses the challenge of making resource-intensive AIGC accessible while selecting ASPs for diverse users under uncertain conditions. It introduces edge-based AaaS, the diffusion-model AGOD algorithm, and D2SAC, which outperforms seven representative DRL algorithms in ASP selection and standard control tasks. The authors also identify the absence of real-world edge-enabled ASP-selection datasets as a future validation need.

Problem
Resource-intensive AIGC models limit accessibility, while diverse users and variable model and server capabilities complicate personalized ASP selection.
Method
The paper deploys AIGC models through wireless-edge AaaS and uses diffusion-based AGOD within DRL to create D2SAC for ASP selection.
Results
D2SAC outperformed seven representative DRL algorithms in the ASP selection problem and various standard control tasks.
Takeaways & Limitations
AGOD can potentially extend to various wireless-network optimization problems, while D2SAC provides a practical approach for ubiquitous AIGC services in the Metaverse.
Takeaways & Limitations
The authors plan to collect and use real-world edge-enabled ASP-selection datasets to validate and refine the algorithm in practical scenarios.

Abstract

from arXiv · show

As Metaverse emerges as the next-generation Internet paradigm, the ability to efficiently generate content is paramount. AIGenerated Content (AIGC) emerges as a key solution, yet the resource intensive nature of large Generative AI (GAI) models presents challenges. To address this issue, we introduce an AIGC-as-a-Service (AaaS) architecture, which deploys AIGC models in wireless edge networks to ensure broad AIGC services accessibility for Metaverse users. Nonetheless, an important aspect of providing personalized user experiences requires carefully selecting AIGC Service Providers (ASPs) capable of effectively executing user tasks, which is complicated by environmental uncertainty and variability. Addressing this gap in current research, we introduce the AI-Generated Optimal Decision (AGOD) algorithm, a diffusion model-based approach for generating the optimal ASP selection decisions. Integrating AGOD with Deep Reinforcement Learning (DRL), we develop the Deep Diffusion Soft Actor-Critic (D2SAC) algorithm, enhancing the efficiency and effectiveness of ASP selection. Our comprehensive experiments demonstrate that D2SAC outperforms seven leading DRL algorithms. Furthermore, the proposed AGOD algorithm has the potential for extension to various optimization problems in wireless networks, positioning it as a promising approach for future research on AIGC-driven services. The implementation of our proposed method is available at: https://github.com/Lizonghang/AGOD.

1 INTRODUCTION

Metaverse-oriented AIGC must overcome deployment costs and user diversity while remaining accessible and personalized. The paper addresses these goals with edge-based AaaS, AGOD, and D2SAC for ASP selection.

Large AIGC models require substantial datasets, complex architectures, high-end hardware, and specialized software, limiting widespread Metaverse access.
Metaverse users have diverse cultural backgrounds, languages, preferences, and needs, making customized AIGC services difficult to provide.
AaaS deploys trained AIGC models on wireless edge servers so users can submit tasks without distributing models to their devices.
Selecting an ASP is challenging because models, user preferences, server capacities, and task requirements vary across the service environment.
AGOD uses diffusion models to generate optimal decisions under environmental uncertainty and variability.
D2SAC applies AGOD within DRL and outperforms seven representative DRL algorithms in ASP selection and standard control tasks.

2 RELATED WORK

The related work situates the paper at the intersection of AIGC for Metaverse, diffusion-model optimization, and DRL. Building on limitations in existing approaches, it proposes AGOD and D2SAC for ASP selection.

AIGC supports rapid digital-content creation for Metaverse applications, extending beyond traditional user-generated and professional content.
Diffusion models learn latent data structure through a forward diffusion process and reverse denoising process, supporting generation and optimization applications.
DRL uses deep neural networks to represent complex state spaces and adapt to sequential decision-making problems such as online ASP selection.
The paper builds on these areas by introducing AGOD and applying it within DRL to form D2SAC for ASP selection.

3 AIGC SERVICES IN WIRELESS NETWORKS

The paper frames wireless-edge AaaS as a way to make AIGC accessible while requiring real-time ASP selection under resource, quality, and uncertainty constraints. Its formulation combines task assignment, human-aware utility, and diffusion-service costs.

AIGC Service Architecture: AaaS deploys AIGC models on edge servers, enabling users on resource-constrained devices to request services without local model deployment.
AIGC Service Architecture: AaaS must handle customized resource requests, human-subjective task evaluation, and varying ASP model qualities and server capacities.
AIGC Service Architecture: Diffusion-model inference energy cost increases with denoising steps, with an initial expenditure attributed to model initialization.
ASP Selection Problem: The assignment problem is NP-complete, and violating resource constraints can crash an ASP and terminate or restart its running tasks.
ASP Selection Problem: ASP selection is a resource-constrained task-assignment problem requiring real-time decisions while reserving resources for future tasks.
Human-Aware Utility: Task utility is defined as ui(Tj) = G(Fi(Tj)), combining an ASP’s AIGC output with a human-aware quality assessment function.
Human-Aware Utility: Different ASPs can produce substantially different human-aware utility values, underscoring the importance of selecting a well-suited provider.

4 AI-GENERATED OPTIMAL DECISION

AGOD treats ASP selection as a discrete decision problem and uses conditioned diffusion to generate optimal decision probabilities from Gaussian noise. Integrated with SAC, AGOD forms D2SAC for exploratory, efficient ASP selection under uncertain environments.

Conditioned diffusion: AGOD starts from Gaussian noise and uses a conditioned reverse diffusion process to recover the optimal decision scheme for the current environment.The forward process mathematically relates the target decision distribution to progressively noisier states, while the reverse process generates the decision from noise.
Motivation: AGOD addresses discrete ASP selection, where the finite solution space creates a combinatorial optimization challenge for conventional continuous-variable techniques.The solution space can grow exponentially with the number of discrete variables.
Decision representation: The decision scheme is represented as probabilities for selecting decisions, conditioned on the environment state and task-related constraints.The environment includes factors such as user tasks and current resource status.
Training: Because optimal decision labels are generally unavailable, AGOD learns exploratorily by maximizing decision value rather than minimizing the supervised MSE objective used by DDPM.Reparameterization is used to address the inability to back-propagate through stochastic sampling.
Reverse process: The denoising network generates noise conditioned on the observation, with hyperbolic tangent scaling limiting the generated noise used to reconstruct the decision probabilities.The reconstructed output is passed through softmax so its elements represent action-selection probabilities.
D2SAC integration: AGOD is integrated into SAC to create D2SAC, combining SAC’s stable policy learning with diffusion-based exploration and decision generation.Experiments report that D2SAC outperforms seven representative DRL algorithms in ASP selection and standard control tasks.

5 DIFFUSION-BASED REINFORCEMENT LEARNING

The paper models ASP selection as a sequential decision problem and uses AGOD as the diffusion-based policy within D2SAC. The architecture combines reinforcement-learning components to optimize utility while accounting for resource constraints, crashes, exploration, and delayed feedback.

Problem Modeling: ASP selection is formulated as an MDP because task arrivals, resource allocation, and ASP outcomes are sequential and unpredictable.The formulation supports real-time decisions that balance current assignments with future resource availability.
Problem Modeling: The state encodes the arriving task’s required resources and completion time together with the current resource status of all ASPs.This state representation supplies the information needed for assignment decisions.
Problem Modeling: Each action assigns the current Metaverse task to one of I ASPs, with AGOD producing selection probabilities over the available providers.During evaluation, the ASP with the highest probability is selected.
Problem Modeling: The reward combines AIGC quality utility with penalties for ASP crashes and disrupted ongoing tasks.The quality component compares task utility with a noise-sample baseline, while the penalties discourage unstable assignments.
Algorithm Architecture: D2SAC uses AGOD as its diffusion-based actor, entropy regularization for exploration, replay memory for delayed rewards, and double critics for conservative value estimates.The architecture also includes target actor and critic networks, an experience replay memory, and the environment; its added denoising computation is O(EC|θ| (T −1)).
Algorithm Architecture: D2SAC retains SAC’s space complexity but adds reverse-diffusion computation to improve performance and convergence.The additional computational cost arises from the T denoising steps in the reverse diffusion process.

6 EXPERIMENTS AND INSIGHTS

Experiments evaluate D2SAC for ASP selection against DRL benchmarks and examine its learning behavior, utility–crash trade-offs, and denoising-step effects. D2SAC achieves strong rewards and convergence while its exploration and computational behavior depend on diffusion settings.

Experimental Setup: Experiments compare D2SAC with seven DRL benchmarks and evaluate it in an ASP-selection environment and standard control tasks.The setup uses 20 ASPs with randomized resource capacities, RePaint3 for AIGC, PIQ for human-aware quality assessment, and simulated task arrivals.
Numerical Results: D2SAC achieves the highest training and test rewards, reaches the Crash Avoid baseline after 190 training steps, and trains in 1.3 hours.The comparison also reports robustness across task arrival rates and superior characteristics on standard Gym control tasks.
Learning Process: D2SAC surpasses Round Robin after about 45 training steps and exceeds the Crash Avoid baseline by 80 steps while progressively approaching the prophet policy.Its learning process moves from random decisions toward policies that balance task completion, load balancing, crash prevention, and user utility.
New and Advanced Abilities: D2SAC outperforms DQN on utility by selectively discarding low-value tasks, while its 1.1% crash rate and lost utility of 5 exceed PPO’s 0.7% and 4.The results show a trade-off between maximizing utility and avoiding crashes; policies with near-zero crashes can deliver lower utility.
Denoising-Step Impact: The optimal denoising step appears to be T = 5: reward rises and then falls as T increases, whereas training time consistently increases.The authors report that small T can maintain satisfying reward and high computational efficiency, contrary to the usual expectation that larger T improves accuracy.
Exploration and Entropy: Increasing denoising steps reduces exploration and can produce suboptimal solutions, while entropy temperature α = 0.05 balances exploration and exploitation.D2SAC retains uncertainty during denoising, but excessive denoising makes action distributions more peaked and limits exploration.

7 CONCLUSION

The paper proposes an edge-enabled AaaS architecture and the AGOD-based D2SAC algorithm for efficient ASP selection under environmental uncertainty. Experiments show D2SAC outperforms seven representative DRL algorithms, while AGOD may extend to other wireless-network optimization problems.

The proposed edge-enabled AaaS architecture supports ubiquitous AIGC functionality in wireless edge networks.
AGOD uses a diffusion model within DRL to create D2SAC for efficient and optimal ASP selection under environmental uncertainty and variability.
D2SAC outperformed seven representative DRL algorithms in the ASP selection problem and various standard control tasks.
AGOD may be applied to various optimization problems in wireless networks, while future work will validate and refine the algorithm using real-world edge-enabled ASP-selection datasets.