Diffusion-based Reinforcement Learning for Edge-enabled AI-Generated Content Services

Introduction

• The introduction frames the goals as ubiquitous AIGC access and human-centric services that maximize user utility, addressing deployment costs and diverse user needs.Large AIGC models require substantial resources, while personalization requires accounting for varied user backgrounds, languages, and preferences.Source: §1 · pp. 1–2
• ASP selection must match users’ requirements and preferences with deployed models’ task capabilities and edge servers’ computing resources.Users differ in interests, and AIGC models vary in the tasks they perform well.Source: §1 · p. 2
• The proposed AaaS architecture deploys AIGC models on wireless edge servers, reducing user-device computational burden while enabling flexible, scalable service delivery.Users upload requests to edge servers, which execute tasks and return results.Source: §1 · p. 2
• AGOD adapts diffusion models to generate optimal ASP-selection decisions under environmental uncertainty and variability.The paper motivates AGOD by noting that DRL exploration–exploitation trade-offs may lead to suboptimal policies.Source: §1 · p. 2
• D2SAC combines diffusion-based AGOD with DRL for ASP selection and is reported to outperform DQN, DRQN, Prioritized-DQN, Rainbow, REINFORCE, PPO, and SAC on ASP selection and standard control tasks.The introduction describes the comparisons as extensive experiments.Source: §1 · p. 2

2.1 Artificial Intelligence-Generated Content in Metaverse

• AIGC can automate rapid digital-content creation for the Metaverse, while prior work explores blockchain-aided AIGC services and edge-product lifecycle management; broad access to large models remains unresolved.Source: §2.1 · p. 2

2.2 Diffusion Model in Optimization

• Diffusion models have applications in image and video generation and molecule design [5] [13]; DQL [24] is limited to offline imitation learning, and both DQL [24] and the incentive mechanism target continuous action spaces.The text notes that offline datasets for online communication scheduling are impractical and that these continuous-action methods do not apply to discrete action spaces.Source: §2.2 · p. 3

2.3 Deep Reinforcement Learning

• DRL captures state representations and adapts to unforeseen online tasks, but high computational requirements can hinder optimal decisions in large state or action spaces.Source: §2.3 · p. 3
• The proposed AaaS approach uses diffusion to generate optimal ASP-selection decisions in AGOD and applies it in DRL to form D2SAC.This addresses limitations in existing research on ASP selection.Source: §2.3 · p. 3

3.1 AIGC-as-a-Service

• Edge-hosted AaaS lets users submit generation requests and customize required computation, enabling on-demand access from resource-constrained devices.The section notes that AIGC models are large and difficult to train and deploy, motivating their placement on network edge devices.Source: §3.1 · p. 3
• The system must accommodate discretionary requests, user-specified computation, subjective task evaluation, and variation in ASP quality and server capacity.These factors include customized denoising-step requests and differences in the maximum number of tasks each server can process within a time window.Source: §3.1 · p. 3

3.2 AIGC Service Provider Selection

• The assignment objective maximizes total user utility across sequential tasks, with each task’s denoising steps representing resource demand that is positively correlated with energy cost.Each ASP has a total resource availability constraint; experiments report increasing energy cost with more denoising steps, alongside an initial cost likely from model initialization.Source: §3.2 · p. 4
• ASP selection is NP-complete and requires real-time decisions because tasks arrive dynamically, utilities are unknown beforehand, and utility depends on the assigned provider’s ability.The scheduler must balance current assignments against reserving resources for future tasks; methods assuming advance knowledge of tasks and utilities are impractical in this setting.Source: §3.2 · pp. 4–5

3.3 Human-aware Utility Function

• Human-aware task utility is modeled as the content-quality assessment G applied to an ASP’s AIGC output Fi(Tj); BRISQUE is given as a no-reference image-quality example.BRISQUE predicts image quality based on human perception and does not require a reference image.Source: §3.3 · p. 5
• Different ASPs show significant variance in human-aware utility for the same service context, underscoring the importance of selecting a well-suited provider.Fig. 1 Part C illustrates differing outputs from various AIGC models for the same text prompt.Source: Figure 1 · p. 4, §3.3 · p. 5

4.1 Motivation of AGOD

• Discrete ASP selection is combinatorial, with an exponentially growing solution space that makes exhaustive search impractical and continuous optimization inaccurate.The discrete variables require jumping between distinct solutions rather than gradually progressing toward an optimum.Source: §4.1 · p. 5

4.2 The Forward Process of Probability Noising

• The forward process represents decisions as probability vectors and adds Gaussian noise until the distribution becomes N(0, I), but AGOD does not execute it because optimal-decision datasets are often unavailable.The process instead establishes a mathematical relationship between the decision distribution and noisy intermediate states.Source: §4.2 · pp. 5–6

4.3 The Reverse Process of Probability Inference

• In reverse inference, a denoising network conditions on state s to iteratively recover decision probabilities from Gaussian noise, using reparameterization and softmax to produce action-selection probabilities.The network scales generated noise with tanh to avoid obscuring the underlying action probabilities.Source: §4.3 · pp. 6–7
• Because labeled optimal decisions are unavailable, AGOD trains its denoising network exploratorily to maximize decision value rather than minimize supervised error.The cited joint decision-value-network approach addresses continuous decision variables, not this discrete setting.Source: §4.3 · p. 7
• Integrating AGOD’s generative exploration and learning with SAC’s efficient, stable policy learning yields the diffusion-based DRL algorithm D2SAC.Source: §4.3 · p. 7

5.1 Problem Modeling

• D2SAC models task assignment as an MDP, using task and ASP-resource features as state, ASP selection as action, and content-quality reward minus crash penalties as reward.The state includes task resource requirements and estimated completion time, plus each ASP’s total and currently available resources; overload penalties discourage crashes and interruptions.Source: §5.1 · pp. 7–8

5.2 Algorithm Architecture

• D2SAC uses AGOD’s observation-conditioned diffusion model as its policy, with double critics, entropy regularization, and slowly updated target networks supporting policy learning.The actor uses the minimum of two critic estimates to reduce overestimation bias, while entropy encourages exploration and target networks are updated softly.Source: §5.2 · pp. 9–10
• Replay memory stores transitions and allows delayed reward feedback to be filled in before off-policy updates, while policy learning proceeds asynchronously without delaying real-time decisions.The learned policy can be updated and used in real time while training runs concurrently.Source: §5.2 · p. 9

5.3 Optimization Goal

• Because online discrete scheduling lacks labeled actions and targets Q-value maximization, D2SAC optimizes policy and action-entropy losses rather than behavior-cloning loss.The paper notes that open datasets for online communication scheduling are impractical to obtain and continuous-control approaches do not apply to discrete action spaces.Source: §5.3 · p. 10

5.4 Complexity Analysis

• D2SAC has the same space complexity as SAC, but adds O(EC|θ|(T −1)) computation for reverse diffusion, an increase associated with higher performance and faster convergence.Its total computational complexity is O(E [CV + TC|θ| + (b + 1) (|θ| + |ϕ|)]); trajectory collection and parameter updates contribute separate terms.Source: §5.4 · p. 10

6.1 Experimental Setup

• With 20 ASPs, the simulation modeled 1,000 users submitting tasks requiring 100–250 denoising steps, with Poisson arrivals at λ = 0.001.ASP capacities ranged from 400 to 1,000 denoising steps; RePaint3 and PIQ modeled AIGC generation and human-aware content quality.Source: §6.1 · pp. 10–11
• D2SAC uses AGOD as its actor with denoising-step encodings and two critics to mitigate overestimation, with Adam-trained actor and critic learning rates of 0.0001 and 0.001.The actor and critics use three fully connected layers; target networks mirror the online networks and are soft-updated at τ = 0.005.Source: §6.1 · p. 11

6.2 Numerical Results

• D2SAC achieved the highest training and test rewards, reached the Crash Avoid baseline after 190 training steps, and had a 1.3-hour training time.Experiments ran for 1,000 training steps and 1 × 10^6 environment steps; time and steps to baseline were measured against Crash Avoid.Source: Table 3 · p. 12, §6.2 · p. 12
• D2SAC outperformed Round Robin after about 45 training steps and exceeded the Crash Avoid baseline by 80 steps, progressively approaching the prophet policy.The authors attribute this learning progression to environment feedback that improves AGOD decisions while maintaining crash avoidance and increasing user utility.Source: §6.2 · p. 12
• DQN achieved the highest utility with no crashes, but D2SAC exceeded DQN in utility; D2SAC crashed 1.1% of tasks with lost utility of 5, versus PPO’s 0.7% and 4.The authors identify DQN as a possible choice when avoiding crashes is paramount.Source: §6.2 · pp. 12–13, Table 5 · p. 14
• D2SAC’s reward peaked at denoising step T = 5 while training time increased with T, suggesting small T can deliver satisfying reward with high computational efficiency.This optimum was substantially below T = 500 reported in; Fig. 7 plots normalized reward and training time across denoising steps.Source: §6.2 · p. 13, Figure 7 · p. 13
• During reverse diffusion, action probabilities became more peaked and could identify multiple optimal actions, while retained uncertainty enabled exploration that diminished as denoising steps increased.The authors link reduced exploration at larger T to the reward decline beyond T = 5.Source: §6.2 · p. 14, Figure 8 · p. 15
• An entropy temperature of α = 0.05 balanced exploration and exploitation; lower α encouraged greedy behavior, whereas higher α made learning slow or stalled.The entropy regularization follows the approach in.Source: §6.2 · pp. 14–15, Figure 9 · p. 15

• D2SAC outperformed seven representative DRL algorithms in both ASP selection and various standard control tasks.Source: §7 · p. 15
• AGOD uses a diffusion model to address environmental uncertainty and variability, and its integration with DRL forms D2SAC for efficient and optimal ASP selection.Source: §7 · p. 15
• The proposed edge-enabled AaaS architecture is designed to enable ubiquitous AIGC functionality.Source: §7 · p. 15
• AGOD has potential application to various optimization problems in wireless networks.Source: §7 · p. 15
• Future research will collect and use real-world datasets on edge-enabled ASP selections to validate and refine the algorithm in practical scenarios.Source: §7 · p. 15