PaperlayerView exact revision on arXiv

Source-linked AI summary

Optimal and Fast Real-time Resources Slicing with Deep Dueling Neural Networks

Nguyen Van Huynh, Dinh Thai Hoang, Diep N. Nguyen, Eryk Dutkiewicz

arXiv:1902.09696v1cs.NIcs.AI

TL;DR

The paper addresses real-time slicing under uncertain, multidimensional demand. It uses an SMDP and deep-dueling reinforcement learning to allocate radio, computing, and storage, and reports the best average reward and convergence rate at learning rate 0.01.

  • Problem

    Real-time slice allocation must handle uncertain requests across radio, computing, and storage, while joint optimization creates a large state space.

    Source: A. Related Work · p. 2, §IV · p. 7

  • Method

    The paper models real-time requests with an SMDP and applies deep dueling, using two streams to train value and advantage functions.

    Source: A. Related Work · p. 2, §VII · p. 14

  • Results

    At learning rate 0.01, deep dueling achieves the best average reward and convergence rate compared with other learning rates.

    Source: 2) Convergence of Deep Reinforcement… · p. 14

  • Takeaways & Limitations

    The proposed framework is presented as able to handle a large state space while supporting real-time allocation across three resource types.

    Source: A. Related Work · p. 2

    For theoretical analysis and performance evaluation, slice arrivals are assumed Poisson and resource occupation times exponential.

    Source: D. State Transition Probability · p. 4

Abstract

from arXiv · show

Effective network slicing requires an infrastructure/network provider to deal with the uncertain demand and real-time dynamics of network resource requests. Another challenge is the combinatorial optimization of numerous resources, e.g., radio, computing, and storage. This article develops an optimal and fast real-time resource slicing framework that maximizes the long-term return of the network provider while taking into account the uncertainty of resource demand from tenants. Specifically, we first propose a novel system model which enables the network provider to effectively slice various types of resources to different classes of users under separate virtual slices. We then capture the real-time arrival of slice requests by a semi-Markov decision process. To obtain the optimal resource allocation policy under the dynamics of slicing requests, e.g., uncertain service time and resource demands, a Q-learning algorithm is often adopted in the literature. However, such an algorithm is notorious for its slow convergence, especially for problems with large state/action spaces. This makes Q-learning practically inapplicable to our case in which multiple resources are simultaneously optimized. To tackle it, we propose a novel network slicing approach with an advanced deep learning architecture, called deep dueling that attains the optimal average reward much faster than the conventional Q-learning algorithm. This property is especially desirable to cope with real-time resource requests and the dynamic demands of users. Extensive simulations show that the proposed framework yields up to 40% higher long-term average return while being few thousand times faster, compared with state of the art network slicing approaches.

I. INTRODUCTION

Network slicing must accommodate diverse service requirements and dynamic demands while jointly managing multiple resource types. The paper frames this as a real-time allocation problem and proposes an SMDP-based framework with deep reinforcement learning.

I. INTRODUCTION

  • Legacy mobile networks struggle to meet emerging services’ adjustable priority and QoS requirements, while providers must manage radio, computing, and storage under dynamic demand.Source: §I · p. 1

A. Related Work

  • Prior methods and optimize radio alone, demand prediction targets short-term reward, and reinforcement learning can converge slowly at scale.Source: A. Related Work · p. 2

B. Main Contributions

  • The proposed SMDP framework jointly allocates radio, computing, and storage to slice requests in real time while accounting for uncertain, dynamic demand.Source: A. Related Work · p. 2, B. Main Contributions · p. 2
  • Deep double Q-learning addresses Q-learning’s difficulty with joint resource optimization, while deep dueling uses two streams to train value and advantage functions concurrently.Source: B. Main Contributions · p. 2
  • Simulations show the proposed framework improves system performance by up to 40% compared with other current approaches.Source: B. Main Contributions · p. 3

II. SYSTEM MODEL

  • Providers supply slices to tenants, whose requests specify service demands; service management accepts or rejects requests, and accepted requests proceed to resource orchestration.Source: §II · p. 3

III. PROBLEM FORMULATION

The provider uses an event-driven SMDP to make slice-admission decisions and maximize long-term return. Its state tracks the running slices by class, subject to resource constraints.

III. PROBLEM FORMULATION

  • The SMDP models provider decisions using decision epochs, states, actions, transition probabilities and sojourn times, and rewards.Source: §III · p. 4
  • The provider’s long-term return is the objective for its admission policy.Source: §III · p. 4

A. Decision Epoch

B. State Space

  • The state vector records the number of simultaneously served slices from each class.Source: B. State Space · p. 4
  • The state space contains the slice-count vectors that satisfy the provider’s resource constraints.Source: B. State Space · p. 4
  • Events track class-specific arrivals, departures, or no change; the all-zero event means no class has an arrival or departure.Source: B. State Space · p. 4

C. Action Space

At an arrival, the provider chooses whether to accept or reject the request. The transition model uses stated arrival and service-time assumptions and uniformization to represent event dynamics.

C. Action Space

D. State Transition Probability

E. Reward Function

The reward model links accepted requests to tenant payments and frames admission as maximizing average return. The section also describes state-space extensions and learning under uncertain, potentially unavailable environment parameters.

E. Reward Function

  • Accepting an arriving slice yields immediate reward r_c, while rejection or no arrival yields zero; r_c represents the tenant’s payment for resources and services.Source: E. Reward Function · p. 6
  • The admission policy is chosen to maximize the provider’s average reward, which is independent of the initial state under the stated unichain assumption.Source: E. Reward Function · p. 6
  • For multiple data centers or scaling events, the model can add states and rewards while retaining accept-or-reject actions.Source: E. Reward Function · pp. 6–7
  • Because arrival and completion parameters may be unavailable or time-varying, the authors consider deep double Q-learning and deep dueling to find an admission policy.Source: E. Reward Function · p. 7

IV. Q-LEARNING FOR DYNAMIC RESOURCE ALLOCATION UNDER UNCERTAINTY OF SLICE SERVICE DEMANDS

Q-learning estimates action values from observed rewards and next states to select a policy with high long-term value. The paper notes that its convergence can be prohibitively slow for the large state space in combinatorial resource slicing.

IV. Q-LEARNING FOR DYNAMIC RESOURCE ALLOCATION UNDER UNCERTAINTY OF SLICE SERVICE DEMANDS

  • Q-learning stores values for state-action pairs and updates them using observed rewards and next states to learn an optimal policy.Source: §IV · p. 7
  • The discount factor γ controls the relative importance of long-term versus short-term rewards in the value function.Source: §IV · p. 7
  • An ε-greedy policy explores by choosing a random action with probability ε and otherwise selects the action with the highest current Q-value.Source: §IV · p. 7
  • For combinatorial resource slicing, the state space can reach tens of thousands, making Q-learning prohibitively slow and practically inapplicable to real-time use.Source: §IV · p. 7

A. Deep Double Q-Learning

Deep double Q-learning adapts Q-learning to resource slicing by using primary and target networks to learn action values from request transitions. Experience replay, delayed target updates, and four state features support training under the slicing dynamics.

A. Deep Double Q-Learning

  • The primary network selects actions, while the target network computes their target Q-values.Actions and resulting rewards and next states are stored as transitions and used for later training.Source: A. Deep Double Q-Learning · p. 8
  • Experience replay trains on randomly sampled past transitions, reusing experience and reducing correlations between observations.The target Q-network is updated periodically rather than with every primary-network update, stabilizing value estimates.Source: A. Deep Double Q-Learning · pp. 8–9
  • Each state is represented by radio, computing, storage, and request-arrival features for estimating action Q-values.The authors state that including these aspects of each state results in a higher convergence rate.Source: A. Deep Double Q-Learning · p. 8
  • Stochastic gradient descent estimates the gradient from a minibatch of replay-memory experiences instead of processing the full memory.The full-memory operation has computational cost O(D), whereas minibatch sampling is used to speed training.Source: A. Deep Double Q-Learning · p. 9

B. Deep Dueling Network

The deep dueling network addresses limited convergence of deep Q-learning methods, particularly in large-scale systems. It estimates state value and action advantage in separate streams and combines them to form action values.

B. Deep Dueling Network

  • Deep dueling targets the limited convergence rates of deep Q-learning and deep double Q-learning, especially in large-scale systems.The authors attribute the remaining limitation to optimizer overestimation.Source: B. Deep Dueling Network · p. 9
  • Separate fully connected streams estimate state value and action advantage, then combine their outputs into the Q-function.The value stream outputs a scalar and the advantage stream an |A|-dimensional vector.Source: B. Deep Dueling Network · p. 10
  • The advantage function measures each action’s relative importance, while the value function measures how good it is to be in a state.The authors say this architecture can yield more robust state-value estimates and improve convergence rate and stability.Source: B. Deep Dueling Network · p. 10

A. Parameter Setting

The simulations use three slice classes and specify request, resource, and neural-network settings for evaluating the algorithms.

A. Parameter Setting

  • The simulations model utility, automotive, and manufacturing slices, with arrival rates of 12, 8, and 10 requests/hour and completion rates of 3 requests/hour for each class.Accepted-request rewards are 1, 2, and 4, respectively.Source: A. Parameter Setting · p. 11
  • Each simulated slice request requires 1 GB storage, 2 CPUs, and 100 Mbps radio resources.Source: A. Parameter Setting · p. 11
  • The deep dueling network uses two streams with 64-unit hidden layers, and its minibatch size is 64.The experience replay buffer holds up to 10,000 transitions, and the target network updates every 1,000 iterations.Source: A. Parameter Setting · p. 11

B. Simulation Results

Deep dueling outperforms the compared slicing approaches in average reward and network performance, including as resource demands and system size vary. Its advantage is especially apparent where Q-learning converges slowly or cannot reach the optimal policy within the tested iterations.

1) Performance Evaluation:

  • Considering radio, computing, and storage resources yields significantly higher average reward than optimizing radio alone, especially when computing and storage are scarce.Average reward increases as computing and storage resources increase, allowing more slice requests to be accepted.Source: 1) Performance Evaluation · p. 11
  • In the small system, reinforcement learning earns significantly higher average reward than greedy, while Q-learning still trails deep dueling.The comparison varies the class-3 reward from 1 to 6.Source: 1) Performance Evaluation · p. 11, Figure 7 · p. 11
  • As class-3 reward rises, reinforcement learning favors higher-reward requests; at reward 6, class-1 requests with reward 1 approach zero.Greedy request counts remain unchanged because greedy admission does not account for immediate reward.Source: 1) Performance Evaluation · p. 12, Figure 8 · p. 12
  • In the large system with over 74,000 state-action pairs, deep dueling obtains the optimal policy in about 2 hours, while Q-learning does not within 10^7 iterations or more than 15 hours.At 10^6 iterations, Q-learning performs like greedy; at 10^7 iterations, it improves but remains inferior to deep dueling.Source: 1) Performance Evaluation · p. 12, Figure 9 · p. 12
  • Deep dueling achieves the best performance across the evaluated cases in average reward and network performance.The comparisons include greedy and Q-learning algorithms across different system sizes and class-3 reward settings.Source: 1) Performance Evaluation · pp. 11–12
  • Deep dueling obtains the best policy across three resource-capacity settings, whereas Q-learning fluctuates at 20 times requested capacity and does not converge to the optimal policy even after 10^7 iterations.The policy comparison measures request-acceptance probabilities by class.Source: 1) Performance Evaluation · pp. 12–13, Figure 11 · p. 13

2) Convergence of Deep Reinforcement Learning Approaches:

The paper reports that deep dueling converges to the optimal policy in increasingly complex resource settings, where other learning approaches require more iterations or fail to converge. It concludes that the framework improves long-term average reward and training speed, and reports that a learning rate of 0.01 performs best among those tested.

2) Convergence of Deep Reinforcement Learning Approaches:

  • With 400 Mbps radio, 8 CPUs, and 4 GB storage, deep reinforcement approaches converge within 10,000 iterations, while Q-learning needs more than 10^6 iterations.The paper attributes the faster convergence to fully connected neural-network layers reducing the curse of dimensionality.Source: 2) Convergence of Deep Reinforcement… · p. 13
  • At 1 Gbps, 10 GB, and 20 CPUs, deep dueling reaches the optimal policy within 15,000 iterations, while the other two deep reinforcement approaches need longer.The compared approaches are deep Q-learning, deep double Q-learning, and deep dueling.Source: 2) Convergence of Deep Reinforcement… · pp. 13–14, Figure 13 · p. 14
  • At 2 Gbps, 20 GB, and 40 CPUs, deep dueling reaches the optimal policy within 20,000 iterations; deep Q-learning and deep double Q-learning do not converge after 100,000.The paper attributes this result to the dueling network's two-stream design, which reduces optimizer overestimation.Source: 2) Convergence of Deep Reinforcement… · p. 14, Figure 13 · p. 14
  • Among the tested learning rates, 0.01 gives deep dueling its best average reward and convergence rate.The paper notes that a rate that is too high may prevent convergence or cause divergence, while a rate that is too low slows training.Source: 2) Convergence of Deep Reinforcement… · p. 14, Figure 14 · p. 14

VII. CONCLUSION

  • Simulations report up to 40% higher long-term average reward and a few thousand times faster performance for the deep-dueling framework than other network-slicing approaches.The conclusion describes joint real-time allocation of radio, computing, and storage resources using an SMDP and reinforcement learning.Source: §VII · p. 14

APPENDIX A THE PROOF OF THEOREM 1

The appendix proof passages describe transition-matrix steps for Theorem 1 and a Cesàro-limit argument for Lemma 1.

APPENDIX A THE PROOF OF THEOREM 1

  • Theorem 1 proof passages define transition matrices and use the forward differential equation P′(t) = P(t)Q to express the transition probabilities.The following passages relate the transition matrix to Q and the identity matrix, then condition on Poisson events.Source: Appendix A · pp. 14–15

APPENDIX B THE PROOF OF LEMMA 1

  • The Lemma 1 proof passages define a Cesàro limit through averages of matrix powers and state that aperiodicity makes the limiting matrix equal to P.The passages express the limiting matrix in component notation before invoking aperiodicity.Source: Appendix B · p. 15

Related papers

Loading 1902.09696v1…