PaperlayerView exact revision on arXiv

Source-linked AI summary

Digital Twin Assisted Deep Reinforcement Learning for Online Admission Control in Sliced Network

Zhenyu Tao, Wei Xu, Xiaohu You

arXiv:2310.09299v3cs.LGcs.SYeess.SPeess.SY

TL;DR

Real-network admission control with DRL is challenged by instability during early training, which can disrupt network operations. The paper uses a supervised neural-network digital twin to assist DRL training after formulating admission decisions as an SMDP and an equivalent discrete-time MDP. Simulations report over 40% higher resource utilization than directly trained state-of-the-art Dueling-DQN while retaining long-term reward optimization.

  • Problem

    Early DRL instability complicates deployment of admission control in real networks, where online training can disrupt operations and reduce resource utilization.

    Source: §I · p. 1

  • Method

    The approach trains a neural-network digital twin through supervised learning to replicate admission policies and assist DRL training, using an equivalent discrete-time MDP formulation.

    Source: §I · p. 2, §IV · p. 6

  • Results

    Over 40% higher resource utilization than directly trained state-of-the-art Dueling-DQN during initial training, while retaining the ability to optimize long-term rewards.

    Source: §VI · p. 11

  • Takeaways & Limitations

    The reported improvement mitigates deployment risk in practical wireless networks while sustaining the model’s effectiveness.

    Source: §VI · p. 11

Abstract

from arXiv · show

The proliferation of diverse wireless services in 5G and beyond has led to the emergence of network slicing technologies. Among these, admission control plays a crucial role in achieving service-oriented optimization goals through the selective acceptance of service requests. Although deep reinforcement learning (DRL) forms the foundation in many admission control approaches thanks to its effectiveness and flexibility, initial instability with excessive convergence delay of DRL models hinders their deployment in real-world networks. We propose a digital twin (DT) accelerated DRL solution to address this issue. Specifically, we first formulate the admission decision-making process as a semi-Markov decision process, which is subsequently simplified into an equivalent discrete-time Markov decision process to facilitate the implementation of DRL methods. A neural network-based DT is established with a customized output layer for queuing systems, trained through supervised learning, and then employed to assist the training phase of the DRL model. Extensive simulations show that the DT-accelerated DRL improves resource utilization by over 40% compared to the directly trained state-of-the-art dueling deep Q-learning model. This improvement is achieved while preserving the model's capability to optimize the long-term rewards of the admission process.

I. INTRODUCTION

Network slicing must allocate limited shared resources among heterogeneous service requests while pursuing objectives such as revenue or fairness. The paper addresses early DRL instability in real-network admission control with a DT-assisted training approach and reports improved resource utilization, particularly during initial training.

I. INTRODUCTION

  • Real-network DRL admission control faces deployment challenges because inaccurate virtual environments can impair transfer, while online training can disrupt operations and reduce resource utilization.The paper identifies highly stochastic actions from randomly initialized DRL models as a source of disruption.Source: §I · p. 1, B. Incorporation of DRL and Admission… · p. 3
  • The paper presents DT-assisted DRL as an initial attempt to address instability during DRL’s initial training stage and reduce deployment risk in real-world networks.The stated aim is to mitigate initial stochasticity.Source: §I · p. 2
  • The paper formulates queued, combinatorial-resource admission decisions as an SMDP and transforms it into an equivalent discrete-time MDP for DRL implementation.The introduction presents this as a contribution to facilitating DRL methods.Source: §I · p. 2
  • The proposed neural-network DT uses a customized output layer for queued requests and supervised learning to replicate admission policies, then assists DRL training.The introduction also characterizes the approach as more stable than traditional DRL training.Source: §I · p. 2
  • The proposed solution improves resource utilization particularly during initial training while maintaining the DRL model’s ability to optimize service-oriented objectives.The introduction describes extensive simulations validating the solution.Source: §I · p. 2

A. Admission Control for Network Slicing

  • Prior admission-control methods include search, heuristics, Q-learning, and integer linear programming, while search and heuristics may be inapplicable or non-optimal in complex systems.DRL was introduced for admission control to maximize long-term revenue.Source: A. Admission Control for Network Slicing · p. 2

B. Incorporation of DRL and Admission Control

  • Prior sliced-network DRL work uses DQN, A2C, and multi-agent PPO, but initial-training instability remains a deployment obstacle, especially for latency-critical services –.The paper identifies this instability as a challenge for real network systems.Source: B. Incorporation of DRL and Admission… · pp. 2–3

C. Digital Twin for Mobile Networks

The paper motivates a policy-focused digital twin for DRL admission control and models a sliced network with queued requests, resource constraints, and event-driven decisions.

C. Digital Twin for Mobile Networks

  • Because accurately mirroring the full wireless network and request behavior is difficult, the study twins the real network’s admission policy to assist DRL training.Earlier wireless-network digital twins replicated components such as topology, RAN, and core networks.Source: C. Digital Twin for Mobile Networks · p. 3

III. SYSTEM MODEL AND PROBLEM FORMULATION

  • The modeled network has end users, tenants, and an infrastructure provider; K denotes the number of slices and tenants.The example uses K = 4 for eMBB, URLLC, mMTC, and other services.Source: §III · p. 3
  • Requests reach tenants, which transfer them to the provider when resources suffice; otherwise, requests queue, while completed services release resources.Queued requests leave if their waiting time exceeds their patience threshold.Source: §III · p. 3
  • Slice-specific arrivals are Poisson, service times are exponential, and queued requests have a maximum hold time; active services use radio, computing, and storage resources.Resource-use proportions vary by service type, such as higher radio use for eMBB and computing slack for URLLC.Source: §III · p. 3
  • The continuous network is modeled as an SMDP, where request arrivals and service completions trigger decisions at event-dependent intervals.The model’s transitions, sojourn times, and rewards depend on the current state and chosen action.Source: §III · p. 4

A. State Space

  • Each decision state records queued requests and running services by slice, subject to resource constraints.The state is represented using request-queue and active-service counts.Source: A. State Space · p. 4

B. Action Space

Admission actions specify how many requests to accept from each slice, allowing selective multi-request admission when queues accumulate.

B. Action Space

  • Actions specify the number of admitted requests for each slice, rather than a single binary accept-or-reject decision.Each slice’s action count is bounded by nmax.Source: B. Action Space · p. 4
  • The per-slice admission cap nmax depends on the policy and service resource use, and can matter when released resources permit admitting multiple queued requests.When resources suffice, arrivals are admitted immediately; the maximum can be reached when requests have accumulated in queues.Source: B. Action Space · p. 4

C. Sojourn Time Distribution

Decision epochs are triggered by request arrivals and service completions, with action-dependent sojourn times and an embedded Markov chain describing transitions.

C. Sojourn Time Distribution

  • Only request arrivals and service completions trigger decisions; queued-request departures do not require an admission decision.Arrivals require an admission choice, while service completion releases resources for queued requests.Source: C. Sojourn Time Distribution · p. 4
  • The interval to the next trigger event is exponentially distributed, with its expected duration determined by arrival and service rates and ongoing services.The subsequent trigger event forms a Poisson process with rate 1/τ(s).Source: C. Sojourn Time Distribution · p. 4
  • Because an admission action can change ongoing-service counts, sojourn time depends on both the state and action; only queue- and resource-valid actions are allowed.The valid action space is defined for each state.Source: C. Sojourn Time Distribution · pp. 4–5

D. Transition Probability

  • The SMDP uses an embedded Poisson process for trigger-event arrivals and an embedded discrete-time Markov chain for transitions between decision epochs.The chain’s transition probabilities describe the next state given the current state and action.Source: D. Transition Probability · p. 5

E. Reward Function

The reward design balances valid-action benefits with penalties, then frames admission control as long-term average reward maximization under a policy.

E. Reward Function

  • Valid actions earn revenue until the next trigger event, while invalid actions incur a penalty for missed resource-optimization opportunities.The reward design includes nonnegative constants for resource charges and the penalty.Source: E. Reward Function · p. 5

F. Problem Formulation

  • If the embedded Markov chain under policy π has no disjoint closed sets, the SMDP’s long-term average reward is constant across initial states.The optimization objective is to maximize long-term average reward over policies.Source: F. Problem Formulation · p. 5
  • DRL optimizes the policy using high-dimensional network data, while the digital twin assists DRL training.The equilibrium probability under a policy is not straightforward to determine in the network.Source: F. Problem Formulation · p. 5

IV. DT-ASSISTED ONLINE DRL SOLUTION

The solution converts the admission problem’s SMDP into an equivalent discrete-time MDP and uses a supervised digital twin of the default policy to initialize and assist online DRL training. Its customized actor output and staged actor–critic training support queued actions while limiting disruption during early learning.

DT-assisted online DRL solution

  • A supervised digital twin learns the default admission policy and transfers its parameters to the actor, giving DRL an initial policy to fine-tune toward the reward objective.The DT is trained from state-action pairs collected as the network makes admission decisions; the policy’s state information may differ from the collected state.Source: §IV · pp. 5–6
  • The SMDP is transformed into a discrete-time MDP to address reward variability from uncertain sojourn times, preserving long-term average reward for each stationary policy.The transformed reward divides the original reward by the expected sojourn time.Source: §IV · p. 6
  • The DT output layer predicts probabilities for each action-vector variable separately and derives an action’s probability from their product.During prediction, variables are selected greedily or by probability-based sampling and concatenated into an action vector.Source: §IV · p. 7
  • DT training treats admission control as classification, using state as input, the default action as label, and cross-entropy loss to replicate the default policy.The training data are collected from network decisions, while DT training is isolated from network operation.Source: §IV · pp. 6–7
  • After copying DT parameters to the actor, the method first trains a frozen-actor critic, then trains actor and critic together to maximize long-term rewards.The critic is trained until it relatively accurately approximates the state value function; both networks then continue training through real-network interaction.Source: §IV · p. 8

A. Experiment Setting

The experiments compare three default admission policies and a directly trained Dueling-DQN baseline, using a shared three-layer, 64-node network architecture for the DT, actor, and critic. The setup also tests a storage-revenue objective alongside the environment’s stated service and resource settings.

A. Experiment Setting

  • The DT, actor, and critic use three-layer feed-forward networks with 64 nodes per layer; DT supervised learning uses batches of 64.Actor and critic training use one sample per decision epoch because only one set of data is obtained at each epoch.Source: A. Experiment Setting · p. 8
  • The comparison includes PRIO, which prioritizes URLLC over eMBB, mMTC, and other services while considering fairness, plus ILP and greedy default policies.ILP maximizes radio-resource utilization at each decision epoch; greedy accepts requests in decreasing order of radio-resource occupation.Source: A. Experiment Setting · pp. 8–9
  • Dueling-DQN is the state-of-the-art comparison baseline, with 256 conventional output nodes for the potential actions, unlike the proposed modified output structure.Its architecture has three layers of 64 nodes each.Source: A. Experiment Setting · p. 9
  • A separate optimization goal maximizes revenue from storage-resource charges, using charge vector c = [0, 0, 100].The simulation uses Python 3.9, PyTorch 1.10, CUDA 11.3, and NumPy on a PC with an NVIDIA GeForce RTX 3080 GPU.Source: A. Experiment Setting · pp. 8–9

1) Supervised Learning-based DT Performance Evaluation:

The DT’s predictive accuracy increases with training-set size and stabilizes as the sample grows. With 100,000 samples, it exceeds 99% accuracy for all three default policies, and that dataset is used in subsequent experiments.

1) Supervised Learning-based DT Performance Evaluation:

  • DT training uses 400 epochs and early stopping with a patience of 20 epochs.Predictive accuracy is evaluated on a test set under different default policies and training-set sizes.Source: 1) Supervised Learning-based DT… · p. 9
  • At 100,000 training samples, DT predictive accuracy exceeds 99% for all three default policies.The authors use the DT trained on this dataset in subsequent experiments.Source: 1) Supervised Learning-based DT… · p. 9
  • Predictive accuracy is positively correlated with training-set size and stabilizes as the sample size increases.The DT was trained for 400 epochs with early stopping after a patience of 20 epochs.Source: 1) Supervised Learning-based DT… · p. 9

2) DT-assisted DRL Performance Evaluation:

DT assistance substantially raises early resource utilization while the admission models continue to pursue long-term reward objectives. The evaluation also shows that results depend partly on the default policy being replicated.

2) DT-assisted DRL Performance Evaluation:

  • During training, the study compares resource utilization and slice-specific request acceptance ratios across methods using four experiments with different random seeds; after training, cumulative rewards check whether the optimization goal is achieved.Fig. 8 reports average reward in 400 iterations after training.Source: 2) DT-assisted DRL Performance… · pp. 9–10, Figure 8 · p. 11
  • Directly trained models initially show low resource utilization and unstable acceptance ratios; modified A2C stabilizes after about 20,000 decision epochs, while Dueling-DQN has wider error bars.Modified A2C tends to accept more mMTC and Other requests to increase storage-resource occupation.Source: Figure 5 · p. 10, 2) DT-assisted DRL Performance… · p. 10
  • DT-assisted models maintain high resource utilization throughout training, while acceptance patterns begin like the default policy and eventually align with directly trained DRL.The initial acceptance pattern is consistent with the default policy; it evolves toward the directly trained DRL pattern as training progresses.Source: Figure 6 · p. 10, 2) DT-assisted DRL Performance… · p. 10
  • All DRL models outperform default admission policies on the preset target; DT-assisted performance is similar or better with PRIO and Greedy defaults but worse with ILP.The authors attribute the ILP-based shortfall primarily to limited training samples and expect it to diminish with further convergence.Source: 2) DT-assisted DRL Performance… · pp. 10–11
  • Up to 41.75% higher resource utilization than directly trained Dueling-DQN and 22.36% higher than the identical directly trained model in the first 4,000 epochs.The reported utilization gaps diminish as the models converge.Source: 2) DT-assisted DRL Performance… · p. 10, Figure 7 · p. 11

VI. CONCLUSION

  • The conclusion reports over 40% higher utilization than directly trained Dueling-DQN and over 20% than the directly proposed model, while retaining long-term reward optimization; a simple greedy substitute may help when defaults are hard to replicate.The reported utilization gains concern the initial training stage.Source: §VI · p. 11

APPENDIX A PROOF OF THEOREM 1

The appendix uses renewal-reward reasoning for the semi-Markov process, then relates its equilibrium behavior to that of the equivalent discrete-time MDP.

APPENDIX A PROOF OF THEOREM 1

  • The proof treats returns to a fixed state as renewal cycles and applies the renewal-reward theorem using cycle reward and cycle length.It also invokes the expected-value version of the theorem over epochs in the first cycle.Source: Appendix A · p. 11
  • The proof uses the Markovian property of reward and sojourn time, together with policy-conditioned i-step transition probabilities, to connect cycle quantities to equilibrium probabilities.Source: Appendix A · p. 12

APPENDIX B PROOF OF THEOREM 2

  • The second proof relates the embedded-chain and discrete-time MDP equilibrium equations, normalizes their probabilities, and derives the equivalent MDP’s long-term average reward.It introduces a positive scaling constant based on expected sojourn time.Source: Appendix B · p. 12
Loading 2310.09299v3…