PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-Tenant Cross-Slice Resource Orchestration: A Deep Reinforcement Learning Approach

Xianfu Chen, Zhifeng Zhao, Celimuge Wu, Mehdi Bennis, Hang Liu, Yusheng Ji, Honggang Zhang

arXiv:1807.09350v2cs.NI

TL;DR

The paper addresses long-term cross-slice resource orchestration for selfish service providers sharing a RAN-only sliced network. It constructs an abstract stochastic game, decomposes each provider’s MDP, and applies online deep reinforcement learning; numerical experiments report significant gains over baselines.

  • Problem

    Cross-slice orchestration couples each service provider’s long-term payoff to global network dynamics and the joint policies of competing providers.

  • Method

    The paper abstracts the stochastic game using local conjectures, linearly decomposes each provider’s MDP, and uses online deep reinforcement learning to learn abstract control policies.

  • Results

    Numerical experiments show significant performance gains from the proposed scheme compared with other baseline schemes.

  • Takeaways & Limitations

    The approach approximates decentralized long-term orchestration policies without information exchange among service providers and without a priori network-dynamics statistics.

Abstract

from arXiv · show

With the cellular networks becoming increasingly agile, a major challenge lies in how to support diverse services for mobile users (MUs) over a common physical network infrastructure. Network slicing is a promising solution to tailor the network to match such service requests. This paper considers a system with radio access network (RAN)-only slicing, where the physical infrastructure is split into slices providing computation and communication functionalities. A limited number of channels are auctioned across scheduling slots to MUs of multiple service providers (SPs) (i.e., the tenants). Each SP behaves selfishly to maximize the expected long-term payoff from the competition with other SPs for the orchestration of channels, which provides its MUs with the opportunities to access the computation and communication slices. This problem is modelled as a stochastic game, in which the decision makings of a SP depend on the global network dynamics as well as the joint control policy of all SPs. To approximate the Nash equilibrium solutions, we first construct an abstract stochastic game with the local conjectures of channel auction among the SPs. We then linearly decompose the per-SP Markov decision process to simplify the decision makings at a SP and derive an online scheme based on deep reinforcement learning to approach the optimal abstract control policies. Numerical experiments show significant performance gains from our scheme.

I. INTRODUCTION

The paper studies multi-tenant RAN-only slicing, where shared infrastructure supports heterogeneous computation and communication services. It formulates cross-slice orchestration as a stochastic game and develops abstraction, decomposition, and deep-reinforcement-learning methods for decentralized long-term resource control.

  • Motivation: Dense networks increase capacity but make coordinated RAN control expensive and complex, while MEC addresses computation-intensive applications under constrained terminal resources.The motivation combines dense-network management challenges with growing computation demands and limited MU battery and CPU resources.
  • System context: RAN-only slicing divides a physical RAN into virtual slices customized for heterogeneous service requests, including computation and communication functionalities.The system supports multiple tenants and their MUs over shared infrastructure.
  • Problem formulation: The paper models cross-slice radio-resource orchestration as a non-cooperative stochastic game in which each SP maximizes its own expected long-term payoff while competing with other SPs.The formulation captures coupling among SP decisions and the global network state.
  • Method: An abstract stochastic game uses local conjectures of competing SP behavior without information exchange and provides a bounded performance regret.This abstraction avoids requiring full network-state sharing among SPs.
  • Method: A linear decomposition simplifies each SP’s MDP, while DRL learns computation-offloading and packet-scheduling policies without a priori network-dynamics statistics.The decomposition enables local computation of state-value functions, and the learning scheme targets optimal abstract control policies.
  • Evaluation: Numerical experiments show that the proposed scheme outperforms three state-of-the-art baseline schemes and achieves significant performance gains over other baselines.The experiments use TensorFlow to verify the theoretical studies.

A. Inter-Tenant Channel Auction

The auction allocates limited orthogonal channels to MUs through an SDN orchestrator using SP bids and centralized feasibility constraints. A VCG mechanism determines winners and payments, providing efficiency, individual rationality, and truthful bidding properties.

  • Auction setting: The system uses J nonoverlapping orthogonal channels with equal bandwidth across discrete scheduling slots of equal duration.MUs move according to a Markov mobility model over the service region.
  • Auction setting: At each scheduling slot, each SP submits a bid representing its potentially needed channels and true value for the channel set.The orchestrator then allocates channels to MUs and computes SP payments.
  • Allocation constraints: Channel allocations prevent adjacent base stations from using the same channel and assign at most one channel per MU and one MU per channel within a base-station coverage area.These assumptions make SP decisions coupled only during channel auctions.
  • Winner determination: The SDN orchestrator determines channel winners using the VCG mechanism, subject to centralized allocation constraints.The allocation variables indicate whether a channel is assigned to a particular MU.
  • Economic properties: When SPs announce true bids, the VCG auction maximizes total value and achieves efficient channel utilization.The auction also gives each SP an expected nonnegative payoff.
  • Economic properties: Truthfulness means no SP can improve its payoff by bidding differently from its true value, making truthful bidding optimal at each scheduling slot.This property supports the auction’s strategic behavior model.

B. Computation and Communication Models

The model represents computation-task arrivals, packet arrivals, mobility, queues, transmission energy, and local CPU energy within scheduling slots. MUs can offload tasks to the MEC slice while transmitting queued packets through the communication slice.

  • Computation model: Each computation task is characterized by input size and required CPU cycles, with Markovian arrivals and two execution options: local processing or MEC offloading.Offloading decisions specify the number of tasks transmitted to the MEC server, while remaining tasks are processed locally within the scheduling slot.
  • Communication model: Packet arrivals are independently distributed across MUs and scheduling slots, while each MU maintains a queue for packets awaiting transmission.The queue length and new packet arrivals form part of the MU’s local network state.
  • Utility and constraints: The utility balances packet delay, packet drops, CPU energy, and transmission energy through positive utility functions and an energy-weighting factor.Finite buffer size limits queue occupancy and determines when packet drops occur.
  • Energy consumption: Transmission energy depends on reliably sending packets through the communication slice, with transmit power constrained by a maximum-power bound.The model also incorporates channel gain and noise power spectral density in its transmission calculations.
  • Energy consumption: Local CPU energy accounts for processing computation tasks, with CPU capacity assumed sufficient to process the maximum task arrivals within one scheduling slot.The CPU-energy model uses effective switched capacitance and mobile-device CPU-cycle frequency.
  • Utility and constraints: The model treats mobility-related handover overhead as fixed and neglects task types that can only be processed locally because they do not affect the optimization goal.These assumptions simplify the cross-slice orchestration problem.

C. Control Policy

Each service provider controls channel auctions, computation offloading, and packet scheduling for its subscribed MUs. The control policy maps observed network states to these coordinated decisions and supports a utility that accounts for service quality and energy consumption.

  • State and policy representation: An MU’s local network state combines its location, computation-task state, and packet-queue state.The global network state is formed by the local states of all MUs.
  • State and policy representation: Each service provider’s policy consists of channel auction, computation offloading, and packet scheduling components.Offloading and scheduling policies are MU-specified and depend on the subscribed MUs’ local states.
  • Per-slot decisions: At each scheduling slot, a service provider announces an auction bid and selects the numbers of computation tasks and packets to offload or transmit.These decisions are generated according to the provider’s control policy.
  • Payoff construction: The instantaneous provider payoff incorporates MU utility, including packet-service outcomes, energy consumption, and provider-specific pricing.The utility terms include packet delay, packet drops, CPU energy, and transmission energy.

IV. PROBLEM STATEMENT AND GAME-THEORETIC SOLUTION

The paper formulates cross-slice orchestration among competing service providers as a stochastic game with Markovian network dynamics and coupled control policies. Each provider seeks a best response maximizing discounted long-term payoff, with Nash equilibrium providing the game-theoretic solution concept.

  • Problem formulation: Cross-slice orchestration is modeled as a stochastic game in which service providers compete over channel auctions, computation offloading, and packet scheduling.The players use control policies over a global network state space.
  • Problem formulation: Random mobility, computation-task arrivals, and packet arrivals make the global network-state evolution Markovian under the joint policies.The transition law depends on channel allocation and the joint auction, offloading, and scheduling policies.
  • Best response: Each service provider maximizes an expected infinite-horizon discounted payoff and seeks a best-response policy given the other providers’ policies.The discount factor satisfies γ ∈ [0, 1).
  • Nash equilibrium: A Nash equilibrium is a tuple of policies in which each provider’s policy is a best response to the policies of the other providers.The paper states that such an equilibrium exists in stationary control policies for the considered discounted stochastic game.
  • Coupled decision making: Provider decisions are coupled because each long-term payoff depends on both the global network-state trajectory and the joint control policy.This coupling motivates the need for a game-theoretic treatment rather than isolated single-provider optimization.

B. Best-Response Approach

The paper abstracts the coupled stochastic game using local payment-based conjectures, then derives best-response policies whose deviation from the original game is bounded.

  • Each SP’s best response in the original stochastic game requires global network dynamics and joint policy information, making Nash-equilibrium operation impractical in the non-cooperative setting.
  • The abstract game classifies payment values into intervals, allowing each SP to represent other SPs’ effects through a local abstract state.The abstract state combines the SP’s local state with a payment interval inferred from the channel auction.
  • Payment classification reduces the abstract state space and enables prediction of expected future payments needed for subsequent auction bids.
  • The abstract policy’s original-game counterpart satisfies Vi(χ, (πi, π∗−i)) ≤ Vi(χ) + 2 · Υi, establishing a bounded regret guarantee.Here, Υi is the largest payment interval width for SP i.
  • Solving the abstract best response still faces unknown transition statistics and an exponentially growing decision space as the number of subscribed MUs increases.

B. Decomposition of Abstract State-Value Function

The per-SP abstract MDP is linearly decomposed into independent single-agent MDPs, enabling local MU decisions while preserving an approximation guarantee.

  • Sequential, SP–MU-independent decisions and additive per-slot payoffs motivate decomposing each SP’s MDP into |Ni| + 1 independent single-agent MDPs.
  • The decomposition separates each MU’s expected long-term utility from the SP’s expected long-term payment.
  • Local MUs make computation offloading and packet scheduling decisions, reducing the SP action space from (A × W)|Ni| to |Ni| spaces of size A × W.
  • The decomposition provides an accuracy guarantee for approximating the abstract state-value function while simplifying decision making.
  • A SP’s optimal bid requires local payment-state information and each subscribed MU’s utility, preference, and relevant task information.

C. Learning Optimal Abstract Control Policy

The learning scheme estimates unknown abstract dynamics from auction outcomes and uses Q-learning-based value learning, with deep reinforcement learning needed for the high-dimensional MU state space.

  • Each SP maintains a three-dimensional table Y_i^k of size S_i·S_i·2 and updates it from channel-auction outcomes to estimate abstract state transitions.
  • The abstract payment-value function U_i(s_i) is learned from auction observations with a learning rate satisfying convergence conditions.
  • Value iteration would require local transition probabilities, whereas Q-learning learns from observed states, actions, utilities, and next states without prior transition statistics.
  • The Q-learning update uses exploration and exploitation, with ε-greedy behavior supporting repeated state-action visitation while using the learned Q-function.
  • Conventional tabular Q-learning becomes impractical in the high-dimensional setting, requiring an MU to update 1.39392 · 10^7 Q-function values in the stated example.

2) Deep Reinforcement Learning:

The paper uses double DQN-based DRL to approximate each MU’s optimal state-action value and learn computation offloading and packet scheduling policies without prior network-dynamics statistics. Experiments implement the scheme in a configured RAN and compare it with three baselines.

  • Deep Reinforcement Learning: Double DQN approximates each MU’s optimal state-action Q-function to address the massive local network state space.Each MU maintains a DQN and target DQN, with experience replay used for training.
  • Deep Reinforcement Learning: At each scheduling slot, MUs observe packet arrivals, select actions, and store transitions in replay memory for mini-batch training.The DQN parameters are updated by minimizing the Bellman-equation error, with periodic target-network resets.
  • Parameter Settings: The experiments use a four-base-station RAN in a 2 × 2 Km2 area with 1600 locations and Poisson packet arrivals.Each MU uses a two-hidden-layer DQN with 16 neurons per layer, replay size M = 5000, Tanh activation, and Adam optimization.
  • Limitations: The tradeoff between neural-network training time and performance improvement with deeper or wider networks remains an open problem.The experiments can also be extended to other network layouts.
  • Parameter Settings: The proposed scheme is evaluated against channel-aware, queue-aware, and random control policies.The numerical experiments are implemented using TensorFlow and compare the proposed method with three state-of-the-art baselines.

B. Experiment Results

The convergence experiment evaluates learning stability and batch-size effects. The scheme converges within 10^4 scheduling slots, while utility improvement saturates beyond batch size 150, motivating O = 200 for later experiments.

  • Convergence performance: 10^4 scheduling slots: the proposed learning scheme converges under λ = 6 packets per slot, J = 9, and batch size O = 200.Convergence is assessed using SP 1 utility and a MU’s loss function across scheduling slots.
  • Convergence performance: Average performance per MU is used to compare utility across the learning procedure.The metric is plotted for different batch-size choices in the lower subplot of Fig. 4.
  • Convergence performance: Batch sizes above 150 produce saturated utility improvement across the learning procedure.Larger batches provide more stable gradient estimates and lower variance, but the gain saturates beyond 150.
  • Convergence performance: O = 200 is used in Experiments 2 and 3 to balance utility improvement against computation overhead.The choice follows the observed batch-size saturation and replay-memory constraint.

2) Experiment 2 – Performance under various

The second experiment evaluates the proposed online DRL orchestration scheme under varying packet arrival rates. It compares queueing, packet-drop, energy, and utility outcomes with three baseline schemes.

  • Online DRL procedure: The online DRL procedure initializes replay memory and DQN networks, observes packet arrivals, and selects actions at each scheduling slot.Experience transitions are stored, mini-batches train the DQN, and target parameters are periodically reset.
  • Online DRL procedure: The SDN-orchestrator uses SP bids to determine auction winners and channel allocations before SP and MU decisions proceed.The sequence includes SP payments, SP state updates, MU computation offloading, and replay-memory updates.
  • Performance results: The proposed scheme achieves a significant gain in average utility per MU compared with the three baseline schemes.The same comparison includes channel-aware, queue-aware, and random control policies.
  • Performance results: The proposed scheme realizes the minimum queue length and packet drops among the compared schemes as packet arrival rates vary.The results are shown in Fig. 5 across the learning procedure.
  • Performance results: The proposed scheme transmits more packets to avoid drops while leaving more computation tasks at mobile devices, increasing average CPU energy consumption.This contrasts with Baseline 2, whose average CPU energy consumption decreases as queue length increases.

3) Experiment 3 – Performance with different J:

The third experiment varies the number of available channels while evaluating queueing, packet drops, energy consumption, and utility. The proposed online learning scheme outperforms the three baselines in this setting.

  • Performance with different J: The experiment measures average queue length, packet drops, CPU energy, transmit energy, and utility per MU versus the number of channels.The packet arrival rate is fixed at λ = 8, and the measures are averaged across the learning period.
  • Performance with different J: With more channel opportunities, the proposed scheme enables more computation offloading and packet transmission.The reported energy pattern is lower average CPU energy consumption and higher average transmit energy consumption.
  • System and approach: The study models competing SPs’ channel orchestration as a stochastic game and uses an abstract stochastic game with linearly decomposed per-SP MDPs.An online DRL scheme then seeks optimal abstract control policies for RAN-only slicing.
Loading 1807.09350v2…