PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Reinforcement Learning Based Mode Selection and Resource Management for Green Fog Radio Access Networks

Yaohua Sun, Mugen Peng, Shiwen Mao

arXiv:1809.05629v1cs.NI

TL;DR

Existing F-RAN resource management often assumes static caching or a single communication mode, despite dynamic cache states and coupled mode and resource decisions. The paper uses DRL for joint UE mode and processor-state control, followed by C-RAN precoding optimization, and reports effective control with transfer learning reducing environmental interactions.

  • Problem

    F-RAN resource management must address dynamic edge-cache states and coupled communication mode selection and resource allocation beyond static single-mode settings.

  • Method

    A DRL controller selects UE communication modes and cloud processor on-off states, while precoding for C-RAN UEs is optimized subsequently under constraints.

  • Results

    The proposal performs best among compared schemes in a heterogeneous caching environment, while transfer learning achieves similar performance with fewer environmental interactions.

  • Takeaways & Limitations

    DRL provides a control approach for reducing long-term F-RAN power consumption under dynamic caching and complex wireless conditions.

Abstract

from arXiv · show

Fog radio access networks (F-RANs) are seen as potential architectures to support services of internet of things by leveraging edge caching and edge computing. However, current works studying resource management in F-RANs mainly consider a static system with only one communication mode. Given network dynamics, resource diversity, and the coupling of resource management with mode selection, resource management in F-RANs becomes very challenging. Motivated by the recent development of artificial intelligence, a deep reinforcement learning (DRL) based joint mode selection and resource management approach is proposed. Each user equipment (UE) can operate either in cloud RAN (C-RAN) mode or in device-to-device mode, and the resource managed includes both radio resource and computing resource. The core idea is that the network controller makes intelligent decisions on UE communication modes and processors' on-off states with precoding for UEs in C-RAN mode optimized subsequently, aiming at minimizing long-term system power consumption under the dynamics of edge cache states. By simulations, the impacts of several parameters, such as learning rate and edge caching service capability, on system performance are demonstrated, and meanwhile the proposal is compared with other different schemes to show its effectiveness. Moreover, transfer learning is integrated with DRL to accelerate learning process.

I. INTRODUCTION

F-RAN resource management must jointly address communication mode selection, heterogeneous resources, and dynamic edge-cache states. The paper proposes DRL-based control to reduce long-term power consumption while subsequently optimizing C-RAN precoding.

  • Related Work and Challenges: Communication mode selection is coupled with resource management, and its optimization is usually NP-hard.Classical optimization methods can therefore incur high computational complexity.
  • DRL Motivation: DRL can provide fast, model-free control from high-dimensional network data while optimizing long-term performance.DQN prediction supports rapid decisions, and replay memory with a target DQN stabilizes training.
  • Proposed Approach: The proposed controller jointly selects UE communication modes and processor on-off states under changing cache states.Precoding for C-RAN UEs is optimized afterward under QoS and cloud computing constraints.
  • Related Work and Challenges: Existing F-RAN studies often assume a static environment or only C-RAN mode, limiting their treatment of dynamic mode selection.Dynamic D2D cache availability and interference further complicate control.
  • Evaluation: The paper evaluates learning parameters and caching capability, compares multiple control schemes, and integrates transfer learning for a new similar environment.The study includes comparisons with Q-learning and random control.

II. SYSTEM MODEL

The modeled downlink F-RAN contains a cloud, RRHs, UEs, paired D2D transmitters, and heterogeneous cloud processors. Each UE selects either D2D or C-RAN service, with mode-specific transmission and interference assumptions.

  • System Components: The downlink F-RAN comprises one cloud, multiple RRHs, UEs with paired D2D transmitters, and a control entity.Cloud processors have heterogeneous computing capabilities and share resources through fiber links.
  • Communication Modes: Each UE operates in either D2D mode or C-RAN mode, represented by a binary communication-mode indicator.The indicator equals one for D2D service and zero for RRH service.
  • C-RAN Transmission: C-RAN UEs receive collaborative RRH transmission using precoding vectors, with data rate determined by the received signal model.The model defines messages, channels, precoders, and noise for C-RAN transmission.
  • D2D Transmission: D2D transmitters use constant transmit power, and D2D rates account for the desired link and interference from other active D2D links.D2D and RRH transmissions use different frequency bands, while UEs within the same mode share a band and interfere.

B. Computing Model

The computing model accounts for cloud processing required by C-RAN UEs, including coding, modulation, and precoding-related signal generation. Local D2D service can reduce cloud computing demand and enable processor shutdowns.

  • Computing Consumption: Cloud computing resource for coding and modulation depends on each C-RAN UE’s data rate.The model introduces the per-UE consumption associated with these baseband operations.
  • Computing Consumption: Computing resource for transmit-signal calculation depends on the number of nonzero elements in the network-wide precoding vector.This links precoding sparsity to cloud processing demand.
  • Computing Consumption: Total system computing consumption aggregates the modeled per-UE processing requirements.Additional constant terms for other baseband operations do not affect the proposed method.
  • D2D Offloading: Serving UEs locally through D2D reduces cloud computing needs and can allow some processors to be turned off to save energy.Only UEs accessing RRHs consume the modeled cloud computing resource.

C. Caching Model

The caching model defines cache availability using both content presence and transmitter willingness, then represents its dynamics with a Markov transition model that informs energy-aware control.

  • C. Caching Model: A D2D cache state is True only when the requested content is cached and the transmitter’s battery level is high enough to share it.The state is False otherwise.
  • C. Caching Model: Cache states are dynamic because paired users may update cached content according to personal interests, while battery levels and content requests vary over time.
  • C. Caching Model: A Markov process with True/False transition probabilities is adopted to characterize cache-state dynamics.The transition notation includes probabilities such as True-to-False changes.
  • C. Caching Model: System energy includes consumption from running processors, fronthaul transmission, and wireless transmission.Processor energy depends on a structure-related parameter, while wireless power covers RRH- or D2D-served UEs and amplifier efficiencies.
  • C. Caching Model: The Markov cache model motivates an MDP formulation whose reward reflects data-rate, computing, and energy-consumption models under QoS and computing constraints.

III. PROBLEM FORMULATION AND DECOUPLING

The paper formulates dynamic F-RAN energy minimization as an MDP, using cache-aware states and actions to control the network under uncertain transitions. Q-learning avoids requiring explicit transition probabilities and yields an optimal policy from learned Q values.

  • III. PROBLEM FORMULATION AND DECOUPLING: The energy-minimization problem is decoupled into joint processor and UE-mode control plus precoding design under a computing-resource constraint.
  • A. The Basics of MDP: Cache states influence D2D eligibility and processor shutdown opportunities, making an MDP natural for dynamic energy minimization.
  • A. The Basics of MDP: The MDP represents states, actions, transition probabilities, and rewards for planning and acting under uncertainty.Its process proceeds from an initial state through actions and state transitions.
  • A. The Basics of MDP: The reward accumulates discounted future rewards, while the policy maps each state to a probability distribution over available actions.
  • A. The Basics of MDP: Q-learning is used when transition probabilities are difficult to obtain because it does not require explicit transition probabilities or an essential predefined reward function.
  • A. The Basics of MDP: The Q function measures expected cumulative reward from a state-action pair while subsequently following a policy.
  • A. The Basics of MDP: After optimal Q values are obtained, the optimal policy selects the action with the greatest Q value for each state.

B. Problem Formulation

The problem formulation models long-term F-RAN energy control through processor states, UE communication modes, and cache states, with DRL selecting discrete controls before constrained precoding optimization. The reward penalizes total system energy while protecting QoS when state transitions or interference create violations.

  • B. Problem Formulation: The controller minimizes long-term energy by controlling processor on-off states, UE communication modes, and RRH precoding vectors.
  • B. Problem Formulation: The state comprises current processor on-off states, UE communication modes, and cache states at D2D transmitters.
  • B. Problem Formulation: Each action changes one processor’s state and one UE’s communication mode, while precoding is handled separately to reduce the action count.
  • B. Problem Formulation: The immediate reward is negative system energy, including running-processor, fronthaul, and wireless-transmission consumption.
  • B. Problem Formulation: Protecting operations reactivate resources and re-optimize precoding when cache transitions, sleeping processors, or D2D interference cause QoS violations.
  • B. Problem Formulation: DRL controls processors and UE modes, after which precoding for RRH transmission is optimized under the resulting conditions.

IV. DRL BASED MODE SELECTION AND RESOURCE MANAGEMENT

The formulation first handles precoding optimization for given processor states and UE modes, then applies a DRL algorithm to control the network under dynamic D2D caching and a complex radio environment.

  • IV. DRL BASED MODE SELECTION AND RESOURCE MANAGEMENT: Precoding is optimized first for fixed processor states and UE communication modes, followed by DRL control under dynamic cache states and radio conditions.

A. Precoding Design with the Computing Resource Constraint

The precoding subproblem is handled iteratively under computing-resource constraints by convexifying difficult terms and updating link sparsity until transmit power converges.

  • Problem transformation: The nonconvex QoS constraint is transformed into a second-order cone constraint, while the l-0 norm is approximated using a re-weighted l-1 norm.These transformations enable iterative solution of the precoding problem.
  • Convex subproblem: Problem (21) is a convex optimization problem that can be efficiently solved by CVX.Convexity follows from the convex objective and constraints, including the re-weighted l-1 norm constraint.
  • Iterative algorithm: Algorithm 1 initializes precoding, computes rates and weights, solves problem (21), and zeros links whose precoding magnitude falls below a threshold.The procedure updates weights after each precoding solution.
  • Convergence: The algorithm repeats the optimization stage until the RRH transmit power consumption converges.This progressively removes UE-antenna links with low transmit power.

B. DRL based Mode Selection and Resource Management

The controller uses DRL to select UE communication modes and processor states in a dynamic caching environment, then optimizes C-RAN precoding for the selected configuration.

  • DRL control policy: DRL addresses mode selection and processor on-off control for an MDP with dynamic cache states and long-term system power consumption as the objective.The learned policy operates without requiring advance knowledge of transition probabilities and rewards.
  • State and action selection: The system state includes UE modes, D2D-transmitter cache states, and processor states, which are input to a DQN producing action Q-values.The controller selects actions using an ε-greedy scheme.
  • Resource-management loop: After each action, the controller optimizes C-RAN precoding with Algorithm 1 and updates the system state through executed actions and cache transitions.This couples discrete control decisions with subsequent radio-resource optimization.
  • Simulation setup: The simulation uses four UEs, paired potential D2D transmitters, two antennas per RRH, and a DQN with 14 input neurons and two hidden layers.Each hidden layer has 24 neurons and uses ReLU activation.
  • Training procedure: Training stores transitions and negative system power consumption rewards in replay memory, then periodically samples mini-batches to update the DQN.A target DQN is initialized from the online network and periodically synchronized.

A. The Impacts of Learning Parameters

The study evaluates how batch size and learning rate affect discounted accumulative system power consumption during DRL training.

  • Batch size: Batch size 32 produces the best performance among the evaluated batch sizes when ρ_m = 0.9 for all UEs.The comparison uses discounted accumulative system power consumption over DRL training.
  • Batch size: A batch size that is too small yields rough gradient estimates, whereas an overly large batch size can trap learning in a local optimum.The two extremes respectively slow policy learning and impair optimization.
  • Learning rate: Learning rate 0.00001 makes DRL learning slow, while learning rate 0.001 results in a local optimum.With batch size 32, the selected learning rate is 0.0001.

B. The Impact of Edge Caching Service Capability

Lower edge caching service capability increases long-term system power consumption because more UEs require RRH service, increasing processor and fronthaul power. Transfer learning reaches performance similar to training from scratch with less training time, but can provide negative guidance when source and target tasks are dissimilar.

  • Lower edge caching service capability leads to higher long-term system power consumption.When cache service capability is poorer, more UEs are served by RRHs, increasing processor and fronthaul power consumption.
  • Expected performance for each caching capability is estimated by testing the corresponding model over 10000 epochs and averaging the results.
  • Transfer learning initializes new DRL models with weights from a model trained at ρm = 0.9 for target environments with ρm = 0.75 and ρm = 0.6.
  • Transfer learning achieves performance similar to training from scratch with much less training time.
  • Transfer learning can provide negative guidance when source and target tasks have low similarity.

D. Performance with Other Baselines

The proposal is evaluated against always-D2D, C-RAN-only DRL, Q-learning, and random-control baselines. Under heterogeneous caching capability, it performs best, while the baselines incur failures, fronthaul and processor costs, or weaker generalization.

  • The evaluation compares the proposal with always-D2D, C-RAN-only DRL, Q-learning-based control, and random control.
  • Under heterogeneous caching capability with ρ1 = 0.5, ρ2 = 0.9, ρ3 = 0.9, and ρ4 = 0.5, the proposal performs best.The comparison uses the performance result shown in Fig. 9.
  • Always-D2D can cause more frequent communication failures because cache states are dynamic and active D2D links interfere with one another.
  • C-RAN-only DRL avoids dynamic-environment effects but induces high fronthaul and processor power consumption by routing all traffic through RRHs.
  • DRL outperforms Q-learning because replay memory reviews historical interactions and DQN generalizes learned knowledge to new situations.
  • The simulations show significant superiority of the DRL approach over the baselines for network control in dynamic and complex wireless environments.
Loading 1809.05629v1…