PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Reinforcement Learning with Spatio-temporal Traffic Forecasting for Data-Driven Base Station Sleep Control

Qiong Wu, Xu Chen, Zhi Zhou, Liang Chen, Junshan Zhang

arXiv:2101.08391v1cs.NIcs.AIcs.DC

TL;DR

Dense BS deployments can waste energy during off-peak periods, while sleep decisions must account for traffic and QoS. DeepBSC combines GS-STN forecasting with reinforcement learning and reports lower costs than the cited baselines.

  • Problem

    Existing sleep-control and forecasting approaches rely on idealized assumptions or omit semantic spatial correlations, while BS decisions must balance energy use and QoS.

    Source: §I · p. 1

  • Method

    DeepBSC combines GS-STN geographical and semantic spatio-temporal traffic forecasting with MDP-based Actor-Critic control using DDPG, benchmark transformation, and an explorer network.

    Source: §I · p. 2

  • Results

    20.5% cost reduction versus ARIMA was reported with GS-STN predictions, and benchmark transformation plus an explorer network yielded 12.8% cost saving versus DQN.

    Source: §I · p. 2

  • Takeaways & Limitations

    The reported results support combining traffic forecasting and deep reinforcement learning for traffic-aware BS sleep control in the evaluated setting.

    Source: §I · p. 2, §VII · p. 13

Abstract

from arXiv · show

To meet the ever increasing mobile traffic demand in 5G era, base stations (BSs) have been densely deployed in radio access networks (RANs) to increase the network coverage and capacity. However, as the high density of BSs is designed to accommodate peak traffic, it would consume an unnecessarily large amount of energy if BSs are on during off-peak time. To save the energy consumption of cellular networks, an effective way is to deactivate some idle base stations that do not serve any traffic demand. In this paper, we develop a traffic-aware dynamic BS sleep control framework, named DeepBSC, which presents a novel data-driven learning approach to determine the BS active/sleep modes while meeting lower energy consumption and satisfactory Quality of Service (QoS) requirements. Specifically, the traffic demands are predicted by the proposed GS-STN model, which leverages the geographical and semantic spatial-temporal correlations of mobile traffic. With accurate mobile traffic forecasting, the BS sleep control problem is cast as a Markov Decision Process that is solved by Actor-Critic reinforcement learning methods. To reduce the variance of cost estimation in the dynamic environment, we propose a benchmark transformation method that provides robust performance indicator for policy update. To expedite the training process, we adopt a Deep Deterministic Policy Gradient (DDPG) approach, together with an explorer network, which can strengthen the exploration further. Extensive experiments with a real-world dataset corroborate that our proposed framework significantly outperforms the existing methods.

I. INTRODUCTION

DeepBSC combines spatio-temporal traffic forecasting with reinforcement learning to control base-station sleep modes while accounting for energy, QoS, and switching costs. Its reported results show lower costs than the stated forecasting and control baselines.

I. INTRODUCTION

  • Traffic-aware sleep control must balance energy savings against QoS degradation from traffic redirection and practical mode-switching costs.The introduction notes that setup time and switching costs make BS mode operation slower than user association.Source: §I · p. 1

A. Mobile Traffic Forecasting

  • Prior traffic forecasting focused on temporal and geographical correlations, although areas with similar traffic patterns need not be geographically close.The paper also identifies idealized traffic-demand and threshold-based assumptions in existing sleep-operation work.Source: §I · p. 1

I. INTRODUCTION

  • The sleep-control problem is formulated as an MDP and solved with Actor-Critic methods using DDPG, benchmark transformation, and an explorer network.Benchmark transformation addresses variance in cost estimation under fluctuating traffic; the explorer network strengthens exploration.Source: §I · p. 2
  • DeepBSC is a data-driven, model-free framework for dynamic sleep control, and GS-STN combines CNN and LSTM with geographical and semantic traffic correlations.The semantic correlations are represented with a traffic similarity graph.Source: §I · p. 2
  • 20.5% cost reduction versus ARIMA was reported with GS-STN traffic predictions, while DeepBSC with benchmark transformation and an explorer network achieved 12.8% cost saving versus DQN.These are distinct comparisons: forecasting baseline and sleep-control baseline, respectively.Source: §I · p. 2

III. TRAFFIC-AWARE SLEEP CONTROL FOR BASE STATIONS

The network model uses half-hour time slots for BS mode decisions. Its cost model accounts for energy consumption, QoS degradation, and the costs of changing BS modes.

A. Network Model

  • Each half-hour time slot keeps BS modes unchanged, while traffic monitors inform dynamic active/sleep decisions across coverage regions that may overlap.Frequent mode changes carry service-migration, delay, and hardware wear costs.Source: A. Network Model · p. 3, Figure 1 · p. 3

B. Base Station Cost Model

  • BS energy cost includes a fixed component and a load-related component rather than scaling simply with traffic load.The fixed component includes circuit and cooling consumption; the load-related component comes from the power amplifier.Source: B. Base Station Cost Model · p. 3
  • The model represents QoS degradation through transmission and service delays, with average queueing delay used to determine service latency.The traffic-dispatching function reallocates regional demand to active BSs; operators can choose a routing scheme, and the experiments use minimum-cost network flow routing.Source: B. Base Station Cost Model · p. 3
  • Switching cost covers service migration, service delay, and hardware wear-and-tear when a BS changes modes between adjacent time slots.The model assigns a cost to switching from sleep to active and assumes the reverse transition costs zero, with any such cost folded into the activation cost if needed.Source: B. Base Station Cost Model · pp. 3–4

C. Problem Formulation

The formulation seeks a BS sleep-control policy that minimizes long-run total system cost. It represents the problem as an MDP whose state uses predicted traffic and the previous slot's BS modes.

C. Problem Formulation

1) State:

  • At the beginning of a time slot, actual traffic volume is unavailable, so the state uses predicted traffic volume and the previous slot's BS modes.The state is represented as s_t = (D̃_t, a_{t−1}).Source: 1) State · p. 4

2) Action and Transition Function:

The framework predicts upcoming traffic before selecting base-station modes, and models sleep control as sequential decisions aimed at minimizing long-term system cost.

2) Action and Transition Function:

  • Each action selects active or sleep modes for all base stations, and the current state and action determine the next-state distribution.The transition is represented by P(s_t+1|s_t, a_t).Source: 2) Action and Transition Function · p. 4

3) Cost Function:

  • The per-slot system cost combines base-station energy use, user QoS degradation, and switching costs; Q-learning estimates accumulated cost to guide long-term minimization.The state-action value is the expected accumulated cost from the current state and action.Source: 3) Cost Function · p. 4

A. Mobile Traffic Forecasting

B. Geographical and Semantic Spatial-Temporal Network

GS-STN combines geographic and semantic spatial signals with temporal traffic modeling to predict next-slot demand; the paper reports improved forecasting over CNN-LSTM.

B. Geographical and Semantic Spatial-Temporal Network

  • A CNN captures geographic correlations by modeling interactions among nearby regions in the traffic map.The spatial neighborhood is represented with an (r+1) × (r+1) grid matrix.Source: B. Geographical and Semantic… · p. 5
  • A semantic traffic-similarity graph captures correlations between grids that need not be geographically close.The graph uses cosine similarity between grid traffic loads, aggregated with a 1D convolution.Source: B. Geographical and Semantic… · p. 5
  • An attention-equipped LSTM models temporal dependencies, and a fully connected layer joins geographic and semantic features to predict next-slot traffic.The model comprises two CNN subnetworks for the spatial dimensions, an LSTM, and a dense layer.Source: B. Geographical and Semantic… · pp. 5–6
  • 8.4% improvement over the CNN-LSTM method in terms of NRMSE is reported for GS-STN traffic forecasting.The model is trained end to end and combines geographic, semantic, and temporal correlations.Source: B. Geographical and Semantic… · p. 6

V. DYNAMIC BASE STATION SLEEP CONTROL

The sleep-control problem is addressed with model-free reinforcement learning and an actor-critic design, using neural networks to handle large state-action spaces.

A. Actor Critic Architecture

  • The actor generates an action from the current state, while the critic evaluates it and uses TD error to guide policy updates toward lower cost.The actor-critic loop repeats until convergence.Source: A. Actor Critic Architecture · p. 6
  • Classical actor-critic value tables are impractical at the paper’s scale of 100 grids and 100 base stations, and some state transitions may never occur.The authors cite exponentially growing storage needs and missing transition examples as obstacles to tabular learning.Source: A. Actor Critic Architecture · p. 6
  • Deep neural networks approximate both the policy and value functions, replacing tabular representations for the large-scale control problem.The actor outputs a deterministic mode-operation vector; the critic evaluates state-action pairs.Source: A. Actor Critic Architecture · p. 6

B. Benchmark Transformation

DeepBSC benchmarks policy cost against a state-specific baseline to reduce the effect of traffic-driven cost fluctuations. An explorer network adds a second action candidate, and selecting it can update the actor toward that action.

B. Benchmark Transformation

  • Benchmark transformation evaluates the gap between current-policy cost and baseline cost, rather than relying on absolute cost that fluctuates with traffic.The baseline approximates the unavailable optimal-policy reference; the gap serves as the criterion for learning how the agent performs against baselines.Source: B. Benchmark Transformation · p. 7
  • BaseDNN learns from collected state–baseline-cost pairs and estimates a state-specific benchmark online during sleep control.Baseline actions are ignored because their estimated cost is treated as an inherent cost of the current environment.Source: B. Benchmark Transformation · p. 7
  • At a half-hour decision scale, benchmark transformation remains useful because stochastic traffic changes and prediction error leave cost-estimation variance even at finer scales.The authors note that much smaller time scales could make prediction and decision-making costly due to their high frequency.Source: B. Benchmark Transformation · p. 7

C. Explorer Network

  • The explorer network perturbs actor parameters to generate another action; if that action has lower cost, the actor is updated toward it.The paper reports that this exploration can make action generation more effective and accelerate training convergence.Source: C. Explorer Network · pp. 7–8

D. DDPG Training Module

DDPG trains the actor and critic using replayed experience and slowly updated target networks. The critic estimates state-action values, while critic feedback guides actor updates despite the unavailable optimal action.

D. DDPG Training Module

  • Slowly tracking target networks constrain target-value changes and improve learning stability.The target actor and critic have the same structures as their learned counterparts.Source: D. DDPG Training Module · p. 8
  • Experience replay stores state, action, cost-gap, and next-state tuples, then samples them randomly for training rather than using only the current experience.The authors state that replay breaks correlations among minibatch experiences; after the proposed enhancements, stored cost gaps maintain small variance.Source: D. DDPG Training Module · p. 8
  • The critic estimates state-action value and is trained to reduce the temporal-difference error against a target value.The critic is updated with randomly sampled minibatches from the replay buffer using stochastic gradient descent.Source: D. DDPG Training Module · p. 8
  • Because the optimal action is nearly impossible to obtain in the large action space, deterministic policy gradients update the actor toward actions with lower critic-estimated cost.The actor and critic parameters are trained by backpropagation; actor updates use the critic gradient with respect to the actor's output action.Source: D. DDPG Training Module · pp. 8–9

E. The Holistic Algorithm

At each time slot, the agent predicts traffic, forms a state from predicted demand and the prior action, and chooses BS modes while estimating baseline cost. It stores observed outcomes and replays experiences to improve its policy.

E. The Holistic Algorithm

  • Each time slot starts with GS-STN predicting current traffic from historical data, which joins the previous action to form the state used for sleep decisions.The agent also estimates the baseline cost for that current state.Source: E. The Holistic Algorithm · p. 9
  • After the slot, the agent observes actual traffic and system cost, stores experience, and samples minibatches to update network parameters.The experience buffer supports subsequent policy learning.Source: E. The Holistic Algorithm · p. 9
  • Because decisions are made on a large time scale, such as half-hour intervals, the algorithm can run in a cloud datacenter or regional central office.The paper identifies these locations as having abundant computing resources.Source: E. The Holistic Algorithm · p. 9

VI. EXPERIMENTS

The experiments use Milan mobile-traffic data and compare GS-STN with forecasting baselines. GS-STN achieves the smallest reported NMAE and NRMSE, with an 8.4% NRMSE improvement over CNN-LSTM.

A. Dataset Description

  • The Milan dataset covers 100 base stations and traffic measurements aggregated to 30-minute intervals across a 10 × 10 grouping of 250 m micro-grids.The underlying map is divided into 100 × 100 micro-grids, and the study groups them for implementation.Source: A. Dataset Description · p. 9

1) Hyperparameter Setting:

  • GS-STN uses 3 × 3 and 1 × 3 convolutional filters with 10 filters per layer, a 48-neuron LSTM, and 100 dense neurons.The LSTM uses 12 time intervals, or six hours; hyperparameters are tuned using random search and cross-validation.Source: 1) Hyperparameter Setting · p. 10

2) Comparing Methods:

3) Traffic Forecasting Performance:

C. Base Station Sleep Evaluation

The evaluation compares DeepBSC variants with established sleep-control policies and examines how benchmark transformation and an explorer network affect learning. It evaluates on the final 10 days after training on the first 20 days.

1) Hyperparameter Settings:

  • The experiment uses a 200-dimensional state containing traffic in 100 grids and operation modes of 100 base stations.The DNN designs and system settings are summarized in Tables III and IV.Source: 1) Hyperparameter Settings · p. 10, Table III · p. 11, Table IV · p. 11
  • The sleep-control baselines include traffic-oblivious control, greedy turn-off, a two-threshold policy, DQN, and DDPG.The two-threshold policy considers traffic volume, QoS degradation cost, and switching cost.Source: 1) Hyperparameter Settings · p. 11
  • The cited passage reports that Deep Deterministic Policy Gradient (DDPG), compared with DQN, can be applied to high-dimensional action spaces through the use of a continuous approximation.Source: 1) Hyperparameter Settings · p. 11
  • Benchmark transformation addresses the large variance in DDPG cost estimation caused by fluctuating mobile traffic demand.The evaluated variants combine benchmark transformation and an explorer network with DDPG.Source: 1) Hyperparameter Settings · p. 11
  • The evaluation compares normalized system cost across different base-station sleep-control methods.The convergence figure compares the training process of different methods.Source: Figure 14 · p. 11, Figure 13 · p. 11

3) Performance Evaluation:

DeepBSC’s DDPG+BT+EN method lowers system cost against the evaluated sleep-control baselines, while traffic forecasting and switching-cost choices also affect outcomes. The paper concludes that its real-world dataset experiments demonstrate the framework’s effectiveness.

3) Performance Evaluation:

  • DDPG-based methods converge faster and achieve at least 6% lower normalized cost than DQN; benchmark transformation yields a 4.464% cost reduction versus DDPG.The explorer network is reported to accelerate convergence.Source: 3) Performance Evaluation · p. 11
  • 7.18%, 12.8%, 21.19%, 29.08%, and 57.33% system-cost reductions are reported for DDPG+BT+EN versus DDPG, DQN, TTP, GOFF, and traffic-oblivious control, respectively.The authors report that each type of cost, including switching and energy cost, is lowest with DDPG+BT+EN.Source: 3) Performance Evaluation · p. 12
  • 5.81% and 20.53% system-cost reductions are reported for traffic predicted by GS-STN versus CNN-LSTM and ARIMA, respectively.The comparison is against traffic demand predicted by the named methods; Fig. 15 presents system costs for different forecasting methods.Source: 3) Performance Evaluation · p. 12, Figure 15 · p. 12
  • At switching-cost ratio 0.1, active base stations range from 26 to 83; at ratio 1, average active-base-station count and total system cost are lowest.At higher switching-cost ratios, the system tends to keep more base stations active, leading to higher system cost.Source: 3) Performance Evaluation · p. 12, Figure 16 · p. 12
  • DDPG+BT+EN produces less variable active-base-station counts and stable QoS, whereas DDPG has a sharp QoS-degradation-cost rise at time slot 45.Figs. 17 and 18 show the two methods’ decisions and corresponding costs over two days.Source: Figure 17 · p. 12, Figure 18 · p. 12, 3) Performance Evaluation · p. 13

VII. CONCLUSION

  • DeepBSC combines GS-STN traffic forecasting with an MDP formulation and Actor-Critic learning, using benchmark transformation and an explorer network.The authors report extensive experiments with a real-world large-scale cellular-network dataset.Source: §VII · p. 13
Loading 2101.08391v1…