Source-linked AI summary
Reinforcement Learning based Multi-Access Control and Battery Prediction with Energy Harvesting in IoT Systems
Man Chu, Hang Li, Xuewen Liao, Shuguang Cui
TL;DR
Energy-harvesting IoT access control and battery prediction are challenging when system dynamics are unknown. The paper develops separate and joint LSTM-based reinforcement-learning methods, with simulations reporting improvements over scheduling benchmarks and low prediction losses.
Problem
Unknown or non-stationary energy dynamics can make prior knowledge of system behavior difficult to obtain for multiuser access control and battery prediction.
Source: A. Related Works and Motivations · p. 1Method
The paper proposes LSTM-DQN access control, deep LSTM battery prediction under round-robin scheduling, and a jointly trained two-layer LSTM-DQN.
Source: B. Our Contributions · p. 2, §VII · p. 11Results
After stabilization, access-control reward is 11.93% higher than MP, while the joint algorithm’s reward is 9.96% higher than MP; prediction losses are about 0.0013 and 0.0175 units, respectively.
Source: §VI · pp. 10–11Takeaways & Limitations
The simulations report that the proposed learning algorithms perform better than their baseline approaches across the studied access-control and prediction tasks.
Source: §VI · pp. 9–11
Abstract
from arXiv · showhide
Energy harvesting (EH) is a promising technique to fulfill the long-term and self-sustainable operations for Internet of things (IoT) systems. In this paper, we study the joint access control and battery prediction problems in a small-cell IoT system including multiple EH user equipments (UEs) and one base station (BS) with limited uplink access channels. Each UE has a rechargeable battery with finite capacity. The system control is modeled as a Markov decision process without complete prior knowledge assumed at the BS, which also deals with large sizes in both state and action spaces. First, to handle the access control problem assuming causal battery and channel state information, we propose a scheduling algorithm that maximizes the uplink transmission sum rate based on reinforcement learning (RL) with deep Q-network (DQN) enhancement. Second, for the battery prediction problem, with a fixed round-robin access control policy adopted, we develop a RL based algorithm to minimize the prediction loss (error) without any model knowledge about the energy source and energy arrival process. Finally, the joint access control and battery prediction problem is investigated, where we propose a two-layer RL network to simultaneously deal with maximizing the sum rate and minimizing the prediction loss: the first layer is for battery prediction, the second layer generates the access policy based on the output from the first layer. Experiment results show that the three proposed RL algorithms can achieve better performances compared with existing benchmarks.
I. INTRODUCTION
The paper studies uplink access control and battery prediction for energy-harvesting IoT systems where energy arrivals may be unpredictable and system dynamics may be unknown. It proposes separate and joint learning-based methods for these tasks.
A. Related Works and Motivations
- Existing online approaches may require system statistics, while unknown or non-stationary energy dynamics make prior knowledge difficult to obtain.The paper motivates model-free learning for settings where such knowledge is unavailable.Source: A. Related Works and Motivations · pp. 1–2
- Prior battery-prediction methods are described as source-restricted or dependent on known harvesting dynamics, motivating a joint design that feeds predictions into access control.The paper identifies joint access control and battery prediction as its focus.Source: A. Related Works and Motivations · p. 2
B. Our Contributions
- With current battery and channel states, LSTM-DQN targets long-term discounted transmission data rather than only instantaneous sum rate.The stated goal is more stable and balanced transmission over a long horizon.Source: B. Our Contributions · p. 2
- With round-robin access and scheduled users’ true battery states embedded in transmissions, deep LSTM prediction minimizes battery-state prediction loss.The loss is defined over predicted versus true battery states within the selected UE set.Source: B. Our Contributions · p. 2
- A two-layer LSTM-DQN jointly predicts battery levels and uses them with channel information to schedule access, optimizing sum rate and prediction loss.The first layer predicts battery levels; the second generates the access policy.Source: B. Our Contributions · p. 2
- The proposed access-control, battery-prediction, and joint algorithms outperform baseline approaches in simulations.The paper evaluates the three algorithms under different scenarios.Source: B. Our Contributions · p. 2
II. SYSTEM MODEL AND PRELIMINARIES
The system models multiple energy-harvesting UEs sharing limited uplink access under finite batteries, with channel and battery states represented in an MDP. DQN approximates action values in the large state and action spaces, while LSTM units process long-range correlated patterns.
A. System Model
- The uplink system has N energy-harvesting UEs and one BS, with K of N selected each time slot and finite-capacity rechargeable batteries.Harvested energy is stored for the next slot; transmission consumes energy, and excess energy is discarded when a battery is full.Source: A. System Model · p. 3
- The BS observes each UE's channel gain and battery state, while UEs use harvested energy as their only power source.Channel conditions remain constant within a slot, and battery states are quantized for analysis.Source: A. System Model · p. 3
B. Preliminaries: Deep Q-Learning and LSTM
- The MDP represents system states, actions, rewards, and transitions; the BS seeks a policy that optimizes long-run reward or loss.The policy maps states to actions, and value functions and Bellman equations support recursive evaluation.Source: B. Preliminaries: Deep Q-Learning and… · pp. 3–4
- Each state contains every UE's channel gain and battery level, making both state and action spaces large in this multiuser setting.The action-space size grows in proportion to the number of UEs.Source: B. Preliminaries: Deep Q-Learning and… · p. 3
- DQN uses a deep neural network to approximate action values across large state-action spaces, where maintaining every Q-value is impractical.The passage notes that nonlinear function approximation can be unstable and lacks usual theoretical guarantees.Source: B. Preliminaries: Deep Q-Learning and… · p. 4
- LSTM units pass information between successive blocks and use four gates to manage what is forgotten, stored, and output.The network is described as recognizing long-range correlated patterns in input and output states.Source: B. Preliminaries: Deep Q-Learning and… · p. 4, Figure 2 · p. 4
C. Performance Metric
The paper evaluates network performance using uplink sum rate and battery-state prediction loss. The prediction loss is based on selected UEs at each slot, while exploration brings all UEs into account over time.
C. Performance Metric
- Uplink sum rate measures the combined rate of transmissions to the BS.The rate is defined per time slot; its expression uses bandwidth and noise power.Source: C. Performance Metric · p. 4
- Prediction loss measures dissimilarity between predicted and true battery states, using only selected UEs' instantaneous states.Because the RL algorithm explores all UEs, all UEs are included in the loss over the long run.Source: C. Performance Metric · p. 4
III. ACCESS CONTROL WITH RL
The access-control problem assumes cooperative BS and UEs, with current channel gains and battery states available to the BS. The BS seeks a policy that maximizes long-term expected uplink sum rate while selecting K UEs per slot.
III. ACCESS CONTROL WITH RL
- The BS uses current UE battery states and channel information to schedule access, then broadcasts the policy to the UEs.Selected UEs transmit at power P; unselected UEs remain idle, and all UEs harvest and store energy for future use.Source: §III · p. 5
A. Problem Formulation
- The system state combines current channel gains and battery states, while each action selects K UEs from the available set.The reward is the received sum rate at the BS, and the objective is to maximize expected cumulative discounted reward.Source: A. Problem Formulation · p. 5
- The access-control optimization maximizes long-term expected uplink sum rate subject to the system constraints.The stated constraints refer to the channel and battery evolution equations and the fixed number of scheduled UEs.Source: A. Problem Formulation · p. 5
B. LSTM-DQN based Access Control Network
The access controller uses an LSTM-DQN network to estimate action values from system states. It balances exploration and exploitation, and trains from replayed transitions using a target value and gradient updates.
B. LSTM-DQN based Access Control Network
- The LSTM-DQN access-control network maps system state input to approximated Q-values for the action space.A fully connected layer adjusts the output vector to match the action-space size.Source: B. LSTM-DQN based Access Control Network · p. 5, Figure 3 · p. 5
- An ϵ-greedy rule explores by choosing random actions with probability ϵ and exploits the greedy action with probability 1 − ϵ.The greedy choice is based on estimated action values, which may be inaccurate before learning converges.Source: B. LSTM-DQN based Access Control Network · p. 5
- Experience replay stores state-action-reward-next-state transitions and randomly samples them to update the network.The paper motivates random sampling as a way to avoid issues from strong correlations among transitions in the same episode.Source: B. LSTM-DQN based Access Control Network · p. 5
- The algorithm computes a target from sampled rewards and next-state Q-values, then uses stochastic gradient descent to update network parameters.The loss is the squared difference between the target and the estimated Q-value for the sampled state-action pair.Source: B. LSTM-DQN based Access Control Network · p. 6
IV. BATTERY PREDICTION WITH RL
The battery predictor is designed to minimize discounted prediction loss without requiring a model of the UE energy process. With access control fixed to round-robin, the BS uses histories of schedules, predicted batteries, and selected UEs’ true batteries to form its state.
IV. BATTERY PREDICTION WITH RL
- The approach reuses an LSTM method for UE battery-state prediction and is data-driven, requiring no UE energy model.Source: §IV · p. 6
A. Problem Formulation
- Round-robin scheduling is fixed, and selected UEs provide their true battery states after transmission for comparison with predictions.The difference between predicted and true states within the selected UE set is the predictor’s performance metric.Source: A. Problem Formulation · p. 6
- The BS state includes a history window of access schedules, predicted battery states, and true battery states from selected UEs.The memory component stores these three histories over W time slots.Source: A. Problem Formulation · p. 6
- The learning objective is to minimize cumulative discounted battery-prediction loss using observed states.The section motivates prediction as useful for energy-neutral operation, where energy expenditure equals harvested energy over ongoing operation.Source: A. Problem Formulation · p. 6, §IV · p. 6
B. Deep LSTM based Battery Prediction Network
The prediction generator maps the BS’s observed history state to battery predictions for all UEs. It is trained from stored experience using stochastic gradient descent and temporal-difference policy evaluation.
B. Deep LSTM based Battery Prediction Network
- The LSTM prediction generator takes the BS history state as input and outputs predicted battery states for all UEs.A fully connected layer adjusts the LSTM output to a one-row vector of N predictions.Source: B. Deep LSTM based Battery Prediction… · p. 6
- Experience replay stores state, prediction, received true battery states, and next-state tuples, which are sampled to update network weights.Source: B. Deep LSTM based Battery Prediction… · p. 7
- Stochastic gradient descent updates the prediction weights to reduce loss, using TD(0) policy evaluation and a target output for each training step.The update uses the prediction error and the gradient of the parameterized prediction values.Source: B. Deep LSTM based Battery Prediction… · p. 7
V. JOINTLY ACCESS CONTROL AND BATTERY PREDICTION BASED ON RL
The joint approach uses a first LSTM layer to predict UE batteries and a second layer to generate access control. It combines transmission sum rate and prediction loss in the received reward, while relaxing the need for current battery states at the BS for access control.
V. JOINTLY ACCESS CONTROL AND BATTERY PREDICTION BASED ON RL
- The joint approach relaxes the BS requirement for current UE battery states, so only current channel gains are needed at each time slot.Source: §V · p. 7
- The described battery-prediction algorithm initializes experience memory and network weights, repeatedly samples transitions, and updates weights using stochastic gradient descent.Source: §V · p. 7
- The first LSTM layer predicts all UEs’ battery states from history, and the second layer uses those predictions to generate the access policy.Source: §V · p. 7
- After selected UEs transmit and report true battery states, the BS stores them for future prediction and receives a reward mixing sum rate with prediction loss.Source: §V · p. 7
A. Problem Formulation
The joint formulation represents the BS state using scheduling, predicted-battery, and selected-UE true-battery histories together with current channel gain. Its policy seeks to maximize discounted reward that balances sum rate against prediction loss.
A. Problem Formulation
- The joint system state combines histories of access indicators, predicted batteries, and selected UEs’ true batteries with the current channel gain.The BS is the centralized controller and stores the histories over a window of W time slots.Source: A. Problem Formulation · pp. 7–8
- The immediate reward combines received sum rate with battery-prediction loss treated as a penalty, balanced by a penalty factor.Source: A. Problem Formulation · p. 8
- The joint learning objective is to find a scheduling policy that maximizes cumulative discounted reward.Source: A. Problem Formulation · p. 8
B. Two-Layer LSTM-DQN based Joint Network
The joint control network combines battery prediction with access-policy selection in a two-layer LSTM-DQN. Its two generators are trained jointly, with predicted battery states and channel information feeding the scheduling decision.
B. Two-Layer LSTM-DQN based Joint Network
- The first LSTM-based prediction generator uses historical system information to estimate UE battery states, while the action generator combines those estimates with channel states to produce Q-values for access control.A fully connected layer adjusts each generator’s output size.Source: B. Two-Layer LSTM-DQN based Joint… · p. 8
- The proposed joint network learns the prediction-generator and action-generator parameters together.The network is trained using gradients that update both parameter sets.Source: B. Two-Layer LSTM-DQN based Joint… · pp. 8–9
- At each time step, the BS uses the predicted battery states and channel gains to generate Q-values, then selects actions with an ϵ-greedy policy.With probability ϵ it selects randomly; otherwise it chooses the action with the highest Q-value.Source: B. Two-Layer LSTM-DQN based Joint… · p. 9
VI. SIMULATION RESULTS
In simulations, the joint LSTM-DQN outperformed the listed online scheduling benchmarks and approached the offline upper bound, while its battery-prediction loss stabilized at a reported low value. The separate access-control and prediction algorithms also showed gains over their respective benchmarks.
Simulation setup
- The simulations used one BS and 30 randomly walking UEs in a simulated LTE uplink, with Poisson energy arrivals and a 5 MHz system bandwidth.The compared approaches included an offline upper bound, MP, round-robin, and random scheduling.Source: §VI · p. 9
Access control
- 11.93% higher average reward than MP, 19.47% higher than round-robin, and 26.05% higher than random scheduling, while 3.3% lower than offline scheduling, for the DQN access-control algorithm after stabilization.Across 2–20 available channels and battery capacities of 5–10 units, the DQN algorithm beat the counterparts.Source: Figure 6 · p. 10, §VI · p. 10, Figure 7 · p. 10
Battery prediction
- 0.0013 units was the reported average UE battery prediction loss for the separate prediction algorithm, which stabilized after about 1000 training steps.The loss was initially large and decreased with training.Source: Figure 8 · p. 10, §VI · p. 10
Joint scheme
- After stabilization, the proposed joint algorithm’s average reward was 9.96% higher than MP, 21.1% higher than round-robin, and 24.7% higher than random scheduling, but 4.14% lower than offline scheduling.Source: §VI · p. 11
- 0.0175 units was the reported average UE battery prediction loss for the joint scheme, which stabilized after about 22000 training steps.The prediction loss stabilized earlier than the average sum rate.Source: Figure 11 · p. 11, §VI · p. 11
VII. CONCLUSION
The paper concludes that its three RL-based methods address access control, battery prediction, and their joint optimization in multi-user energy-harvesting communications. Simulations under different conditions illustrate their effectiveness.
VII. CONCLUSION
- The paper develops three RL-based methods for access control and battery prediction in a multi-user energy-harvesting communication system.The methods address scheduling, prediction, and the joint problem.Source: §VII · p. 11
- The joint two-layer LSTM network is trained with deep Q-learning to maximize long-term discounted sum rate and minimize cumulative battery-prediction loss simultaneously.The prediction network feeds the access-control network.Source: §VII · p. 11
- Simulations under different conditions illustrate the effectiveness of the proposed RL-based methods.Source: §VII · p. 11