Source-linked AI summary
Learning-Based Computation Offloading for IoT Devices with Energy Harvesting
Minghui Min, Dongjin Xu, Liang Xiao, Yuliang Tang, Di Wu
TL;DR
The paper addresses computation offloading for energy-harvesting IoT devices whose battery and channel states complicate optimal decisions. It proposes hotbooting Q-learning and a CNN-based fast DQN, with simulations showing faster learning and improved energy, delay, task-drop, and utility performance. The study leaves joint local power control and offloading decision-making for future work.
Problem
Energy harvesting makes offloading harder because energy and channel states must be considered while battery levels link decisions across time.
Method
The paper uses model-free reinforcement learning, combining hotbooting Q-learning with a CNN-based fast DQN for state-space compression.
Results
The fast DQN converges in approximately 1000 time slots and improves utility, energy consumption, computation delay, and task-drop ratio versus the benchmark schemes.
Takeaways & Limitations
The proposed schemes can achieve optimal offloading performance after sufficiently long learning, while fast DQN accelerates learning in dynamic MEC.
Takeaways & Limitations
Joint local power control and offloading decision making remains an open technical challenge.
Abstract
from arXiv · showhide
Internet of Things (IoT) devices can apply mobile-edge computing (MEC) and energy harvesting (EH) to provide the satisfactory quality of experiences for computation intensive applications and prolong the battery lifetime. In this article, we investigate the computation offloading for IoT devices with energy harvesting in wireless networks with multiple MEC devices such as base stations and access points, each with different computation resource and radio communication capability. We propose a reinforcement learning based computation offloading framework for an IoT device to choose the MEC device and determine the offloading rate according to the current battery level, the previous radio bandwidth to each MEC device and the predicted amount of the harvested energy. A "hotbooting" Q-learning based computation offloading scheme is proposed for an IoT device to achieve the optimal offloading performance without being aware of the MEC model, the energy consumption and computation latency model. We also propose a fast deep Q-network (DQN) based offloading scheme, which combines the deep learning and hotbooting techniques to accelerate the learning speed of Q-learning. We show that the proposed schemes can achieve the optimal offloading policy after sufficiently long learning time and provide their performance bounds under two typical MEC scenarios. Simulations are performed for IoT devices that use wireless power transfer to capture the ambient radio-frequency signals to charge the IoT batteries. Simulation results show that the fast DQN-based offloading scheme reduces the energy consumption, decreases the computation delay and the task drop ratio, and increases the utility of the IoT device in dynamic MEC, compared with the benchmark Q-learning based offloading.
SYSTEM MODEL
The system models an energy-harvesting IoT device that selects one of multiple MEC devices and controls how much task data to offload. The task may be processed locally, fully offloaded, or split between local and MEC computation.
- The system contains an energy-harvesting IoT device and M MEC devices.
- The IoT device selects MEC device i according to the radio channel state.
- The quantized offloading rate x(k) determines the task split between local execution and MEC processing.x(k)=0 means fully local execution; x(k)=1 means full offloading; intermediate values split the task.
Computation-Offloading Model
The offloading model characterizes transmission bandwidth, duration, and energy consumption when the IoT device sends task data to a selected MEC device. These quantities depend on the offloaded data and radio transmission conditions.
- The IoT device transmits tasks to selected MEC device i using radio bandwidth B_i(k).
- The transmission duration T_i(k) represents the time required to offload x(k)C(k) bits to MEC device i.
- The IoT device consumes offloading energy E_i(k) when transmitting tasks to MEC device i.
- Offloading energy depends on transmit power P_i(k) and transmission duration T_i(k).
Local-Computing Model
The local-computing model relates execution workload to CPU-cycle frequency, latency, and energy consumption. Dynamic voltage and frequency scaling allows the IoT device to adjust local-computing energy within a bounded frequency range.
- Local-Computing Model: Local execution processes (1−x(k))C(k) input bits using N CPU cycles per input bit.The resulting workload is (1−x(k))C(k)N CPU cycles.
- Local-Computing Model: Dynamic voltage and frequency scaling controls local execution energy by adjusting the CPU frequency f_m.
- Local-Computing Model: The CPU frequency f_m is bounded by f_max, limiting the IoT device's computation capability.
- Local-Computing Model: The local-computing model defines execution latency T_0(k) and energy E_0(k) for the locally processed task portion.
- Local-Computing Model: The effective capacitance coefficient ς depends on the chip architecture.
Energy Harvesting Model
Energy harvesting couples battery state, radio conditions, and offloading decisions across time. The paper formulates model-free reinforcement-learning schemes, including hotbooting Q-learning and a CNN-based fast DQN, to optimize utility under energy constraints.
- Energy Harvesting Model: Harvested energy ρ(k) is stored in the IoT battery and estimated as ˆρ(k) from harvesting history and a modeling method.The estimation error is △(k)=ρ(k)−ˆρ(k).
- Energy Harvesting Model: In wireless-powered MEC, harvested energy is ˜ρ(k)=υη(k)h(k), combining harvesting efficiency, transmitter power, and channel gain.
- Energy Harvesting Model: Energy-harvesting devices require both energy-state and channel-state information because battery levels temporally link decisions across time slots.
- Energy Harvesting Model: The offloading process is modeled as an MDP with state information comprising previous bandwidths, predicted harvested energy, and current battery level.
- Energy Harvesting Model: Hotbooting Q-learning initializes Q-values from offloading experience in similar environments to accelerate early learning.
- Energy Harvesting Model: Insufficient battery energy causes task drops, while utility accounts for data-sharing gains, task-drop loss, energy consumption, and computation delay.
- Energy Harvesting Model: The Q-learning learning time grows with the action-state space, and larger spaces can cause serious performance degradation.
FAST DQN-BASED COMPUTATION OFFLOADING
The fast DQN scheme uses a CNN and hotbooting to accelerate learning for dynamic computation offloading. It represents recent system experience, selects MEC devices and offloading rates, and updates CNN weights through replayed experiences.
- Fast DQN architecture: The fast DQN scheme uses a CNN as a nonlinear Q-value approximator to compactly store information across similar states and accelerate hotbooting Q-learning convergence.The CNN outputs Q-values for computation offloading policies from the current experience sequence.
- Experience representation: The CNN input extends the current system state with the previous W system state-action pairs, forming an experience sequence for learning.When fewer than W experiences exist, the device chooses the offloading policy randomly.
- Offloading loop: At each time slot, the device selects an MEC device and quantized offloading rate, sends x(k)C(k) data, and observes bandwidths and battery level.The policy uses ε-greedy selection after observing CNN-produced Q-values.
- Learning updates: The device evaluates utility from delay, energy consumption, task-drop loss, and data-sharing gains, then stores experiences and trains CNN weights with random minibatches and SGD.Experience replay samples stored transitions for repeated updates.
- Fast DQN architecture: Hotbooting initializes CNN weights from emulational experience instead of random values to accelerate learning.The device stores emulational state-action-utility transitions and uses the resulting weights for initialization.
PERFORMANCE EVALUATION
The paper analyzes optimal single-slot offloading as an upper bound for reinforcement-learning schemes. Depending on transmission conditions, the optimal policy is either full offloading to the highest-bandwidth MEC device or full local execution.
- Optimal policy: Single-slot optimal offloading provides a performance upper bound for evaluating reinforcement-learning schemes over sufficiently long dynamic processes.The device chooses the MEC device and offloading proportion that maximize utility.
- Optimal policy: When transmission delay and energy consumption are low, Q-learning eventually selects full offloading to the MEC device with the largest radio bandwidth.The converged policy is a∗ = [i∗, 1] with probability 1 after sufficiently long learning.
- Optimal policy: When transmission delay to the highest-bandwidth MEC device is high, the device processes all computation tasks locally to maximize utility.The converged policy is a∗ = [i∗, 0] with probability 1 after sufficiently long learning.
SIMULATION RESULTS
Simulations evaluate learning convergence and offloading performance under dynamic MEC conditions with wireless power transfer. Fast DQN converges rapidly and improves utility, task-drop ratio, energy consumption, and delay relative to benchmark schemes.
- Dynamic learning performance: Fast DQN achieves optimal computation-offloading performance after convergence, with all computation tasks offloaded to an MEC device under the stated conditions.The simulation uses 100-bit tasks, 6 W energy-transmitter power, and three MEC devices with randomly selected bandwidths.
- Dynamic learning performance: Fast DQN converges within approximately 1000 time slots, while hotbooting Q-learning and Q-learning require much longer to converge.The fast DQN scheme achieves optimal performance after convergence under the stated simulation conditions.
- Dynamic learning performance: 2 times higher utility is achieved by fast DQN at time slot 1000 than by hotbooting Q-learning.Hotbooting Q-learning itself has faster learning and higher utility than benchmark Q-learning.
- Task-drop performance: 82 percent lower task drop ratio results when RF transmit power increases from 6 to 10 for hotbooting Q-learning.At RF transmit power 8, fast DQN has a task drop ratio 53 percent below hotbooting Q-learning, which is 68 percent below benchmark Q-learning.
- Task-size sensitivity: 40 percent higher energy consumption, 46 percent longer computation delay, and 4 times higher task drop ratio occur when task size increases from 100 to 140 bits under fast DQN.At 120 bits, fast DQN uses 23 percent less energy, has 4 percent shorter delay, and achieves 50 percent lower task drop ratio than hotbooting Q-learning.
CONCLUSION
The paper presents reinforcement-learning offloading for energy-harvesting IoT devices, including low-complexity hotbooting Q-learning and CNN-based fast DQN. It derives convergence conditions and performance bounds, while identifying joint local power control and offloading as future work.
- Contributions: The proposed framework selects MEC devices and offloading rates without requiring MEC, local-computation, or energy-consumption models.It uses reinforcement learning for energy-harvesting IoT computation offloading.
- Contributions: Hotbooting Q-learning accelerates learning through transfer learning while maintaining low computational complexity.Fast DQN instead compresses the learning state space with a CNN.
- Analysis and validation: Performance bounds and conditions for fully offloaded and locally processed computation are provided under two typical MEC scenarios.Simulations with RF-signal wireless power transfer verify theoretical results and demonstrate the proposed schemes.
- Future work: Joint local power control and offloading decision-making remains a future research problem.The paper characterizes this direction as technically challenging.