PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-agent Reinforcement Learning for Resource Allocation in IoT networks with Edge Computing

Xiaolan Liu, Jiadong Yu, Yue Gao

arXiv:2004.02315v1eess.SPcs.LG

TL;DR

The paper studies multi-user IoT edge offloading where users contend for limited computation and spectrum resources and may observe only local information. It models joint offloading and resource choices as a stochastic game and uses independent multi-agent Q-learning; simulations report feasibility and energy-efficiency advantages over benchmark schemes without extra gateway channel-estimation cost.

Abstract

from arXiv · show

To support popular Internet of Things (IoT) applications such as virtual reality, mobile games and wearable devices, edge computing provides a front-end distributed computing archetype of centralized cloud computing with low latency. However, it's challenging for end users to offload computation due to their massive requirements on spectrum and computation resources and frequent requests on Radio Access Technology (RAT). In this paper, we investigate computation offloading mechanism with resource allocation in IoT edge computing networks by formulating it as a stochastic game. Here, each end user is a learning agent observing its local environment to learn optimal decisions on either local computing or edge computing with the goal of minimizing long term system cost by choosing its transmit power level, RAT and sub-channel without knowing any information of the other end users. Therefore, a multi-agent reinforcement learning framework is developed to solve the stochastic game with a proposed independent learners based multi-agent Q-learning (IL-based MA-Q) algorithm. Simulations demonstrate that the proposed IL-based MA-Q algorithm is feasible to solve the formulated problem and is more energy efficient without extra cost on channel estimation at the centralized gateway compared to the other two benchmark algorithms.

I. INTRODUCTION

IoT edge computing brings processing closer to users, but limited edge capacity and competing computation, spectrum, and radio-access resources make allocation difficult. The paper addresses multi-user offloading with distributed learning, against prior work centered on single-user or centralized settings.

I. INTRODUCTION

  • Edge devices have limited capacity, while offloading consumes computation and spectrum resources; resource allocation across computation, radio access technologies, and spectrum is therefore important.The paper describes moving data to distant cloud centers as requiring wide spectrum and potentially causing prohibitive transmission delay.Source: §I · p. 1

A. Related Work

  • Prior computation-offloading research includes single-user optimization, multi-user resource allocation, and approaches using TDMA, OFDMA, energy harvesting, or millimeter-wave communications.The related work also describes market-based resource allocation for computation, spectrum, and radio access technologies.Source: A. Related Work · p. 1

B. Motivation and Contributions

  • The paper identifies a gap in multi-user IoT offloading under local observations, noting that earlier learning work often focused on single users or centralized control.It proposes a stochastic game and independent learners based multi-agent Q-learning for this setting.Source: B. Motivation and Contributions · p. 2
  • The proposed IL-based MA-Q algorithm lets users learn offloading policies independently, without inter-user communication, and is reported superior to centralized and random schemes while using less energy without extra channel-estimation cost.The paper also frames the scheme as reducing computation burden on the centralized gateway.Source: B. Motivation and Contributions · p. 2

C. Organization

  • The paper is organized from system modeling and Markov-game formulation through the IL-based MA-Q algorithm, simulations, and conclusions.Figure 1 is identified as the computation-offloading model in IoT networks.Source: C. Organization · p. 2, Figure 1 · p. 3

II. SYSTEM MODEL

The system model considers users that can process tasks locally or offload them to an edge server, selecting radio resources for transmission. It specifies local and edge execution, channel-based offloading, and a slotted decision process.

II. SYSTEM MODEL

  • The network comprises single-antenna users and an edge server; users with limited computation capacity choose local execution or gateway offloading for latency-constrained tasks.Users select a radio access technology and one of its orthogonal subchannels; each technology operates on a different frequency.Source: §II · p. 3

1) Local Computing Model:

  • Local execution consumes user-device computing energy, with task delay determined by the user's computation capacity.The model defines the required CPU cycles per bit and the energy consumed per CPU cycle.Source: 1) Local Computing Model · p. 3

2) Edge Computing Model:

  • Edge execution uses the edge server's greater computation capacity and models its computation time and energy for processing offloaded tasks.The edge server is described as having a more stable power supply than local devices.Source: 2) Edge Computing Model · p. 3

B. Task offloading Model

  • For offloading, users select transmit power, radio access technology, and subchannel; transmission is modeled with channel gains, interference, and noise.The model assumes Rayleigh fading and discrete transmit-power control; zero transmit power denotes local execution.Source: B. Task offloading Model · pp. 3–4
  • In each non-overlapping slot, users execute decisions made in the preceding slot, receive gateway feedback, and decide task execution for the following slot.Communication parameters are assumed unchanged within each slot.Source: B. Task offloading Model · p. 4, Figure 2 · p. 4

III. PROBLEM FORMULATION WITH MARKOV GAME

The computation-offloading problem is modeled as a Markov game in which multiple users jointly choose task execution and communication resources.

III. PROBLEM FORMULATION WITH MARKOV GAME

  • The paper formulates joint task-execution, transmit-power, radio-access-technology, and subchannel selection by multiple users as a Markov game.The formulation concerns IoT edge-computing networks.Source: §III · p. 4
  • The modeled decisions include both whether to execute locally or offload and which communication resources to use.The choices are made by multiple users in the network.Source: §III · p. 4
  • The formulation treats the resource-allocation decisions as a joint multi-user problem rather than separate single-user choices.The passage specifies joint selection across task execution, transmit power, RAT, and subchannel.Source: §III · p. 4

A. Computation Offloading Problem Formulation

The formulation evaluates offloading and resource decisions through a long-term system-cost objective. Users act from local observations and face unobserved actions and channel gains from other users.

A. Computation Offloading Problem Formulation

  • The system cost weights energy consumption and task execution delay, while transmit power trades transmission rate and delay against interference to other users.Offloading also consumes shared spectrum and computation resources that could otherwise be allocated to other users.Source: A. Computation Offloading Problem… · p. 4
  • Each user's state includes local execution or gateway transmission, and its action selects transmit power, radio access technology, and subchannel.The action aims to maximize long-term reward, with cost represented as negative reward and therefore minimized over time.Source: A. Computation Offloading Problem… · pp. 4–5
  • Users observe their own actions, channel gain, and remaining edge capacity, but not other users' actions or channel gains.The paper frames the multi-user problem as a non-cooperative stochastic game and proposes multi-agent reinforcement learning to solve it.Source: A. Computation Offloading Problem… · p. 5

B. Stochastic Game

The paper models multi-user computation offloading as a stochastic game, where each end user learns independently and its reward and next state depend on the joint actions.

B. Stochastic Game

  • The game framework is used to model multi-user computation offloading while minimizing system cost.Source: B. Stochastic Game · p. 5
  • Each end user is an agent that independently chooses transmit power, radio access technology, and sub-channel based on its observed state.The user's reward and next state depend on the actions of all users.Source: B. Stochastic Game · p. 5
  • The stochastic game is represented by the tuple <S, U, A, P, R>, covering states, player count, joint actions, transitions, and rewards.The environment state and joint action set are products of the respective users' states and actions.Source: B. Stochastic Game · p. 5

C. Computation Offloading Game Formulation

The formulation treats each end user's resource choices as an action and defines its state from local observations, including indicators of transmission, gateway recognition, and capacity.

1) Action:

  • At each time slot, an end user's action includes its transmit power level, radio access technology, and sub-channel selection.Source: 1) Action · p. 5
  • The action space combines each user's transmit-power, radio-access-technology, and sub-channel choices.Source: 1) Action · p. 5

2) State:

  • Because users do not cooperate, each user's state is defined from its local observations.Source: 2) State · p. 5
  • The state includes an indicator of whether the user's transmit power is zero and an indicator of whether the gateway recognizes its offloading.Recognition is determined by whether the received SINR is above or below its threshold.Source: 2) State · p. 5
  • The other-user action vector a−i(k) represents the actions of all end users except ui.Source: 2) State · p. 5
  • The formulation describes the user's expected discounted return conditioned on its current state.Source: 2) State · p. 6
  • Ji(k) indicates whether the gateway broadcasts that its computation capacity can support all tasks from offloading users.Source: 2) State · p. 6

3) Reward:

Users evaluate strategies through rewards shaped by joint actions and seek a Nash equilibrium; the paper then introduces its multi-agent reinforcement-learning framework and Q-learning algorithm.

3) Reward:

  • A user's immediate reward depends on its own action and the actions of the other end users in the current state.Source: 3) Reward · p. 6
  • Each user seeks a strategy that maximizes its discounted expected reward, with the state transition determined by all users' strategies.Source: 3) Reward · p. 6
  • A Nash equilibrium is a strategy set in which no user can obtain a higher reward by changing only its own strategy.Source: 3) Reward · p. 6
  • The paper states that its formulated computation-offloading game has a Nash equilibrium and aims for each user to find one for every state.Source: 3) Reward · p. 6

IV. PROPOSED MULTI-AGENT REINFORCEMENT LEARNING ALGORITHM

  • The paper proposes a multi-agent reinforcement-learning framework and a multi-agent Q-learning algorithm to address the formulated offloading game.Source: §IV · p. 6

A. MARL Framework for Multi-user IoT edge computing network

The framework models users' joint actions as driving a Markov game, then uses state-action values and independent learning to find each user's strategy.

A. MARL Framework for Multi-user IoT edge computing network

B. Multi-Agent Q-learning for Computation Offloading

The proposed IL-based MA-Q algorithm has each user learn an individual policy through independent Q-learning. It uses sampled returns and temporal-difference updates, while the convergence discussion bounds the result to a near-optimal policy under finite state and action approximations.

B. Multi-Agent Q-learning for Computation Offloading

  • Because state-transition probabilities and rewards are impractical to obtain, sampled returns approximate expected cumulative reward, and temporal-difference learning recursively updates Q-values.The passage notes that Monte Carlo learning over complete episodes makes mean-return calculation complicated.Source: B. Multi-Agent Q-learning for… · p. 7
  • The learning rate α is bounded by 0 < α ≤ 1 and is scheduled from given initial and final values over the algorithm's maximum iterations.The learning-rate schedule is stated to support Q-learning convergence.Source: B. Multi-Agent Q-learning for… · p. 8
  • Each end user independently runs Q-learning and maintains only its own Q table to learn an individual optimal strategy for its MDP.The algorithm addresses computation offloading with resource allocation in a multiuser IoT edge network.Source: B. Multi-Agent Q-learning for… · pp. 7–8

C. Convergence Performance of the Proposed Algorithm

  • With finite state and action approximations and Monte Carlo policy exploration, the paper says it can obtain a near-optimal policy.The convergence discussion states that Q-learning's convergence result assumes continued updates of all state-action pairs and infinite policy exploration.Source: C. Convergence Performance of the… · p. 8

V. SIMULATION RESULTS

Simulations examine IL-based MA-Q convergence, performance as the number of users changes, and computation offloading and channel allocation. The algorithm has lower average cost than the Random benchmark and, for fewer than 27 users, the Centralized benchmark, while avoiding centralized channel estimation.

V. SIMULATION RESULTS

  • Each user independently runs a Q-table and interacts with the environment; simulations use normally distributed user-to-gateway distances and WiFi and LoRa access technologies.The described setting uses µ = 1000, σ = 3, a 1 km radius, Rayleigh fading parameter B = 3, and path-loss parameter a = 2.5.Source: §V · p. 8
  • User u1's average cost converges as iterations increase, while ǫ = 0.9 produces worse cost because it prompts too many random actions.The simulation uses 10,000 episodes for the curves.Source: §V · pp. 8–9, Figure 4 · p. 9
  • Average cost per time slot rises with more users because limited channels make gateway access harder for each user.Fig. 5 compares different agent numbers, measuring average cost for user u1 with ǫ = 0.2.Source: §V · p. 9, Figure 5 · p. 9
  • With 30 users and 23 channels, 20 users offload tasks and 10 compute locally; the users access channels without collisions.Eight users offload using LoRa channels, while the others use WiFi channels.Source: §V · p. 9, Figure 6 · p. 9
  • For fewer than 27 users, IL-based MA-Q has the lowest average cost; with more users, its cost is slightly higher than Centralized but remains much better than Random.The distributed method reduces gateway computation burden and avoids the extra channel-estimation cost of Centralized.Source: Figure 7 · p. 9, §V · p. 10

VI. CONCLUSIONS

The paper formulates multi-user IoT edge computation offloading as resource allocation and proposes distributed IL-based MA-Q. Simulations show feasibility across network scales, collision avoidance, and lower system cost than two benchmarks.

VI. CONCLUSIONS

  • IL-based MA-Q lets end users learn computation policies independently, reducing computation burden on the centralized gateway.Simulations verify feasibility at different IoT network scales; compared with two benchmarks, the scheme avoids radio access collisions, achieves lower system cost, and improves gateway computation capacity through distributed learning.Source: §VI · p. 10
Loading 2004.02315v1…