Resource Management for Blockchain-enabled Federated Learning: A Deep Reinforcement Learning Approach

I. INTRODUCTION

• BFL stores model updates and payment records in a decentralized blockchain, addressing centralized FL management and reliability concerns.Blockchain is presented as a secure, decentralized database for recording transactions and model updates.Source: §I · p. 1
• Mining can increase training latency, while mobile-device energy and CPU constraints can reduce network lifetime and training efficiency.The latency increase becomes more significant as the block generation rate decreases.Source: §I · p. 1
• The MLMO must jointly choose device data and energy usage and the blockchain block generation rate while targeting model accuracy and minimizing latency, energy, and incentive cost.These decisions are difficult because both mobile and blockchain environments are dynamic and uncertain.Source: §I · p. 1
• The paper uses Deep Q-Network-based DRL to derive MLMO decisions without prior network knowledge.Simulation results compare the proposed scheme with baseline schemes.Source: §I · pp. 1–2

A. System Description

• The system contains one MLMO, N mobile devices, and a blockchain network, with orthogonal channels connecting the MLMO to devices through a base station.Orthogonal channels are used to avoid interference among devices.Source: A. System Description · p. 2
• The MLMO sends each device a global model and payment record, after which devices train locally using their data, energy, and CPU resources.Payments indicate the device resources and blockchain mining rate associated with the training process.Source: A. System Description · p. 2
• Devices send local model updates and payment records to associated miners, which broadcast transactions for blockchain processing.The transaction contains the local model update and the payment record supplied by the MLMO.Source: A. System Description · p. 2
• Training iterations continue until the global model reaches a specified accuracy, with devices reporting resource availability before each iteration.Energy availability is explicitly communicated to the MLMO before each iteration.Source: A. System Description · p. 2

B. Modeling The Blockchain Network

• Transactions arrive independently at an average rate and are modeled as a Poisson process, while block generation is also treated as stochastic.Transactions wait in miners’ memory before inclusion in newly mined blocks.Source: B. Modeling The Blockchain Network · p. 2
• The blockchain is represented as an M/M/c queue with one mining winner serving transactions, so c = 1.Mining uses proof of work, followed by cross-verification and block-propagation delays before blockchain append.Source: B. Modeling The Blockchain Network · p. 2
• The queue-state probability is P_m = (1 − ρ)ρ^m, where utilization is ρ = λ/μ and λ and μ denote transaction arrival and block generation rates.The stochastic input and output make the number of transactions in the system non-deterministic.Source: B. Modeling The Blockchain Network · p. 2

III. PROBLEM FORMULATION

• The MLMO chooses the number of data and energy units obtained from each device and the mining process’s block generation rate.These choices form the action variables for resource management under uncertain BFL conditions.Source: §III · pp. 2–3
• The objective is to achieve the model-accuracy target while minimizing energy consumption and system latency with reasonable payment.The formulation captures multiple competing resource-management objectives.Source: §III · p. 3
• Because the BFL network is uncertain, the problem is expressed as a stochastic optimization over state space, action space, state transitions, and a reward function.This formulation supports the subsequent reinforcement-learning solution.Source: §III · p. 3

A. State Space

• The state space S contains device states and the blockchain queue state m.It is defined as S = {f, c, m}.Source: A. State Space · p. 3
• The vector f records each device’s CPU shares allocated to training, bounded by Fmax.Each fi is the number of CPU shares used by device i, with fi ∈ {0, 1, . . . , Fmax}.Source: A. State Space · p. 3
• The vector c records each device’s available training energy, bounded by Cmax.Each ci is the number of energy units available to device i, with ci ∈ {0, 1, . . . , Cmax}.Source: A. State Space · p. 3

B. Action Space

• Each training iteration requests di data units and ei energy units from device i.The associated CPU-cycle requirement depends on the requested data and the number of CPU cycles needed per data unit.Source: B. Action Space · p. 3
• A low µ increases latency, whereas a high µ lowers latency but can increase miner payment because mining consumes more computing power.The block generation rate must satisfy µ0 ≥ λ so the blockchain queue remains stable.Source: B. Action Space · p. 3
• The action space A = {d, e, µ} contains data vectors, energy vectors, and the block generation rate µ.The requested resources cannot exceed device availability, including di ≤ Dmax and ei ≤ min(ci, Emax).Source: B. Action Space · p. 3

C. State Transition

• The transition function T(st, at) represents transitions of both device states and the blockchain state.The MLMO observes the resulting state at the next iteration.Source: C. State Transition · p. 3
• A device’s next energy state equals its current energy minus requested training energy plus acquired charging energy.The charging amount ki is modeled as a Poisson-distributed quantity.Source: C. State Transition · p. 3
• The MLMO has no advance information about the state transition, which represents uncertainty in the environment.The transition therefore captures uncertain device and blockchain conditions.Source: C. State Transition · p. 3

D. Reward Function

• The MLMO’s objective is to achieve target model accuracy while minimizing energy consumption, training latency, and payment.Model accuracy increases monotonically with the total data requested from devices.Source: D. Reward Function · p. 3
• The reward accounts for total data D, energy E, latency L, and incentive cost I using scale factors αD, αE, αL, and αI.These components are normalized by their corresponding maximum values to account for different scales.Source: D. Reward Function · pp. 3–4
• Latency L combines training, transmission, and block-generation delays, while a higher µ reduces mining latency but increases miner-related cost.Block-generation latency includes cross-verification, propagation, and mining components.Source: D. Reward Function · p. 3
• The proposed DQN approximates the optimal long-term policy because tabular Q-learning becomes inefficient for large state and action spaces.The policy selects an action from A given the observed state S.Source: D. Reward Function · p. 4

IV. DEEP REINFORCEMENT LEARNING ALGORITHM

• The DQN uses a Q-network to map MLMO states to Q-values for all possible actions, enabling approximate optimal action selection.The network takes the MLMO state as input and outputs Q(s, a; θ) for each action.Source: §IV · p. 4
• At each iteration, the MLMO selects an action with an ϵ-greedy policy, observes reward and next state, and stores the transition in replay memory.Mini-batches sampled from replay memory are used to update the network weights.Source: §IV · p. 4
• The Q-network weights are updated using the temporal-difference error between a target value and the current Q-value.The update uses the learning rate, gradient, and target value defined for the online network.Source: §IV · p. 4
• The target network weights are periodically copied from the online network, and training proceeds as an episodic task until the required number of data units is reached.Within each episode, the MLMO averages local model updates from the mobile devices.Source: §IV · p. 4

V. PERFORMANCE EVALUATION

• DQN’s cumulative reward is 38%, 100%, and 237% higher than Q-learning, Greedy, and Random, respectively.The Greedy scheme obtains a negative reward because it always requests the maximum data amount, increasing energy consumption and latency.Source: Figure 3 · p. 4, §V · p. 4
• DQN reduces energy consumption by up to 72% compared with Greedy by incorporating energy cost into its reward function.Greedy requests the maximum number of data units, causing devices to consume more training energy.Source: §V · p. 4, Figure 4 · p. 5
• DQN reduces system latency by up to 12% compared with Greedy by adapting data and energy decisions to device energy states.For low-energy devices, the MLMO requests fewer data units, whereas Greedy does not consider latency cost.Source: Figure 5 · p. 5, §V · p. 5
• DQN incurs lower total payment than Greedy but slightly higher payment than the other baselines while balancing payment, energy consumption, and latency.The comparison is presented in the total-payment evaluation.Source: Figure 6 · p. 5, §V · p. 5
• As device data quality increases, DQN allocates more data units to higher-quality devices, helping the MLMO reach the accuracy target faster.With quality ratios [1 : 1 : 1], [2 : 2 : 1], [3 : 2 : 1], and [4 : 2 : 1], allocation increasingly favors devices 1 and 2 as their quality rises.Source: Figure 7 · p. 5, §V · p. 5

• The proposed DRL scheme outperforms baseline schemes in energy consumption and training latency with reasonable cost.Source: §VI · p. 5
• The BFL system is modeled with the blockchain network represented as an M/M/1 queue.Source: §VI · p. 5
• The authors formulate a stochastic optimization problem for MLMO resource management and develop DRL to solve it.Source: §VI · p. 5