Source-linked AI summary
When Deep Reinforcement Learning Meets Federated Learning: Intelligent Multi-Timescale Resource Management for Multi-access Edge Computing in 5G Ultra Dense Network
Shuai Yu, Xu Chen, Zhi Zhou, Xiaowen Gong, Di Wu
TL;DR
UDEC lacks efficient joint management of heterogeneous resources, low-overhead decisions, and privacy protection. The paper proposes I-UDEC with two-timescale DRL and federated training, reporting up to 31.87% lower task execution time.
Problem
UDEC solutions lack efficient use of heterogeneous resources, low-overhead offloading and allocation, and privacy protection.
Method
I-UDEC jointly optimizes application partitioning, resource allocation, and service caching through fast- and slow-timescale DRL with federated training.
Results
31.87% average task execution-time reduction is reported against LES.
Takeaways & Limitations
The framework combines AI, blockchain, and federated learning for resource management and privacy-oriented distributed training in UDEC.
Takeaways & Limitations
Traditional federated learning struggles to aggregate differently shaped local models when service-popularity states vary across SCceNBs.
Abstract
from arXiv · showhide
Ultra-dense edge computing (UDEC) has great potential, especially in the 5G era, but it still faces challenges in its current solutions, such as the lack of: i) efficient utilization of multiple 5G resources (e.g., computation, communication, storage and service resources); ii) low overhead offloading decision making and resource allocation strategies; and iii) privacy and security protection schemes. Thus, we first propose an intelligent ultra-dense edge computing (I-UDEC) framework, which integrates blockchain and Artificial Intelligence (AI) into 5G ultra-dense edge computing networks. First, we show the architecture of the framework. Then, in order to achieve real-time and low overhead computation offloading decisions and resource allocation strategies, we design a novel two-timescale deep reinforcement learning (\textit{2Ts-DRL}) approach, consisting of a fast-timescale and a slow-timescale learning process, respectively. The primary objective is to minimize the total offloading delay and network resource usage by jointly optimizing computation offloading, resource allocation and service caching placement. We also leverage federated learning (FL) to train the \textit{2Ts-DRL} model in a distributed manner, aiming to protect the edge devices' data privacy. Simulation results corroborate the effectiveness of both the \textit{2Ts-DRL} and FL in the I-UDEC framework and prove that our proposed algorithm can reduce task execution time up to 31.87%.
1 INTRODUCTION
UDEC must jointly address heterogeneous, time-varying resource use, low-cost scheduling, and privacy protection. The paper proposes I-UDEC with two-timescale DRL and federated training to optimize these concerns.
- UDEC integrates MEC with ultra-dense networks to reduce transmission delay and support real-time computing.
- The introduction identifies full resource utilization, dynamic low-cost scheduling, and privacy-preserving services as key UDEC challenges.
- I-UDEC jointly optimizes application partitioning, resource allocation, and service caching placement in ultra-dense edge environments.
- The framework supports hybrid offloading to remote cloud servers, nearby edge servers, or nearby mobile devices through D2D offloading.
- 2Ts-DRL separates delay-sensitive application partitioning and resource allocation into a fast tier and delay-insensitive caching into a slow tier.
- Federated learning trains the DRL agent distributively to protect edge users’ sensitive service-request information.
2 RELATED WORK
Prior work addresses UDEC offloading, edge intelligence, and blockchain separately or incompletely. The paper positions 2Ts-DRL with federated training as a joint approach to offloading, resource allocation, and service caching.
- MEC edge servers provide computing, storage, connectivity, and services, but their limited resources motivate dense deployment.
- Existing UDEC studies include greedy and game-theoretical offloading, while some omit service caching or restrict the participating devices.
- The paper’s notation covers edge devices, SCceNB servers, storage, CPU frequencies, task models, queues, states, offloading, and subcarrier allocation.
- Edge intelligence moves AI computation toward the network edge for real-time and reliable intelligent services.
- Blockchain has been studied for secure data transmission and scalable access in edge and fog computing networks.
- The proposed 2Ts-DRL uses fast and slow learning processes, while federated learning distributes model training to protect privacy.
COMPUTING
The paper presents I-UDEC as an artificial-intelligence- and blockchain-enhanced ultra-dense edge computing framework, then introduces its architecture and models.
- I-UDEC enhances UDEC through computation offloading, resource allocation, and service caching placement.
3.1 The Architecture of I-UDEC
The I-UDEC architecture enhances offloading with deep learning and organizes edge-network information through a controller. Its user, data, and control planes support on-demand network configuration.
- I-UDEC enhances computation offloading, resource allocation, and service caching placement as its three main UDEC functions.
- Deep Q-learning supports real-time offloading decisions, while the framework also addresses security issues during offloading.
- The framework comprises user, data, and control planes, with edge devices supplying offloading requirements.
- The controller maintains edge-device, edge-server, service, and application information to optimize network configurations on demand.
3.2 Network Model
The network model combines small-cell MEC, EDs, cellular and D2D links, and service caching to support heterogeneous edge computing. It models task partitioning into dependent subtasks that may execute locally or be offloaded to cloud, edge, or neighboring devices.
- Network architecture: The I-UDEC network contains a remote cloud, macro base station, SCceNBs, and EDs, with SCceNBs providing more computation and storage resources.Each ED has X_m CPU cores, each SCceNB has Y CPU cores, and SCceNB storage capacity is C_s.
- Network architecture: Cellular and D2D links use OFDMA, separating EDs in frequency so devices under one SCceNB do not interfere.The model uses a subcarrier-based fixed channel assignment scheme.
- Channel model: During each scheduling period, ED locations are fixed and wireless channels are stable, although both may change across periods because of mobility.The assumptions support period-based modeling of time-varying network conditions.
- Channel model: Cellular uplink rates are modeled over AWGN channels using bandwidth, transmit power, distance, noise, fading, subcarrier allocation, BER, and path-loss parameters.Estimated channel states are sent to the controller, and the uplink rate index feeds an uplink rate weight matrix.
- D2D and energy model: EDs can communicate with nearby EDs through D2D links when their distance is below a predefined threshold, with D2D rates determined by link and transmission parameters.The D2D rate index feeds a D2D rate weight matrix; the model focuses on ED uplink energy consumption and ignores downlink and SCceNB computation energy.
- Service caching: Services require cached libraries and databases, with limited SCceNB and ED storage represented by centralized SCceNB and distributed ED placement matrices.The remote cloud stores all services, while binary placement indices indicate whether a service is cached.
- Task model: Tasks are represented as dependent subtasks, each characterized by workload, input and output data sizes, required service, and CPU-cycle demand.The model uses sequential DNN partitioning, where resource-intensive hidden-layer subtasks can be dynamically offloaded to edge servers.
3.3 Computation Offloading in I-UDEC
Computation offloading in I-UDEC proceeds through service caching placement, offloading decisions, and resource allocation. The hybrid model allows subtasks to execute locally or be sent to nearby EDs, SCceNBs, or the remote cloud.
- Offloading process: The offloading process consists of service caching placement, computation offloading decisions, and resource allocation.Caching placement is intended to shorten the service distance between EDs and serving SCceNBs under limited edge storage.
- Offloading options: Each ED may execute subtasks locally or offload them to another ED through D2D, a serving SCceNB through wireless uplink, or the remote cloud through uplink and fronthaul.This is the framework's hybrid computation offloading scenario.
- Decision process: SCceNBs maintain virtual task queues containing pending subtasks and output offloading decisions for associated EDs in discrete decision periods.Each SCceNB serves its closest EDs and represents queue state with a task-queue matrix.
- Decision process: An action value of -1 sends a subtask to the remote cloud, 0 sends it to the serving SCceNB, and j sends it to ED j; m = j denotes local execution.The action vector is defined for each task-queue element in the current decision period.
- Execution time: The model computes total execution time from the execution times of subtasks currently held in each SCceNB task queue.The queue-level execution time aggregates the per-element execution times.
- Service and latency constraints: Offloading is constrained by service placement: required services must be cached at the selected ED or SCceNB, while cloud offloading includes SCceNB-to-cloud latency.The end-to-end cloud latency is represented by t_e^n.
4 PROBLEM FORMULATION AND ALGORITHM DE-
The I-UDEC optimization jointly minimizes long-term task execution time and system resource usage through a two-tier DQN operating at different timescales. Fast decisions handle application partitioning and resource allocation, while slower decisions update service caching placement.
- Problem formulation: The optimization jointly minimizes long-term task execution time and computation, communication, and storage resource usage in I-UDEC.
- Problem formulation: DRL addresses large-scale I-UDEC decision making by using deep neural networks to improve reinforcement-learning speed and performance.DQL replaces the Q-table with a DQN, enabling learning in larger state and action spaces.
- 2Ts-DRL framework: The two-tier 2Ts-DRL agent jointly optimizes application partitioning, resource allocation, and service caching placement at different timescales.The bottom tier handles delay-sensitive application partitioning and resource allocation in a fast timescale; the top tier handles delay-insensitive caching decisions.
- State space: The MDP state includes task-queue information, service-caching states, and available computation and communication resources.State components include virtual task queues, caching placement, CPU availability, and resource-related system information.
- Action space: The action space comprises application partitioning, subcarrier allocation, and service caching placement policies.Application partitioning and subcarrier allocation are delay-intensive and immediate, whereas caching updates occur every αt seconds.
- Reward function: The reward combines execution time with computation, communication, and storage usage to guide long-term cost minimization.Caching rewards account for execution time and storage usage, while partitioning and allocation rewards account for execution time, subcarriers, and occupied CPUs.
- Reward function: The reward formulation constrains offloaded execution time and resource consumption relative to local processing, storage capacity, and available communication and computation resources.DQN maximizes long-term reward by using negatively written cumulative costs in the two timescales.
- Reward function: The decision agent outputs near-optimal offloading, resource-allocation, and caching strategies, with caching updated less frequently than computation offloading.
5 FEDERATED LEARNING-BASED MODEL TRAIN-
The paper trains the service-caching placement agent distributively with federated learning, using local service-request data and centralized model aggregation. The procedure combines local updates with federated averaging and asynchronous DRL training.
- Federated model training: The FL-based method distributively trains the service-caching placement agent while using edge devices’ local service-request information.SCceNBs receive a global model, train local models on current requests, and submit updated parameters.
- Federated model training: Each training round downloads global weights, performs parallel local DRL updates, uploads the trained weights, and broadcasts federated-averaged weights.The algorithm repeats this process across decision periods.
- Federated model training: The global model update uses a learning-rate-scaled aggregate update, where ψ represents the learning rate.
- Federated model training: The controller weights model aggregation by each SCceNB’s local-dataset quantity and generates the next service-caching strategy after repeated iterations.The resulting strategy is broadcast for the next caching placement decision period.
- Federated model training: A3C trains the agents asynchronously through multiple local and global actor-critic networks.Agents update local actor and critic parameters while interacting with different environment copies and policies.
- Federated model training: The distributed training design is described as reducing memory usage, avoiding correlated experiences, and being faster and more robust than traditional DRL.It avoids the replay memory used by standard DRL and combines experiences from agents encountering different states and transitions.
6 PERFORMANCE EVALUATION
Simulations evaluate 2Ts-DRL against benchmark execution, allocation, and caching policies in a UDEC environment. The proposed method reduces execution time and energy consumption while achieving cache-hit performance close to centralized training.
- 2Ts-DRL converges with tier 1 faster than tier 2 because tier 1 has smaller state and action spaces.The total utility of both tiers is high initially and decreases as episodes increase.
- 31.87%, 14.96%, 27.16% and 22.61% average task-execution-time reductions are achieved versus LES, EES-FRAS-PSCPP, RES-FRAS-PSCPP and CES-FRAS, respectively.
- Average task execution time increases with ED density because devices share limited computation and communication resources; for 2Ts-DRL it rises sharply when M > 8.Each SCceNB serves up to 8 EDs in the simulation, so some devices use local execution beyond that density.
- Federated training produces cache-hit performance near centralized training while protecting edge-device service-request privacy.Centralized training remains fixed through decision period 1000, after which its cache-hit probability grows with episodes; federated training provides the reported privacy protection.
- DRL-based caching policies outperform PSCPP in cache-hit probability as popularity skew δ increases, while all policies are similar at δ = 0.At δ = 0, service popularity is uniform.
7 EXTENSION AND FUTURE WORKS
The paper identifies blockchain and personalized federated learning as extensions for unresolved security, resource-management, and heterogeneous caching challenges. Personalized federated learning is proposed for areas with differing service-popularity distributions and offloading requirements.
- Blockchain integration is proposed to address I-UDEC security and decentralized resource-management challenges.Suggested functions include reducing key-management costs, easing distributed SCceNB maintenance, and monitoring the control plane for malicious behavior.
- Personalized federated learning is proposed because SCceNBs in different areas may have different service-caching states and popularity distributions.A global model would be trained first, followed by SCceNB-specific personalization using local information.
8 CONCLUSIONS
The paper studies joint computation offloading, resource allocation, and service caching placement in ultra-dense edge computing networks. It proposes a two-timescale deep reinforcement learning approach with federated training to optimize these decisions while protecting sensitive service-request information.
- The optimization jointly addresses computation offloading, resource allocation, and service caching placement in ultra-dense edge computing networks.
- 2Ts-DRL separates delay-sensitive decisions into a fast-timescale bottom tier and delay-insensitive decisions into a slow-timescale top tier.
- Federated learning trains the 2Ts-DRL model in a distributed manner to protect edge users’ sensitive service-request information.