PaperlayerView exact revision on arXiv

Source-linked AI summary

Federated Reinforcement Learning: Techniques, Applications, and Open Challenges

Jiaju Qi, Qihao Zhou, Lei Lei, Kan Zheng

arXiv:2108.11887v2cs.LGcs.AI

TL;DR

FRL addresses practical RL challenges while preserving privacy, but its relatively new literature lacks a dedicated comprehensive survey. This paper tutorials FRL, classifies it into HFRL and VFRL, surveys applications, and identifies open issues, concluding that further research is needed across the field.

  • Problem

    Existing FL surveys do not focus on FRL, while practical FRL must address privacy, communication, security, and participant-management challenges.

  • Method

    The paper provides a systematic FRL tutorial, defines HFRL and VFRL, surveys application studies, and analyzes open research issues.

  • Results

    The survey summarizes FRL schemes across edge computing, control optimization, and attack detection, alongside identified open issues for future research.

  • Takeaways & Limitations

    FRL is presented as a promising RL branch combining safer, more efficient learning with federated privacy and experience sharing.

Abstract

from arXiv · show

This paper presents a comprehensive survey of Federated Reinforcement Learning (FRL), an emerging and promising field in Reinforcement Learning (RL). Starting with a tutorial of Federated Learning (FL) and RL, we then focus on the introduction of FRL as a new method with great potential by leveraging the basic idea of FL to improve the performance of RL while preserving data-privacy. According to the distribution characteristics of the agents in the framework, FRL algorithms can be divided into two categories, i.e. Horizontal Federated Reinforcement Learning (HFRL) and Vertical Federated Reinforcement Learning (VFRL). We provide the detailed definitions of each category by formulas, investigate the evolution of FRL from a technical perspective, and highlight its advantages over previous RL algorithms. In addition, the existing works on FRL are summarized by application fields, including edge computing, communication, control optimization, and attack detection. Finally, we describe and discuss several key research directions that are crucial to solving the open problems within FRL.

I. INTRODUCTION

The paper introduces Federated Reinforcement Learning (FRL) as a privacy-preserving integration of Federated Learning (FL) and Reinforcement Learning (RL), motivated by RL's practical and sample-efficiency challenges. It surveys FRL foundations, applications, and open research issues.

  • RL background: RL agents learn through environmental interaction, selecting actions under policies to maximize cumulative reward.RL has been applied to games, robotics, and communication, among other areas.
  • FRL motivation: Large state and action spaces make exhaustive exploration impractical, while low sample efficiency can slow RL learning.Information exchange between agents is presented as a way to accelerate learning.
  • FRL motivation: FRL research addresses gaps between simulated and real environments and integrates partial observations or rewards held by different agents.Aggregation can combine information from multiple environments or agents without directly sharing raw data.
  • FRL motivation: FRL combines FL and RL so agents can address sequential decision-making tasks while preserving data privacy.The paper maps FL dimensions to FRL: samples, features, and labels correspond to environments, states, and actions.
  • Survey scope: This survey provides an FRL tutorial, summarizes applications, and identifies open issues for future research.Its stated application coverage includes edge computing, communications, control optimization, and attack detection.
  • FL foundations: FL enables parties to train a shared model without aggregating their local datasets, exchanging model information instead of raw data.The paper describes privacy protection, secure communication, generality, and performance close to an ideal centralized model as FL characteristics.
  • FL foundations: The federated model is trained from distributed datasets using local parameters, a global loss function, and weighted aggregation.FedAvg is presented as a widely used gradient-descent-based approach, with aggregation weights that may reflect participant data volume or relative impact.
  • FL foundations: FedAvg coordinates local updates through a coordinator that distributes models, receives participant information, and updates the global model.The local update uses a learning rate γ, and the coordinator can apply privacy-preserving encryption to exchanged updates.

B. Architecture of federated learning

Federated learning uses either a coordinator-mediated client-server architecture or a coordinator-free peer-to-peer architecture. The two designs differ in how participants exchange and aggregate model updates, with peer-to-peer communication requiring more messages.

  • Architectural classification: FL architecture is classified by application characteristics into client-server and peer-to-peer models.
  • Client-server model: Client-server FL consists of participants that train locally and a coordinator that distributes and aggregates model updates.The coordinator initializes the model, receives updates, combines them, and sends the resulting model back.
  • Peer-to-peer model: Peer-to-peer FL lets equal participants communicate directly without relying on a central coordinator.Participants must agree on transfer order, using modes such as cyclic or random transfer.
  • Peer-to-peer model: Peer-to-peer aggregation can improve architectural security by removing the centralized server but requires more communication resources and computation.The additional burden comes from exchanging information through more messages.

C. Categories of federated learning

Federated learning categories are determined by how participants’ feature, label, and sample spaces overlap. HFL aligns features, VFL aligns samples, and FTL addresses unaligned datasets through knowledge transfer.

  • Category framework: FL categories are defined from partitions of feature, label, and sample spaces, illustrated through HFL, VFL, and FTL examples.
  • HFL: HFL uses aligned feature and label spaces with different sample IDs to increase data volume without transmitting original data.It corresponds to horizontal partitioning, where participants hold similar features for different samples.
  • VFL: VFL uses different feature and label spaces with shared sample IDs to build a model from complementary participant features.Feature fusion can support analysis and even infer new features without releasing private data.
  • FTL: FTL applies when participants’ datasets are unaligned in both samples and features, transferring invariants from resource-rich to resource-scarce domains.It addresses distributed privacy constraints and can help when data or labels are scarce.

III. REINFORCEMENT LEARNING

Reinforcement learning trains agents through repeated agent-environment interaction, using states, actions, rewards, returns, value functions, and policies. Its objective is to learn a policy that maximizes long-term return.

  • RL model: RL agents observe states, choose actions, and receive subsequent states and rewards from the environment.The process is organized into discrete time steps and may involve episodic or continuing tasks.
  • Returns: A return summarizes rewards over time, with discounted return using γ to control how strongly future rewards influence current value.When γ = 0 the agent considers only current reward, whereas γ = 1 includes the full future value.
  • Value functions: The state-value function estimates expected return from a state under policy π, while the action-value function also conditions on the chosen action.
  • RL objective: The RL objective is long-term reward maximization rather than maximizing the reward from a single action.
  • Policies: RL produces a policy that specifies action probabilities for states and seeks an optimal policy maximizing the value function.

B. Categories of reinforcement learning

Reinforcement learning algorithms are categorized as value-based, policy-based, and actor-critic methods. Figure 7 presents these categories alongside representative algorithms.

  • RL algorithms are divided into value-based and policy-based methods, with actor-critic methods combining characteristics of both.
  • Figure 7 summarizes the categories and representative algorithms of reinforcement learning.

1) Value-based methods:

Value-based methods estimate action values and improve policies from them, whereas policy-based methods directly model policies and optimize expected cumulative rewards. These approaches have complementary strengths and weaknesses, motivating combined architectures.

  • Value-based methods:: The Bellman equation recursively relates the current action-value function to subsequent states, rewards, and actions.
  • Value-based methods:: Value-based methods iteratively calculate Qπ(s,a) and improve the strategy by selecting actions with the highest values.Q-learning and SARSA are representative value-based algorithms.
  • Value-based methods:: Q-learning is model-free and estimates action values through environment interaction, but is most suitable for simple problems with small state spaces.
  • Policy-based methods:: Policy-based methods directly fit the policy function, mapping states to actions without estimating a value function.
  • Policy-based methods:: Policy gradient optimizes policy parameters using the gradient of expected reward, and its return term is obtained only after an episode.Consequently, policy-gradient updates occur per episode rather than per time step.
  • Policy-based methods:: Policy-based methods support continuous action spaces and smoother action-probability changes, but require many trajectories and can suffer high variance and unstable convergence.Value-based methods instead face difficulties with direct action output, continuous actions, and bias in value estimates.

3) Actor-critic methods:

Actor-critic methods combine direct policy generation with value-based policy evaluation. Deep reinforcement learning extends these approaches with neural networks for large or continuous state and action spaces.

  • 3) Actor-critic methods:: Actor-critic architecture combines value-based and policy-based characteristics to address their respective weaknesses and the tension between bias and variance.
  • 3) Actor-critic methods:: The actor receives a state and outputs an action according to πθ, while the critic evaluates the state-action pair using Q.
  • C. Deep reinforcement learning: Deep reinforcement learning uses multilayer neural networks to approximate value or policy functions in large or continuous state and action spaces.
  • C. Deep reinforcement learning: DQN replaces the Q-learning table with a neural network that approximates the Q-value function.
  • C. Deep reinforcement learning: The survey taxonomy includes policy-based, actor-critic, partially observable, and multi-agent deep reinforcement learning algorithms.

A. Federated reinforcement learning background

Federated reinforcement learning applies federated collaboration to reinforcement learning under privacy protection, addressing limited exploration, non-IID environments, and partial observations. The survey distinguishes HFRL from VFRL and details HFRL’s architecture, workflow, and benefits.

  • A. Federated reinforcement learning background: FRL addresses practical RL challenges including low sample efficiency, limited exploration, simulation-to-real gaps, and incomplete observations.
  • A. Federated reinforcement learning background: FRL integrates federated learning and reinforcement learning to accelerate learning, protect agent privacy, and address Non-IID data.
  • B. Horizontal federated reinforcement learning: HFRL uses independent environments with aligned state and action spaces, whereas VFRL supports agents sharing an environment while interacting through different observations or features.
  • B. Horizontal federated reinforcement learning: In HFRL, environments, state spaces, and action spaces correspond to FL’s datasets, feature spaces, and label spaces, respectively.
  • B. Horizontal federated reinforcement learning: An HFRL server-client workflow initializes models, trains agents locally through SARS cycles, encrypts parameter uploads, aggregates models, and redistributes updates.
  • B. Horizontal federated reinforcement learning: HFRL can accelerate training through shared experience, accommodate non-identical environments with personalized updates, and permit asynchronous participation.

C. Vertical federated reinforcement learning

VFRL applies vertical federated learning to reinforcement learning when agents share an environment but observe different state spaces, including partially observable settings. It aggregates privacy-preserving information from decision-oriented and support-oriented agents to improve cooperative policy learning.

  • Definition: VFRL adapts vertical federated learning to reinforcement learning by assigning agents different feature-like state spaces within the same environment.The aggregation of local observations and action spaces forms the global state and action spaces.
  • Architecture: Agents’ local partial observations collectively represent the global environment state, while individual agents may have substantially different observations.The architecture maps VFL’s vertically partitioned features to agents’ state spaces.
  • Agent roles: VFRL distinguishes decision-oriented agents that act and receive rewards from support-oriented agents that only observe the environment.Support-oriented agents contribute local states despite taking no actions or receiving no rewards.
  • Procedure: The basic VFRL procedure initializes local models, trains them from SARS tuples, encrypts intermediate products, aggregates them federally, and returns encrypted parameters for local updates.The six-step process alternates local interaction and training with encrypted federation and model synchronization.
  • Relation to MARL: VFRL targets POMDP scenarios in which agents share an environment but have different partial observations, a setting that standard fully observed MARL algorithms does not cover.The survey notes that partial observability is important for real-world applications.
  • Relation to MARL: Unlike MARL, VFRL can include agents unable to generate operation policies, actions, or rewards, enabling broader multi-agent problem formulations.This distinction is illustrated by FedRL settings where one agent cannot build its own policies and rewards.
  • Advantages: VFRL preserves privacy during cooperation by exchanging federated information without exposing participants’ raw real-time data.The survey presents privacy protection as an advantage for multi-agent learning in a shared environment.

D. Other types of FRL

FRL also includes federated approaches that do not fit strictly into HFRL or VFRL. These methods federate reward information or model weights across agents and environments to improve policy learning.

  • General FRL: Some FRL methods implement federation between agents without being classified as HFRL or VFRL.They borrow less directly from the FL partitioning framework while still coordinating agents.
  • Federated Reward Shaping: Federated Reward Shaping shares federated information through reward shaping to improve policy quality and training speed.FRS uses a server-client architecture and can be combined with different RL algorithms.
  • Multi-task FRL: Multi-task FRL with adversaries smooths average weights to learn a unified policy jointly optimized across environments with potentially different MDPs.Unlike HFRL, the participating environments need not share state and action spaces.

V. APPLICATIONS OF FRL

FRL has been applied across edge computing, communications, control optimization, attack detection, and other tasks. This application-focused discussion surveys relevant scenarios and research status.

  • Application fields: The survey discusses FRL applications in edge computing, communications, control optimization, attack detection, and related tasks.The section is intended to characterize applicable scenarios and the current research status of FRL.

A. FRL for edge computing

FRL is applied to edge computing to support privacy-aware, distributed optimization across caching, resource allocation, and task scheduling. Reported studies use hierarchical, multi-level, and federated DRL mechanisms to improve learning efficiency, delays, hit rates, resource utilization, and adaptability.

  • Edge computing optimization: “In-Edge AI” uses collaboration among edge nodes to exchange learning parameters for mobile edge computing, caching, and communication optimization.The framework is reported to have high performance and relatively low learning overhead, while adapting to its environment.
  • Edge caching: AWFDRL optimizes federated aggregation weights for DQN-based cooperative edge caching, improving average delay, hit rate, and offload traffic.The method addresses imbalance in local model quality and improves learning efficiency; convergence is verified in simulations.
  • Edge caching: A dueling deep Q-network with HFRL aggregates local caching models in the cloud and outperforms three classical methods and two RL algorithms.The reported gains concern reducing content request delays and increasing cache hit rates in fog radio access networks.
  • Edge caching: HFDRL hierarchically groups edge devices and uses cluster heads for aggregation, improving average content access delay and hit rate on MovieLens simulations.Storage partitioning distributes content across clusters, while hierarchical categorization addresses response-time disadvantages associated with different cluster sizes.
  • Vehicular edge caching: CoCaRL applies multi-level FRL and two-level aggregation to vehicular cooperative caching, achieving high hit rate, adaptability, and fast convergence in complex environments.The method reduces communication overhead while DRL optimizes caching policies between roadside units.
  • Resource allocation: FRL studies also address edge resource allocation, including data offloading, blockchain-integrated computation decisions, and concurrent task scheduling.The supplied passages identify H-MAAC, I-UDEC, and CFRL as approaches for these optimization tasks, but do not report their outcomes here.

C. FRL for control optimization

FRL is surveyed across control optimization and attack detection, where agents share learning experience or models while preserving privacy. Applications include robotic navigation, autonomous driving, IoT control, jamming defense, traffic monitoring, and smart-home energy management.

  • Control optimization: FRL-based control optimization enables agents to share learning experience or policy parameters without exposing private data.Applications include robotic navigation, autonomous driving, IoT-device control, and smart-home energy management.
  • Attack detection: 39.9% higher average accuracy was achieved by federated jamming detection than by the distributed mechanism on CRAWDAD and ns-3 FANET datasets.The approach uses model-free Q-learning on distributed UAVs to cooperatively learn jamming-attack detection models.
  • Attack detection: Up to 22.83% higher DDoS attack-detection performance was obtained with DeepMonitor than with FlowStat.DeepMonitor uses a federated DDQN algorithm for fine-grained traffic monitoring at SDN-based IoT edge networks and mitigates flow-table overflows.
  • Other applications: FRL-based smart-home energy management outperforms other approaches in convergence speed, appliance energy consumption, and number of agents under heterogeneous environments.Local home energy-management systems upload models to a global server, which updates the global model using FedSGD.
  • Other applications: FRL supports privacy-preserving collaborative perception for autonomous vehicles by sharing information with low latency through region quadtrees.The cited approach addresses how many quadtree blocks and what resolution should be transmitted.

F. Lessons Learned

The survey finds that FRL commonly combines classical DRL with periodic parameter or gradient aggregation, while RL can also optimize FL participation. Most work targets HFRL, but heterogeneous environments, security, communication, and limited VFRL research remain open challenges.

  • Technical lessons: Most FRL studies use DQN or DDPG locally and periodically aggregate actor or critic parameters or gradients through a coordinator.Aggregation may be synchronous or asynchronous, with updated values returned to participants.
  • Technical lessons: Aggregation weights can reflect recent cumulative rewards, action error rates, or device-related indicators to address participant imbalance and performance discrepancies.These criteria are used to evaluate local models during global-model training.
  • Technical lessons: RL can optimize FL by selecting participants or candidate edges when unstable connectivity makes simultaneous participation impractical.The stated aim is to speed convergence.
  • FRL categories: Most existing research focuses on HFRL because participants commonly perform similar decision-making tasks across individual environments, whereas VFRL receives little attention.Examples include caching allocation, offloading optimization, and attack monitoring.
  • Open challenges: Future FRL implementation must address challenges inherited from FL, including security and communication issues, alongside FRL-specific problems.The survey presents these as critical open issues for future research.
  • Open challenges: Slightly different transition dynamics across agents can cause convergence issues in realistic HFRL scenarios.Proximal terms and probabilistic agent selection are proposed as possible ways to address heterogeneity and reduce convergence time or training loss.

B. Agents without rewards in VFRL

VFRL includes assistance agents that observe environments and share knowledge but lack their own actions and rewards, challenging traditional RL models. FRL research also faces communication, security, participant-management, and incentive constraints.

  • Special VFRL agents observe the environment and pass knowledge to other agents, without generating their own actions or rewards.
  • Traditional RL models handle this setting poorly because they often assign assistance agents public states, actions, or rewards lacking practical significance.
  • FRL requires exchanging parameters, gradients, or intermediate results, making communication cost significant under limited resources and large participant populations.
  • FRL inherits privacy and security vulnerabilities from FL, including reward tampering that can shift evaluative functions in decision-making tasks.
  • Join and exit mechanisms remain underdeveloped; synchronous training can be delayed indefinitely when offline or compromised participants fail to upload information.
  • FRL lacks a clear incentive mechanism and quantitative standard for evaluating agents’ contributions to collaborative learning.
Loading 2108.11887v2…