PaperlayerView exact revision on arXiv

Source-linked AI summary

Federated Reinforcement Learning: Techniques, Applications, and Open Challenges

Jiaju Qi, Qihao Zhou, Lei Lei, Kan Zheng

arXiv:2108.11887v2cs.LGcs.AI

TL;DR

FRL addresses practical RL challenges while preserving privacy, but its relatively new literature lacks a dedicated comprehensive survey. This paper tutorials FRL, classifies it into HFRL and VFRL, surveys applications, and identifies open issues, concluding that further research is needed across the field.

Abstract

from arXiv · show

This paper presents a comprehensive survey of Federated Reinforcement Learning (FRL), an emerging and promising field in Reinforcement Learning (RL). Starting with a tutorial of Federated Learning (FL) and RL, we then focus on the introduction of FRL as a new method with great potential by leveraging the basic idea of FL to improve the performance of RL while preserving data-privacy. According to the distribution characteristics of the agents in the framework, FRL algorithms can be divided into two categories, i.e. Horizontal Federated Reinforcement Learning (HFRL) and Vertical Federated Reinforcement Learning (VFRL). We provide the detailed definitions of each category by formulas, investigate the evolution of FRL from a technical perspective, and highlight its advantages over previous RL algorithms. In addition, the existing works on FRL are summarized by application fields, including edge computing, communication, control optimization, and attack detection. Finally, we describe and discuss several key research directions that are crucial to solving the open problems within FRL.

I. INTRODUCTION

The paper introduces Federated Reinforcement Learning (FRL) as a privacy-preserving integration of Federated Learning (FL) and Reinforcement Learning (RL), motivated by RL's practical and sample-efficiency challenges. It surveys FRL foundations, applications, and open research issues.

RL background

  • RL agents learn through environmental interaction, selecting actions under policies to maximize cumulative reward.RL has been applied to games, robotics, and communication, among other areas.Source: §I · p. 1

FRL motivation

  • Large state and action spaces make exhaustive exploration impractical, while low sample efficiency can slow RL learning.Information exchange between agents is presented as a way to accelerate learning.Source: §I · p. 1
  • FRL research addresses gaps between simulated and real environments and integrates partial observations or rewards held by different agents.Aggregation can combine information from multiple environments or agents without directly sharing raw data.Source: §I · p. 2
  • FRL combines FL and RL so agents can address sequential decision-making tasks while preserving data privacy.The paper maps FL dimensions to FRL: samples, features, and labels correspond to environments, states, and actions.Source: §I · p. 2

Survey scope

  • This survey provides an FRL tutorial, summarizes applications, and identifies open issues for future research.Its stated application coverage includes edge computing, communications, control optimization, and attack detection.Source: §I · p. 2

FL foundations

  • FL enables parties to train a shared model without aggregating their local datasets, exchanging model information instead of raw data.The paper describes privacy protection, secure communication, generality, and performance close to an ideal centralized model as FL characteristics.Source: A. Federated learning definition and… · pp. 2–3
  • The federated model is trained from distributed datasets using local parameters, a global loss function, and weighted aggregation.FedAvg is presented as a widely used gradient-descent-based approach, with aggregation weights that may reflect participant data volume or relative impact.Source: A. Federated learning definition and… · pp. 3–4
  • FedAvg coordinates local updates through a coordinator that distributes models, receives participant information, and updates the global model.The local update uses a learning rate γ, and the coordinator can apply privacy-preserving encryption to exchanged updates.Source: A. Federated learning definition and… · p. 4

B. Architecture of federated learning

Federated learning uses either a coordinator-mediated client-server architecture or a coordinator-free peer-to-peer architecture. The two designs differ in how participants exchange and aggregate model updates, with peer-to-peer communication requiring more messages.

Architectural classification

Client-server model

  • Client-server FL consists of participants that train locally and a coordinator that distributes and aggregates model updates.The coordinator initializes the model, receives updates, combines them, and sends the resulting model back.Source: B. Architecture of federated learning · pp. 4–5

Peer-to-peer model

  • Peer-to-peer FL lets equal participants communicate directly without relying on a central coordinator.Participants must agree on transfer order, using modes such as cyclic or random transfer.Source: B. Architecture of federated learning · p. 5, Figure 2 · p. 6
  • Peer-to-peer aggregation can improve architectural security by removing the centralized server but requires more communication resources and computation.The additional burden comes from exchanging information through more messages.Source: B. Architecture of federated learning · p. 5

C. Categories of federated learning

Federated learning categories are determined by how participants’ feature, label, and sample spaces overlap. HFL aligns features, VFL aligns samples, and FTL addresses unaligned datasets through knowledge transfer.

Category framework

HFL

  • HFL uses aligned feature and label spaces with different sample IDs to increase data volume without transmitting original data.It corresponds to horizontal partitioning, where participants hold similar features for different samples.Source: C. Categories of federated learning · p. 6

VFL

  • VFL uses different feature and label spaces with shared sample IDs to build a model from complementary participant features.Feature fusion can support analysis and even infer new features without releasing private data.Source: C. Categories of federated learning · p. 7

FTL

  • FTL applies when participants’ datasets are unaligned in both samples and features, transferring invariants from resource-rich to resource-scarce domains.It addresses distributed privacy constraints and can help when data or labels are scarce.Source: C. Categories of federated learning · pp. 7–8

III. REINFORCEMENT LEARNING

Reinforcement learning trains agents through repeated agent-environment interaction, using states, actions, rewards, returns, value functions, and policies. Its objective is to learn a policy that maximizes long-term return.

RL model

  • RL agents observe states, choose actions, and receive subsequent states and rewards from the environment.The process is organized into discrete time steps and may involve episodic or continuing tasks.Source: A. Reinforcement learning definition… · p. 9

Returns

  • A return summarizes rewards over time, with discounted return using γ to control how strongly future rewards influence current value.When γ = 0 the agent considers only current reward, whereas γ = 1 includes the full future value.Source: A. Reinforcement learning definition… · p. 10

Value functions

RL objective

Policies

B. Categories of reinforcement learning

Reinforcement learning algorithms are categorized as value-based, policy-based, and actor-critic methods. Figure 7 presents these categories alongside representative algorithms.

1) Value-based methods:

Value-based methods estimate action values and improve policies from them, whereas policy-based methods directly model policies and optimize expected cumulative rewards. These approaches have complementary strengths and weaknesses, motivating combined architectures.

Value-based methods:

  • The Bellman equation recursively relates the current action-value function to subsequent states, rewards, and actions.Source: 1) Value-based methods · p. 11
  • Value-based methods iteratively calculate Qπ(s,a) and improve the strategy by selecting actions with the highest values.Q-learning and SARSA are representative value-based algorithms.Source: 1) Value-based methods · p. 11
  • Q-learning is model-free and estimates action values through environment interaction, but is most suitable for simple problems with small state spaces.Source: 1) Value-based methods · p. 11

Policy-based methods:

  • Policy-based methods directly fit the policy function, mapping states to actions without estimating a value function.Source: 2) Policy-based methods · pp. 11–12
  • Policy gradient optimizes policy parameters using the gradient of expected reward, and its return term is obtained only after an episode.Consequently, policy-gradient updates occur per episode rather than per time step.Source: 2) Policy-based methods · p. 12
  • Policy-based methods support continuous action spaces and smoother action-probability changes, but require many trajectories and can suffer high variance and unstable convergence.Value-based methods instead face difficulties with direct action output, continuous actions, and bias in value estimates.Source: 2) Policy-based methods · p. 12

3) Actor-critic methods:

Actor-critic methods combine direct policy generation with value-based policy evaluation. Deep reinforcement learning extends these approaches with neural networks for large or continuous state and action spaces.

3) Actor-critic methods:

  • Actor-critic architecture combines value-based and policy-based characteristics to address their respective weaknesses and the tension between bias and variance.Source: 3) Actor-critic methods · p. 12
  • The actor receives a state and outputs an action according to πθ, while the critic evaluates the state-action pair using Q.Source: 3) Actor-critic methods · p. 13

C. Deep reinforcement learning

A. Federated reinforcement learning background

Federated reinforcement learning applies federated collaboration to reinforcement learning under privacy protection, addressing limited exploration, non-IID environments, and partial observations. The survey distinguishes HFRL from VFRL and details HFRL’s architecture, workflow, and benefits.

A. Federated reinforcement learning background

B. Horizontal federated reinforcement learning

C. Vertical federated reinforcement learning

VFRL applies vertical federated learning to reinforcement learning when agents share an environment but observe different state spaces, including partially observable settings. It aggregates privacy-preserving information from decision-oriented and support-oriented agents to improve cooperative policy learning.

Definition

  • VFRL adapts vertical federated learning to reinforcement learning by assigning agents different feature-like state spaces within the same environment.The aggregation of local observations and action spaces forms the global state and action spaces.Source: C. Vertical federated reinforcement… · p. 19

Architecture

  • Agents’ local partial observations collectively represent the global environment state, while individual agents may have substantially different observations.The architecture maps VFL’s vertically partitioned features to agents’ state spaces.Source: C. Vertical federated reinforcement… · p. 19

Agent roles

  • VFRL distinguishes decision-oriented agents that act and receive rewards from support-oriented agents that only observe the environment.Support-oriented agents contribute local states despite taking no actions or receiving no rewards.Source: C. Vertical federated reinforcement… · pp. 19–20

Procedure

  • The basic VFRL procedure initializes local models, trains them from SARS tuples, encrypts intermediate products, aggregates them federally, and returns encrypted parameters for local updates.The six-step process alternates local interaction and training with encrypted federation and model synchronization.Source: C. Vertical federated reinforcement… · p. 20

Relation to MARL

  • VFRL targets POMDP scenarios in which agents share an environment but have different partial observations, a setting that standard fully observed MARL algorithms does not cover.The survey notes that partial observability is important for real-world applications.Source: C. Vertical federated reinforcement… · pp. 19, 21
  • Unlike MARL, VFRL can include agents unable to generate operation policies, actions, or rewards, enabling broader multi-agent problem formulations.This distinction is illustrated by FedRL settings where one agent cannot build its own policies and rewards.Source: C. Vertical federated reinforcement… · pp. 21, 23

Advantages

  • VFRL preserves privacy during cooperation by exchanging federated information without exposing participants’ raw real-time data.The survey presents privacy protection as an advantage for multi-agent learning in a shared environment.Source: C. Vertical federated reinforcement… · p. 23

D. Other types of FRL

FRL also includes federated approaches that do not fit strictly into HFRL or VFRL. These methods federate reward information or model weights across agents and environments to improve policy learning.

General FRL

  • Some FRL methods implement federation between agents without being classified as HFRL or VFRL.They borrow less directly from the FL partitioning framework while still coordinating agents.Source: D. Other types of FRL · p. 23

Federated Reward Shaping

  • Federated Reward Shaping shares federated information through reward shaping to improve policy quality and training speed.FRS uses a server-client architecture and can be combined with different RL algorithms.Source: D. Other types of FRL · p. 23

Multi-task FRL

  • Multi-task FRL with adversaries smooths average weights to learn a unified policy jointly optimized across environments with potentially different MDPs.Unlike HFRL, the participating environments need not share state and action spaces.Source: D. Other types of FRL · p. 23

V. APPLICATIONS OF FRL

FRL has been applied across edge computing, communications, control optimization, attack detection, and other tasks. This application-focused discussion surveys relevant scenarios and research status.

Application fields

  • The survey discusses FRL applications in edge computing, communications, control optimization, attack detection, and related tasks.The section is intended to characterize applicable scenarios and the current research status of FRL.Source: §V · p. 24

A. FRL for edge computing

FRL is applied to edge computing to support privacy-aware, distributed optimization across caching, resource allocation, and task scheduling. Reported studies use hierarchical, multi-level, and federated DRL mechanisms to improve learning efficiency, delays, hit rates, resource utilization, and adaptability.

Edge computing optimization

  • “In-Edge AI” uses collaboration among edge nodes to exchange learning parameters for mobile edge computing, caching, and communication optimization.The framework is reported to have high performance and relatively low learning overhead, while adapting to its environment.Source: A. FRL for edge computing · p. 24

Edge caching

  • AWFDRL optimizes federated aggregation weights for DQN-based cooperative edge caching, improving average delay, hit rate, and offload traffic.The method addresses imbalance in local model quality and improves learning efficiency; convergence is verified in simulations.Source: A. FRL for edge computing · p. 24
  • A dueling deep Q-network with HFRL aggregates local caching models in the cloud and outperforms three classical methods and two RL algorithms.The reported gains concern reducing content request delays and increasing cache hit rates in fog radio access networks.Source: A. FRL for edge computing · p. 24
  • HFDRL hierarchically groups edge devices and uses cluster heads for aggregation, improving average content access delay and hit rate on MovieLens simulations.Storage partitioning distributes content across clusters, while hierarchical categorization addresses response-time disadvantages associated with different cluster sizes.Source: A. FRL for edge computing · p. 25

Vehicular edge caching

  • CoCaRL applies multi-level FRL and two-level aggregation to vehicular cooperative caching, achieving high hit rate, adaptability, and fast convergence in complex environments.The method reduces communication overhead while DRL optimizes caching policies between roadside units.Source: A. FRL for edge computing · p. 25

Resource allocation

  • FRL studies also address edge resource allocation, including data offloading, blockchain-integrated computation decisions, and concurrent task scheduling.The supplied passages identify H-MAAC, I-UDEC, and CFRL as approaches for these optimization tasks, but do not report their outcomes here.Source: A. FRL for edge computing · p. 25

C. FRL for control optimization

FRL is surveyed across control optimization and attack detection, where agents share learning experience or models while preserving privacy. Applications include robotic navigation, autonomous driving, IoT control, jamming defense, traffic monitoring, and smart-home energy management.

Control optimization

Attack detection

  • 39.9% higher average accuracy was achieved by federated jamming detection than by the distributed mechanism on CRAWDAD and ns-3 FANET datasets.The approach uses model-free Q-learning on distributed UAVs to cooperatively learn jamming-attack detection models.Source: D. FRL for attack detection · p. 28
  • Up to 22.83% higher DDoS attack-detection performance was obtained with DeepMonitor than with FlowStat.DeepMonitor uses a federated DDQN algorithm for fine-grained traffic monitoring at SDN-based IoT edge networks and mitigates flow-table overflows.Source: D. FRL for attack detection · p. 28

Other applications

  • FRL-based smart-home energy management outperforms other approaches in convergence speed, appliance energy consumption, and number of agents under heterogeneous environments.Local home energy-management systems upload models to a global server, which updates the global model using FedSGD.Source: E. FRL for other applications · p. 29
  • FRL supports privacy-preserving collaborative perception for autonomous vehicles by sharing information with low latency through region quadtrees.The cited approach addresses how many quadtree blocks and what resolution should be transmitted.Source: E. FRL for other applications · p. 29

F. Lessons Learned

The survey finds that FRL commonly combines classical DRL with periodic parameter or gradient aggregation, while RL can also optimize FL participation. Most work targets HFRL, but heterogeneous environments, security, communication, and limited VFRL research remain open challenges.

Technical lessons

  • Most FRL studies use DQN or DDPG locally and periodically aggregate actor or critic parameters or gradients through a coordinator.Aggregation may be synchronous or asynchronous, with updated values returned to participants.Source: 1) Lessons learned from the… · p. 29
  • Aggregation weights can reflect recent cumulative rewards, action error rates, or device-related indicators to address participant imbalance and performance discrepancies.These criteria are used to evaluate local models during global-model training.Source: 1) Lessons learned from the… · p. 30
  • RL can optimize FL by selecting participants or candidate edges when unstable connectivity makes simultaneous participation impractical.The stated aim is to speed convergence.Source: 2) Lessons learned from the… · p. 30

FRL categories

  • Most existing research focuses on HFRL because participants commonly perform similar decision-making tasks across individual environments, whereas VFRL receives little attention.Examples include caching allocation, offloading optimization, and attack monitoring.Source: 3) Lessons learned from categories of… · p. 31

Open challenges

  • Future FRL implementation must address challenges inherited from FL, including security and communication issues, alongside FRL-specific problems.The survey presents these as critical open issues for future research.Source: §VI · p. 31
  • Slightly different transition dynamics across agents can cause convergence issues in realistic HFRL scenarios.Proximal terms and probabilistic agent selection are proposed as possible ways to address heterogeneity and reduce convergence time or training loss.Source: A. Learning convergence in HFRL · pp. 31–32

B. Agents without rewards in VFRL

VFRL includes assistance agents that observe environments and share knowledge but lack their own actions and rewards, challenging traditional RL models. FRL research also faces communication, security, participant-management, and incentive constraints.

  • Special VFRL agents observe the environment and pass knowledge to other agents, without generating their own actions or rewards.Source: B. Agents without rewards in VFRL · p. 32
  • Traditional RL models handle this setting poorly because they often assign assistance agents public states, actions, or rewards lacking practical significance.Source: B. Agents without rewards in VFRL · p. 32
  • FRL requires exchanging parameters, gradients, or intermediate results, making communication cost significant under limited resources and large participant populations.Source: C. Communications · p. 32
  • FRL inherits privacy and security vulnerabilities from FL, including reward tampering that can shift evaluative functions in decision-making tasks.Source: D. Privacy and Security · pp. 32–33
  • Join and exit mechanisms remain underdeveloped; synchronous training can be delayed indefinitely when offline or compromised participants fail to upload information.Source: E. Join and exit mechanisms design · p. 33
  • FRL lacks a clear incentive mechanism and quantitative standard for evaluating agents’ contributions to collaborative learning.Source: F. Incentive mechanisms · p. 34

Related papers

Loading 2108.11887v2…