PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-Task Federated Reinforcement Learning with Adversaries

Aqeel Anwar, Arijit Raychowdhury

arXiv:2103.06473v1cs.LGcs.AI

TL;DR

The paper addresses adversarial manipulation in multi-task federated reinforcement learning, where agents in different environments jointly learn a unified policy. It proposes the adaptive AdAMInG attack and the communication-adaptive ComA-FedRL defense. Across GridWorld and AutoNav experiments, AdAMInG generally outperforms other attacks, while ComA-FedRL recovers near-optimal policies under attack.

  • Problem

    MT-FedRL can lose near-optimal performance for individual environments when adversarial agents make the agents’ goals non-positively correlated.

  • Method

    The paper proposes AdAMInG, which accounts for federated aggregation, and ComA-FedRL, which adapts communication frequency based on adversarial probability.

  • Results

    Across GridWorld and AutoNav, AdAMInG outperforms other attack methods almost every time, while ComA-FedRL recovers near-optimal policies under adversarial attack.

  • Takeaways & Limitations

    The results support adaptive attacks and communication-adaptive defense as effective approaches for adversarial MT-FedRL settings.

Abstract

from arXiv · show

Reinforcement learning algorithms, just like any other Machine learning algorithm pose a serious threat from adversaries. The adversaries can manipulate the learning algorithm resulting in non-optimal policies. In this paper, we analyze the Multi-task Federated Reinforcement Learning algorithms, where multiple collaborative agents in various environments are trying to maximize the sum of discounted return, in the presence of adversarial agents. We argue that the common attack methods are not guaranteed to carry out a successful attack on Multi-task Federated Reinforcement Learning and propose an adaptive attack method with better attack performance. Furthermore, we modify the conventional federated reinforcement learning algorithm to address the issue of adversaries that works equally well with and without the adversaries. Experimentation on different small to mid-size reinforcement learning problems show that the proposed attack method outperforms other general attack methods and the proposed modification to federated reinforcement learning algorithm was able to achieve near-optimal policies in the presence of adversarial agents.

I. INTRODUCTION

The paper studies federated reinforcement learning across multiple environments, where agents share policy parameters rather than local data to learn a unified policy. It focuses on adversarial manipulation of MT-FedRL and introduces attack and defense methods for preserving policy quality.

  • Adversarial threat: Adversaries can poison data or models during training or inference, degrading learned policies; this paper specifically studies model-poisoning attacks against MT-FedRL.The introduction distinguishes data-poisoning from directly manipulating the learned model or policy.
  • Contributions: The paper studies MT-FedRL with model-poisoning adversaries on GridWorld and drone autonomous-navigation problems.These experiments cover medium and large problem settings, including tabular and neural-network-based reinforcement learning.
  • Contributions: The proposed AdAMInG attack accounts for federated smoothing aggregation and minimizes information gain to make attacks effective with a small scaling factor.The attack is designed to force the server to forget information learned from non-adversarial agents.
  • Contributions: The proposed ComA-FedRL defense modifies communication frequency according to adversarial probability while retaining the conventional federated-learning objective.The paper presents it as a modification intended to work with and without adversaries.
  • Federated reinforcement learning: MT-FedRL learns a unified policy across multiple environments while keeping agents’ local training data private and sharing policy parameters with a central server.Agents locally optimize policy parameters, which the server aggregates into updated parameters for the agents.
  • MT-FedRL objective: The unified policy maximizes the sum of long-term discounted returns, but may fail to remain near-optimal for individual environments when agents’ goals are not positively correlated.The paper identifies adversarial agents as an instance of this problem because their goals conflict with those of non-adversarial agents.

IV. MT-FEDRL WITH ADVERSARIES

In adversarial MT-FedRL, malicious policy parameters can prevent convergence to the jointly optimal unified policy and reduce discounted returns. Attack success depends on the adversarial update, non-adversarial learning rates, and the number of adversarial agents.

  • The unified policy is jointly optimal when agents’ goals are positively correlated, but may fail to be near-optimal for individual environments when goals are not positively correlated.
  • Adversarial policy parameters can shift the server’s unified-policy update away from the desired consensus θ∗.
  • Attack success is more likely when non-adversarial learning rates are small and the number of adversarial agents is large.
  • An adversary can reduce discounted returns by providing adversarial policies to the server, shifting the unified policy toward poor actions or less informative policy distributions.
  • The threat model scales an attack parameter with λk, whose magnitude affects attack strength and whose smaller value is preferred when attack performance is equal.

V. COMMON ATTACK MODELS

The paper reviews common attack models and motivates an adaptive attack model for MT-FedRL. It focuses on a single adversarial agent, with extension to multiple adversaries described as straightforward.

  • This section compares common attack models and proposes an adaptive attack model for MT-FedRL.
  • The adversarial objective is to shift the policy distribution toward actions that perform poorly.
  • The analysis focuses on one adversarial agent, while extending the threat models to multiple adversarial agents is described as straightforward.

A. Random Policy Attack (Rand)

The Random policy attack uses policy parameters sampled from a Gaussian distribution and assumes the adversary lacks knowledge for estimating a better attack. With sufficiently large scaling, it can shift the policy distribution toward randomness.

  • The Random policy attack samples adversarial policy parameters from a Gaussian distribution with mean 0 and standard deviation σ.
  • The attack assumes the adversary has no knowledge to estimate the best attack method.
  • With a sufficiently large scaling factor λk, the Random policy attack can shift the policy distribution toward a random distribution.

B. Opposite Goal Policy Attack (OppositeGoal)

The OppositeGoal attack learns locally to minimize discounted return and shares a scaled policy update with the server. Its intended uniform-distribution effect requires strict similarity among environments and aligned action preferences, so those assumptions often fail.

  • The OppositeGoal attack trains an adversary to minimize, rather than maximize, the long-term discounted return.
  • The adversary updates its policy after each episode and shares a scaled version of that update with the server.
  • For a uniform policy distribution, all environments must generate sufficiently similar policies, and opposite-goal and normal-goal action selections must align in the specified way.
  • The attack aims to produce complete information loss when all environments are sufficiently similar and opposite-goal training counteracts normal-goal training.
  • These assumptions are often too strict because environment dynamics, policy initialization, and local minima can differ; with large scaling, the attack instead shifts consensus toward an opposite-goal policy.

C. Adversarial Attack by Minimizing Information Gain (AdAMInG)

AdAMInG designs an adaptive model-poisoning attack around MT-FedRL's smoothing average, using locally available policy information to make the server forget non-adversarial agents' learning. Its effectiveness depends on the scaling factor, agent count, and learning rate, while requiring more memory than simpler attacks.

  • Attack objective: Forcing the smoothing average to lose information from non-adversarial agents is the central objective, enabling a single adversary to attack with a small scaling factor.The method targets information removal rather than directly steering the policy toward an opposite goal.
  • Attack construction: AdAMInG estimates the non-adversarial agents' smoothing average from the adversary's previous shared parameters and the server's federated parameters.The attack then constructs the policy parameters shared with the server without requiring environment data for learning.
  • Scaling-factor analysis: As the number of agents increases with λ_k = 1, g_ss approaches 1, making AdAMInG harder because non-adversarial updates increasingly dominate the server average.The authors also state that a non-zero g_ss can still yield a successful attack when the learning rate δ is small.
  • Scaling-factor analysis: A scaling factor equal to the number of non-adversarial agents guarantees a successful AdAMInG attack under the stated formulation.When the learning rate is not high, successful attacks can still occur with a smaller scaling factor.
  • Cost and limitation: The attack requires twice the memory of OppositeGoal or Rand, but avoids learning from sampled environmental data and therefore saves compute cost.The additional memory stores both the adversary's previous and newly computed policy parameters.

VI. DETECTING ATTACKS - COMA-FEDRL

ComA-FedRL adaptively varies communication intervals according to agents’ estimated adversarial behavior, reducing an adversary’s influence without permanently excluding agents.

  • VI. DETECTING ATTACKS - COMA-FEDRL: ComA-FedRL assigns communication intervals according to confidence that an agent is adversarial, communicating less frequently with agents judged more suspicious.The method avoids definitive adversary labels because agents may be misclassified or may disguise their behavior.
  • VI. DETECTING ATTACKS - COMA-FEDRL: The algorithm includes client updates, evaluation, communication-interval updates, and policy reassignment when all agents are active.The supplied algorithm description includes parallel agent operations and reassignment of another agent’s policy for evaluation.
  • VI. DETECTING ATTACKS - COMA-FEDRL: The GridWorld evaluation uses 12 environments to assess policies across multiple tasks.
  • VI. DETECTING ATTACKS - COMA-FEDRL: During pre-training, policies are evaluated across environments and agents with below-threshold average cumulative reward are marked possibly adversarial.Possibly adversarial agents receive higher communication intervals, while possibly non-adversarial agents communicate more frequently.

VII. EXPERIMENTATION

The experiments evaluate single-adversary MT-FedRL from tabular GridWorld to neural-network AutoNav using policy-gradient reinforcement learning.

  • VII. EXPERIMENTATION: Experiments use single-adversary MT-FedRL and cover GridWorld and AutoNav, progressing from tabular to neural-network-based reinforcement learning problems.Both experimental settings use policy-gradient RL methods.

A. GridWorld - Tabular RL

GridWorld experiments show that AdAMInG is more robust than common attacks to scaling and agent-count conditions, while ComA-FedRL sharply reduces adversarial impact.

  • A. GridWorld - Tabular RL: With scaling factor 1, AdAMInG achieves psa = 98%, whereas random-policy and OppositeGoal attacks are only half as effective.With optimal scaling factor n^-1, all three attacks achieve psa > 96%.
  • A. GridWorld - Tabular RL: Higher learning rates reduce attack performance because local policy updates outweigh the adversarial server update; AdAMInG remains most resilient to learning-rate and agent-count changes.
  • A. GridWorld - Tabular RL: For AdAMInG, larger learning rates reach an intermediate consensus earlier, yielding poorer attack performance but higher consensus-policy standard deviation.
  • A. GridWorld - Tabular RL: Increasing the number of agents can be compensated by increasing λ, producing similar AdAMInG attack performance for (λ = 1, n = 8) and (λ = 2, n = 12).The same relationship appears in discounted return across training episodes.
  • A. GridWorld - Tabular RL: ComA-FedRL reduces psa to as low as < 10% compared with conventional FedRL under different attack methods.The method assigns higher communication intervals to probable adversaries.

B. AutoNav - NN based RL

AutoNav evaluates MT-FedRL in four 3-D drone-navigation environments using image-based neural policies and shows that AdAMInG is highly effective against FedRL, while ComA-FedRL restores performance.

  • B. AutoNav - NN based RL: In AutoNav, the drone maps 320×180×3 RGB images to 25 actions using a neural-network policy in four indoor environments.The C3F2 network estimates action probabilities, and depth-based rewards encourage obstacle avoidance.
  • B. AutoNav - NN based RL: AdAMInG achieves psa of ∼99.5%, while all other attack methods achieve psa < 6% in AutoNav.
  • B. AutoNav - NN based RL: ComA-FedRL recovers the adversary-induced decrease in mean safe flight and reduces psa to < 10% compared with FedRL.Mean Safe Flight measures distance traveled before crashing across environments.

VIII. CONCLUSION

The paper introduces AdAMInG, an adaptive model-poisoning attack for MT-FedRL, and ComA-FedRL, a communication-adaptive defense. On GridWorld and AutoNav, AdAMInG generally outperforms other attacks while ComA-FedRL recovers near-optimal policies under attack.

  • AdAMInG accounts for the federated reinforcement learning aggregation operator when constructing its attack.
  • AdAMInG outperforms other attack methods almost every time on GridWorld maze solving and AutoNav drone navigation.
  • ComA-FedRL varies agents’ communication frequency according to their probability of being adversarial.
  • ComA-FedRL recovers from adversarial attacks and produces near-optimal policies on the evaluated problems.

APPENDIX

The appendix describes ComA-FedRL’s communication scheduling procedure, including local pre-training, cross-evaluation, reward-based interval assignment, and the listed training hyperparameters.

  • During pre-training, agents learn from local data and periodically share policies with the server for cross-evaluation.Policies are evaluated across randomly selected environments, and cumulative rewards are recorded.
  • Cross-evaluation runs policies on randomly selected environments and records cumulative rewards, while accounting for adversaries that may fake returned rewards.
  • After pre-training, cross-evaluated rewards determine each agent’s communication interval, with higher intervals assigned to poorer-performing policies.The mapping uses a reward threshold to assign different intervals.
  • Algorithm 3 is titled “Update Communication Intervals” and includes initializing low and high communication settings and a reward threshold for each agent.The procedure also averages rewards across episodes.
  • Table IV lists training hyperparameters for the GridWorld and AutoNav experiments.
Loading 2103.06473v1…