PaperlayerView exact revision on arXiv

Source-linked AI summary

Challenges and Countermeasures for Adversarial Attacks on Deep Reinforcement Learning

Inaam Ilahi, Muhammad Usama, Junaid Qadir, Muhammad Umar Janjua, Ala Al-Fuqaha, Dinh Thai Hoang, Dusit Niyato

arXiv:2001.09684v2cs.LGcs.AIcs.CR

TL;DR

DRL’s broad real-world use is constrained by vulnerabilities to adversarial attacks, especially in critical applications. This paper surveys DRL fundamentals, attacks, countermeasures, robustness evaluation, and open research challenges. It concludes that existing defenses are often attack-specific, while current metrics and benchmarks remain insufficient for comprehensive robustness assessment.

  • Problem

    DRL is vulnerable to adversarial attacks, while existing robustness benchmarks and metrics do not adequately measure resilience across the wide range of attacks and defenses.

  • Method

    The paper provides a comprehensive survey of DRL fundamentals, adversarial attacks, countermeasures, benchmarks, metrics, and open research challenges.

  • Results

    Existing DRL defenses are generally effective only against specified attack types and provide no guarantees against other attacks.

  • Takeaways & Limitations

    Robust DRL research needs broader benchmarks and standards for measuring resilience, alongside defenses that address diverse adversarial strategies.

  • Takeaways & Limitations

    Real-environment attacks are often black-box, with limited queries and no direct access to sensors, constraining adversarial capabilities and attack design.

Abstract

from arXiv · show

Deep Reinforcement Learning (DRL) has numerous applications in the real world thanks to its outstanding ability in quickly adapting to the surrounding environments. Despite its great advantages, DRL is susceptible to adversarial attacks, which precludes its use in real-life critical systems and applications (e.g., smart grids, traffic controls, and autonomous vehicles) unless its vulnerabilities are addressed and mitigated. Thus, this paper provides a comprehensive survey that discusses emerging attacks in DRL-based systems and the potential countermeasures to defend against these attacks. We first cover some fundamental backgrounds about DRL and present emerging adversarial attacks on machine learning techniques. We then investigate more details of the vulnerabilities that the adversary can exploit to attack DRL along with the state-of-the-art countermeasures to prevent such attacks. Finally, we highlight open issues and research challenges for developing solutions to deal with attacks for DRL-based intelligent systems.

I. INTRODUCTION

DRL extends reinforcement learning for complex sequential decisions but remains vulnerable to adversarial attacks, motivating a comprehensive review of attacks, defenses, benchmarks, and open challenges.

  • I. INTRODUCTION: DRL combines deep learning with reinforcement learning to address complex sequential decision-making problems.RL uses trial and error, while DRL improves its applicability through advances in deep learning.
  • I. INTRODUCTION: Adversaries can add imperceptible input perturbations to DRL systems to cause malfunctions, creating security concerns for critical applications.The paper identifies smart grids and autonomous vehicles as examples requiring vulnerability analysis before deployment.
  • I. INTRODUCTION: This survey reviews DRL fundamentals, adversarial attacks, potential countermeasures, robustness metrics, and available benchmarks.Its contributions include a taxonomy of advanced DRL algorithms and a comprehensive survey of attacks and defenses.
  • A. Fundamentals of DRL: A generic RL problem is modeled as an MDP in which an agent observes states, chooses actions, receives rewards, and seeks maximum long-term expected reward.The environment may be partially or fully observable, and actions may be discrete or continuous.
  • A. Fundamentals of DRL: Policies map perceived environmental states to actions and may be deterministic or stochastic, while value and advantage functions evaluate states and actions.The advantage function is defined as A(s, a) = Q(s, a) − V(s).

B. Security of ML

Machine-learning security attacks target either training or inference, and adversarial examples use carefully crafted perturbations to induce targeted misclassification.

  • B. Security of ML: ML security attacks are divided into training-phase attacks and inference-phase attacks.Training attacks seek to make the learned model or policy faulty by introducing malicious changes.
  • B. Security of ML: An adversarial example adds a small, imperceptible perturbation δ to an input x so classifier f(.) assigns targeted class t.The perturbation is obtained by iteratively approximating an optimization problem.
  • B. Security of ML: Adversarial examples are categorized according to objective, attacker knowledge, attack frequency, and specificity.Figure 4 presents a basic taxonomy of attacks on machine learning.

1) Attacks based on adversary’s knowledge:

Adversarial attacks are classified by how much the adversary knows about the target model, ranging from complete knowledge to no model knowledge.

  • White-box attacks assume complete knowledge of the target algorithm, training data, testing data, and model parameters.
  • Gray-box attacks use limited model knowledge, surrogate models, transferability, or restricted query access.
  • Black-box attacks assume that the adversary knows neither the model nor its attributes.

2) Attacks based on adversary’s goals:

DRL attacks can be organized by adversarial objective, specificity, and broader attack scope, reflecting both conventional ML goals and sequential decision-making vulnerabilities.

  • Objective-based attacks include confidence reduction, misclassification, targeted misclassification, and source/target misclassification.
  • Targeted attacks pursue specific output classes, whereas non-targeted attacks seek to misclassify the maximum number of samples.
  • DRL attacks are harder to recognize because adversarially forced actions can be difficult to distinguish from actions intentionally selected by the agent.
  • Because DRL uses sequential exploration and an MDP with state, action, reward, and transition components, adversaries have multiple attack surfaces.
  • The survey further divides DRL attacks into active attacks that change agent behavior and passive attacks that infer model or reward details.
  • Most surveyed DRL attacks perturb the state space, while fewer target the reward or action spaces.

A. Attacks Perturbing the State space

State-space attacks manipulate observations or related inputs to reduce reward, force undesirable actions or states, and exploit weaknesses in DRL policies.

  • Policy induction attacks train an adversarial policy and generate inputs that make a target DQN follow its actions.
  • Strategically-timed attacks perturb selected time steps to minimize reward, whereas enchanting attacks lure agents toward predefined target states.
  • Gradient-based attacks perturb observations for DQN and DDPG, with improved methods steering agents toward predefined adversarial states.
  • Value-guided perturbation reduces the number of attacked states, while the last proposed approach performs as well as perturbing all states.
  • General-purpose critical point and antagonist attacks address limitations of earlier attacks, including restricted applicability to continuous action spaces.
  • Per-observation and universal-mask attacks were proposed to make policy-control attacks more practically feasible.
  • Static reward impact maps identify state features with large cumulative-reward influence, but their generation time is a stated limitation.
  • Cooperative multi-agent attacks can exploit access to one agent to attack the whole system.

2) Manipulating the Environment:

Environment-manipulation attacks alter surroundings, sensory inputs, or training data to mislead DRL agents, often targeting navigation or learned policy behavior.

  • Confusing obstacles added to pathfinding maps disrupted local information and succeeded against A3C at least 99.91% of the time.
  • Online sequential environment attacks exploit temporal state consistency and use model querying without back-propagation.
  • An adversarial agent can create natural observations in the shared environment that induce a legitimate PPO agent to follow a target policy.
  • Timing-based adaptive attacks include a white-box weighted majority algorithm and a black-box adversarial-strategic agent.
  • Modifying 0.025% of training data enabled a Trojan attack that induced hidden policy behaviors.
  • A unique response to a state sequence can watermark a policy while minimally affecting performance and help detect unauthorized replication.
  • Real-time attacks on robots in dynamic environments remain relatively underexplored, including attacks that tamper with sensory data to induce wrong routes.
  • Adding noise to a DRL-based channel autoencoder’s feedback channel for a time interval caused a performance drop.

B. Attacks Perturbing the Reward function

The survey organizes DRL attacks by the targeted MDP component, including reward signals, actions, states, and models. Reward and action manipulations can reduce performance or steer behavior under different attacker access settings.

  • Reward and state attacks: Attackers can manipulate reward signals and states in online DRL settings under white-box or black-box access.Reward flipping changes binary signals, while state manipulation targets the agent’s observations.
  • Reward and state attacks: Training-time environment poisoning can manipulate rewards and transition dynamics to force an agent toward an adversary-selected target policy.The attack is considered in undiscounted infinite-horizon settings and tested in offline and online scenarios.
  • Action-space attacks: LAS attacks are more lethal than MAS attacks because temporally coupled constraints can attack the agent’s dynamic information.The attacks minimize cumulative reward and can remain effective under limited resources.
  • Action-space attacks: Query-based action-space attacks against cyber-physical DRL systems can be reduced by half through adversarial training.The attack is designed for settings where complex cyber-physical models are difficult to obtain.
  • Model-space attacks: Model extraction attacks use iterative queries and imitation learning to replicate a victim DRL policy, with adversarial regret measuring the resulting attack impact.Adversarial regret is the difference between the maximum return achievable by the trained policy and the return from adversarial-policy actions.
  • Attack taxonomy: The survey categorizes DRL attacks by state, action, reward, and model spaces, while practical attacks may indirectly affect multiple pipeline components through environmental perturbations.Query-efficient attacks are identified as a need for real-world black-box settings.

IV. DEFENSES AGAINST ADVERSARIAL ATTACKS ON DRL

The survey reviews defenses grouped into adversarial training, robust learning, detection, distillation, and game-theoretic approaches. These methods improve robustness in several settings, but their effectiveness often remains attack-specific.

  • Defense taxonomy: The defense taxonomy includes adversarial training, robust learning, adversarial detection, defensive distillation, and game-theoretic approaches.The approaches use adversarial examples, robust MDP alternatives, or interactions with adversaries.
  • Adversarial training: Adversarial training retrains DRL models with adversarial and legitimate examples to improve robustness against related perturbations.Robustness may not extend to adversarial examples generated through other methods.
  • Adversarial training: Noise-augmented training increases resilience to gradient-based attacks by helping DRL models represent system uncertainties.Pattanaik et al. describe this procedure as equivalent to Robust Control.
  • Adversarial training: Adversarially trained policies can become robust to test-time attacks after adversarial samples in memory reach a critical limit.The cited work examines both training-time and test-time attacks on DQN.
  • Adversarial training: NoisyNets are more resilient to training-time attacks than epsilon-greedy policy learning, which the authors associate with improved generalizability and reduced transferability.The method is proposed for attacks such as FGSM.
  • Robust and game-theoretic learning: Population-based adversary sampling and minimax policy learning address robustness when training conditions include adversaries or differ from test conditions.These approaches use populations of adversaries or zero-sum objectives.
  • Robust and game-theoretic learning: Robust learning with Wasserstein constraints improves robustness on low- and high-dimensional control tasks, while DDPG shows less improvement than TRPO and PPO.The reported comparison identifies acceptable performance for TRPO and PPO but not significant robustness gains for DDPG.
  • Robust and game-theoretic learning: In adversarially perturbed payoff games, the single-player stochastic dynamics yield no regret almost surely regardless of observation-noise level.The result is reported for the single-player case with an arbitrarily changing environment.

C. Robust Learning

Robust learning modifies training, rewards, policies, or risk modeling to withstand adversarial perturbations. Reported benefits include improved rewards, resilience under attacks, safer behavior, and better generalization, with effectiveness depending on the setting.

  • Training and policy robustness: Parameter-noise retraining mitigates training- and test-time attacks in black-box and white-box settings across three Atari games.The cited experiments use FGSM-crafted adversarial samples and report strong performance for parameter-noise agents.
  • Reward robustness: Perturbed rewards improve PPO’s average reward by 67.5% at a 10% error rate and 46.7% at a 30% error rate.The framework uses a reward confusion matrix and an unbiased reward estimator for noisy inputs.
  • Training and policy robustness: Distributionally robust policy iteration restricts sub-optimal policies in high-dimensional state and action spaces through robust Bellman operators and dynamic risk levels.The method is motivated by robustness to non-stationary environments and state-space noise.
  • Training and policy robustness: AR-DDPG learns action-robust policies under probabilistic and noisy-action MDP criteria, improving security and performance even without perturbations.The techniques are evaluated in multiple MuJoCo environments.
  • Reward robustness: A neural-network noise filter can estimate true rewards, but beyond a perturbation probability of 0.5 the agent begins learning from adversarial samples.The method compares estimated rewards with received rewards to filter noisy samples.
  • Training and policy robustness: RS-DQN matches DQN without attacks, remains robust when DQN fails under attacks, and gains resilience to strong attacks when combined with adversarial training.RS-DQN separates a robustly trained student policy network from a normally trained Q-network.
  • Risk and safety robustness: Risk-averse robust adversarial reinforcement learning produces fewer crashes than a normal agent in a TORCS self-driving environment.The method models risk using an ensemble of q-value networks and pairs a risk-averse agent with a risk-seeking adversary.
  • Risk and safety robustness: FRARL policies generalize better and violate fewer safety specifications in test scenarios than techniques similar to RARL.FRARL integrates temporal-logic falsification and removes the need to construct an extra adversary reward function.

D. Adversarial Detection

Adversarial detection identifies perturbed inputs without changing the original model, while related defenses use supervisory agents, engineered features, or distillation. The survey emphasizes that current defenses are often attack-specific and benchmarking remains insufficient.

  • Detection methods: Adversarial detection separates true and adversarial samples with a specialized model, allowing perturbed inputs to be disregarded without modifying the original DRL model.The surveyed detection method uses action-conditioned frames to address attacks in decision space.
  • Detection methods: A supervisory agent can detect unexpected adversarial examples when an agent learns sub-policies before its ultimate policy.The cited setting involves state-space corruption during training at specific periods.
  • Detection methods: A robot path-finding detector predicts adversarial inputs using five factors whose weights are calculated by principal component analysis.The factors include energy-point, key-point, and path gravitation, included angle, and placid point.
  • Alternative robust formulations: Threatened MDPs with level-2 reasoning can approximately estimate adversarial behavior, whereas normal Q-learning is exploitable by the adversary.The framework targets adversaries affecting reward generation.
  • Defensive distillation: Defensive distillation transfers learning information from a complex model to a simpler one, but the surveyed discussion indicates that distillation alone may be ineffective.The discussion reports expected entropy regularized distillation as faster while guaranteeing convergence, and robust policy distillation without adversarial examples during student training.
  • Limitations and evaluation: The survey groups defenses into five categories and notes that most are effective only against specified attack types rather than providing guarantees against others.It also reports that few defenses address DRL systems whose observations do not involve images.
  • Limitations and evaluation: Current DRL benchmarks are insufficient for measuring security because evaluations must account for the temporal domain rather than accuracy alone.The survey identifies applicable robustness and resilience evaluation as an unmet need.
  • Limitations and evaluation: Adversarial budget measures perturbable observation features and perturbation probability, while adversarial regret measures the reward difference between unperturbed and perturbed agents after an episode.These measures support definitions of test-time resilience and test-time robustness.

A. Test-time Resilience and Robustness

The paper distinguishes test-time resilience from robustness and surveys procedures and metrics for evaluating DRL under adversarial policies and attacks.

  • Test-time resilience is the minimum number of perturbations required to cause the maximum reduction in return at time t.
  • Test-time robustness is measured as the maximum achievable adversarial regret.
  • The resilience procedure can approximate the state-action value function through policy imitation when it is unavailable.
  • It trains an adversarial agent against the target policy, reports optimal adversarial return and maximum adversarial regret, and applies the policy across episodes.
  • Mean test-time resilience is the average adversarial return over episodes, whereas robustness replaces this with average adversarial regret.
  • Existing metrics include performance gap, percentage of target action, and time to failure, but cover only part of DRL algorithms.

C. Attacking DRL: Tools and Platforms

The paper surveys DRL implementation and testing platforms while identifying unresolved challenges in robustness, multitask learning, and standardized evaluation.

  • Tools and Platforms: DRL can be implemented with multiple toolkits or combinations of toolkits for testing algorithms and attacks in simulated environments.
  • Tools and Platforms: OpenAI Gym supplies gaming environments and can be combined with TensorFlow for neural-network action selection from states.
  • Tools and Platforms: OpenAI Baselines, RLCoach, and Horizon provide implementations or integrated mechanisms for implementing and testing DRL algorithms.
  • Open Challenges: Existing defenses generally protect only against the attacks for which they were designed, leaving vulnerability to proactive adversaries.
  • Open Challenges: Current DRL algorithms typically learn one task at a time and require retraining from scratch for different games.
  • Open Challenges: The paper identifies inadequate robustness benchmarks and calls for standardized measures covering diverse adversarial attacks and defenses.

D. System Design and Transferability

The paper frames secure DRL as a system-design problem involving robust rewards, computationally feasible defenses, privacy, explainability, transfer learning, and competing efficiency requirements.

  • System Design: DRL system design needs standards for unsupervised learning and reward functions because agents may learn irrelevant features or intermediary errors.
  • System Design: Ensemble defenses may exponentially increase model complexity and reduce real-time performance, making computationally efficient DRL protection challenging.
  • Privacy: Model-extraction attacks threaten learned-model integrity through illegal duplication; proposed responses include increasing attack costs, watermarking policies, and constrained randomization.
  • Explainability: Current DRL explainability techniques do not specifically target testers, developers, and the general public as distinct audiences.
  • Transferability: Transfer learning can reduce target-model training data and time, but DRL research remains limited to some specific algorithms.
  • Roadmap: Secure DRL must jointly pursue task efficiency, computational efficiency, adversarial robustness, and sample efficiency, requiring tradeoffs among them.
  • Roadmap: The proposed roadmap prioritizes sample-efficient inherently robust algorithms, human-oriented explainability, and metrics for robustness and resilience.
Loading 2001.09684v2…