Challenges and Countermeasures for Adversarial Attacks on Deep Reinforcement Learning

I. INTRODUCTION

• DRL combines deep learning with reinforcement learning to address complex sequential decision-making problems.RL uses trial and error, while DRL improves its applicability through advances in deep learning.Source: §I · p. 1
• Adversaries can add imperceptible input perturbations to DRL systems to cause malfunctions, creating security concerns for critical applications.The paper identifies smart grids and autonomous vehicles as examples requiring vulnerability analysis before deployment.Source: §I · p. 1
• This survey reviews DRL fundamentals, adversarial attacks, potential countermeasures, robustness metrics, and available benchmarks.Its contributions include a taxonomy of advanced DRL algorithms and a comprehensive survey of attacks and defenses.Source: §I · p. 2

A. Fundamentals of DRL

• A generic RL problem is modeled as an MDP in which an agent observes states, chooses actions, receives rewards, and seeks maximum long-term expected reward.The environment may be partially or fully observable, and actions may be discrete or continuous.Source: A. Fundamentals of DRL · pp. 2–3
• Policies map perceived environmental states to actions and may be deterministic or stochastic, while value and advantage functions evaluate states and actions.The advantage function is defined as A(s, a) = Q(s, a) − V(s).Source: A. Fundamentals of DRL · pp. 3–4

B. Security of ML

• ML security attacks are divided into training-phase attacks and inference-phase attacks.Training attacks seek to make the learned model or policy faulty by introducing malicious changes.Source: B. Security of ML · p. 4
• An adversarial example adds a small, imperceptible perturbation δ to an input x so classifier f(.) assigns targeted class t.The perturbation is obtained by iteratively approximating an optimization problem.Source: B. Security of ML · p. 4
• Adversarial examples are categorized according to objective, attacker knowledge, attack frequency, and specificity.Figure 4 presents a basic taxonomy of attacks on machine learning.Source: B. Security of ML · p. 4

• White-box attacks assume complete knowledge of the target algorithm, training data, testing data, and model parameters.Source: 1) Attacks based on adversary’s… · p. 4
• Gray-box attacks use limited model knowledge, surrogate models, transferability, or restricted query access.Source: 1) Attacks based on adversary’s… · p. 5
• Black-box attacks assume that the adversary knows neither the model nor its attributes.Source: 1) Attacks based on adversary’s… · p. 5

• Objective-based attacks include confidence reduction, misclassification, targeted misclassification, and source/target misclassification.Source: 2) Attacks based on adversary’s goals · p. 5
• Targeted attacks pursue specific output classes, whereas non-targeted attacks seek to misclassify the maximum number of samples.Source: 3) Attacks based on adversary’s… · p. 5
• DRL attacks are harder to recognize because adversarially forced actions can be difficult to distinguish from actions intentionally selected by the agent.Source: C. Security of DRL · p. 5
• Because DRL uses sequential exploration and an MDP with state, action, reward, and transition components, adversaries have multiple attack surfaces.Source: C. Security of DRL · p. 6
• The survey further divides DRL attacks into active attacks that change agent behavior and passive attacks that infer model or reward details.Source: C. Security of DRL · p. 6
• Most surveyed DRL attacks perturb the state space, while fewer target the reward or action spaces.Source: §III · p. 6

• Policy induction attacks train an adversarial policy and generate inputs that make a target DQN follow its actions.Source: 1) Manipulating the Observations · p. 6
• Strategically-timed attacks perturb selected time steps to minimize reward, whereas enchanting attacks lure agents toward predefined target states.Source: 1) Manipulating the Observations · p. 6, Figure 7 · p. 8
• Gradient-based attacks perturb observations for DQN and DDPG, with improved methods steering agents toward predefined adversarial states.Source: 1) Manipulating the Observations · p. 7
• Value-guided perturbation reduces the number of attacked states, while the last proposed approach performs as well as perturbing all states.Source: 1) Manipulating the Observations · p. 7
• General-purpose critical point and antagonist attacks address limitations of earlier attacks, including restricted applicability to continuous action spaces.Source: 1) Manipulating the Observations · p. 7
• Per-observation and universal-mask attacks were proposed to make policy-control attacks more practically feasible.Source: 1) Manipulating the Observations · p. 7
• Static reward impact maps identify state features with large cumulative-reward influence, but their generation time is a stated limitation.Source: 1) Manipulating the Observations · p. 8
• Cooperative multi-agent attacks can exploit access to one agent to attack the whole system.Source: 1) Manipulating the Observations · p. 8

• Confusing obstacles added to pathfinding maps disrupted local information and succeeded against A3C at least 99.91% of the time.Source: 2) Manipulating the Environment · p. 8
• Online sequential environment attacks exploit temporal state consistency and use model querying without back-propagation.Source: 2) Manipulating the Environment · pp. 8–9
• An adversarial agent can create natural observations in the shared environment that induce a legitimate PPO agent to follow a target policy.Source: 2) Manipulating the Environment · p. 9
• Timing-based adaptive attacks include a white-box weighted majority algorithm and a black-box adversarial-strategic agent.Source: 2) Manipulating the Environment · p. 9
• Modifying 0.025% of training data enabled a Trojan attack that induced hidden policy behaviors.Source: 3) Manipulating the Training Data · p. 9
• A unique response to a state sequence can watermark a policy while minimally affecting performance and help detect unauthorized replication.Source: 3) Manipulating the Training Data · p. 9
• Real-time attacks on robots in dynamic environments remain relatively underexplored, including attacks that tamper with sensory data to induce wrong routes.Source: 4) Manipulating the Sensors · p. 9
• Adding noise to a DRL-based channel autoencoder’s feedback channel for a time interval caused a performance drop.Source: 4) Manipulating the Sensors · p. 9

Reward and state attacks

• Attackers can manipulate reward signals and states in online DRL settings under white-box or black-box access.Reward flipping changes binary signals, while state manipulation targets the agent’s observations.Source: B. Attacks Perturbing the Reward… · p. 9
• Training-time environment poisoning can manipulate rewards and transition dynamics to force an agent toward an adversary-selected target policy.The attack is considered in undiscounted infinite-horizon settings and tested in offline and online scenarios.Source: B. Attacks Perturbing the Reward… · pp. 9–10

Action-space attacks

• LAS attacks are more lethal than MAS attacks because temporally coupled constraints can attack the agent’s dynamic information.The attacks minimize cumulative reward and can remain effective under limited resources.Source: C. Attacks Perturbing the Action space · p. 10
• Query-based action-space attacks against cyber-physical DRL systems can be reduced by half through adversarial training.The attack is designed for settings where complex cyber-physical models are difficult to obtain.Source: C. Attacks Perturbing the Action space · p. 10

Model-space attacks

• Model extraction attacks use iterative queries and imitation learning to replicate a victim DRL policy, with adversarial regret measuring the resulting attack impact.Adversarial regret is the difference between the maximum return achievable by the trained policy and the return from adversarial-policy actions.Source: D. Attacks Perturbing the Model Space · p. 10

Attack taxonomy

• The survey categorizes DRL attacks by state, action, reward, and model spaces, while practical attacks may indirectly affect multiple pipeline components through environmental perturbations.Query-efficient attacks are identified as a need for real-world black-box settings.Source: E. Discussion · pp. 10–11

Defense taxonomy

• The defense taxonomy includes adversarial training, robust learning, adversarial detection, defensive distillation, and game-theoretic approaches.The approaches use adversarial examples, robust MDP alternatives, or interactions with adversaries.Source: §IV · p. 12

Adversarial training

• Adversarial training retrains DRL models with adversarial and legitimate examples to improve robustness against related perturbations.Robustness may not extend to adversarial examples generated through other methods.Source: A. Adversarial Training · p. 12
• Noise-augmented training increases resilience to gradient-based attacks by helping DRL models represent system uncertainties.Pattanaik et al. describe this procedure as equivalent to Robust Control.Source: A. Adversarial Training · p. 12
• Adversarially trained policies can become robust to test-time attacks after adversarial samples in memory reach a critical limit.The cited work examines both training-time and test-time attacks on DQN.Source: A. Adversarial Training · p. 12
• NoisyNets are more resilient to training-time attacks than epsilon-greedy policy learning, which the authors associate with improved generalizability and reduced transferability.The method is proposed for attacks such as FGSM.Source: A. Adversarial Training · p. 12

Robust and game-theoretic learning

• Population-based adversary sampling and minimax policy learning address robustness when training conditions include adversaries or differ from test conditions.These approaches use populations of adversaries or zero-sum objectives.Source: A. Adversarial Training · p. 12, B. Game-theoretic Approach · p. 12
• Robust learning with Wasserstein constraints improves robustness on low- and high-dimensional control tasks, while DDPG shows less improvement than TRPO and PPO.The reported comparison identifies acceptable performance for TRPO and PPO but not significant robustness gains for DDPG.Source: B. Game-theoretic Approach · p. 13
• In adversarially perturbed payoff games, the single-player stochastic dynamics yield no regret almost surely regardless of observation-noise level.The result is reported for the single-player case with an arbitrarily changing environment.Source: B. Game-theoretic Approach · p. 13

Training and policy robustness

• Parameter-noise retraining mitigates training- and test-time attacks in black-box and white-box settings across three Atari games.The cited experiments use FGSM-crafted adversarial samples and report strong performance for parameter-noise agents.Source: C. Robust Learning · p. 13

Reward robustness

• Perturbed rewards improve PPO’s average reward by 67.5% at a 10% error rate and 46.7% at a 30% error rate.The framework uses a reward confusion matrix and an unbiased reward estimator for noisy inputs.Source: C. Robust Learning · p. 13

Training and policy robustness

• Distributionally robust policy iteration restricts sub-optimal policies in high-dimensional state and action spaces through robust Bellman operators and dynamic risk levels.The method is motivated by robustness to non-stationary environments and state-space noise.Source: C. Robust Learning · p. 13
• AR-DDPG learns action-robust policies under probabilistic and noisy-action MDP criteria, improving security and performance even without perturbations.The techniques are evaluated in multiple MuJoCo environments.Source: C. Robust Learning · p. 13

Reward robustness

• A neural-network noise filter can estimate true rewards, but beyond a perturbation probability of 0.5 the agent begins learning from adversarial samples.The method compares estimated rewards with received rewards to filter noisy samples.Source: C. Robust Learning · p. 13

Training and policy robustness

• RS-DQN matches DQN without attacks, remains robust when DQN fails under attacks, and gains resilience to strong attacks when combined with adversarial training.RS-DQN separates a robustly trained student policy network from a normally trained Q-network.Source: C. Robust Learning · pp. 13–14

Risk and safety robustness

• Risk-averse robust adversarial reinforcement learning produces fewer crashes than a normal agent in a TORCS self-driving environment.The method models risk using an ensemble of q-value networks and pairs a risk-averse agent with a risk-seeking adversary.Source: C. Robust Learning · p. 14
• FRARL policies generalize better and violate fewer safety specifications in test scenarios than techniques similar to RARL.FRARL integrates temporal-logic falsification and removes the need to construct an extra adversary reward function.Source: C. Robust Learning · p. 14

Detection methods

• Adversarial detection separates true and adversarial samples with a specialized model, allowing perturbed inputs to be disregarded without modifying the original DRL model.The surveyed detection method uses action-conditioned frames to address attacks in decision space.Source: D. Adversarial Detection · p. 14
• A supervisory agent can detect unexpected adversarial examples when an agent learns sub-policies before its ultimate policy.The cited setting involves state-space corruption during training at specific periods.Source: D. Adversarial Detection · p. 15
• A robot path-finding detector predicts adversarial inputs using five factors whose weights are calculated by principal component analysis.The factors include energy-point, key-point, and path gravitation, included angle, and placid point.Source: D. Adversarial Detection · p. 15

Alternative robust formulations

• Threatened MDPs with level-2 reasoning can approximately estimate adversarial behavior, whereas normal Q-learning is exploitable by the adversary.The framework targets adversaries affecting reward generation.Source: D. Adversarial Detection · p. 15

Defensive distillation

• Defensive distillation transfers learning information from a complex model to a simpler one, but the surveyed discussion indicates that distillation alone may be ineffective.The discussion reports expected entropy regularized distillation as faster while guaranteeing convergence, and robust policy distillation without adversarial examples during student training.Source: E. Defensive Distillation · p. 15

Limitations and evaluation

• The survey groups defenses into five categories and notes that most are effective only against specified attack types rather than providing guarantees against others.It also reports that few defenses address DRL systems whose observations do not involve images.Source: F. Discussion · p. 15
• Current DRL benchmarks are insufficient for measuring security because evaluations must account for the temporal domain rather than accuracy alone.The survey identifies applicable robustness and resilience evaluation as an unmet need.Source: §V · p. 15
• Adversarial budget measures perturbable observation features and perturbation probability, while adversarial regret measures the reward difference between unperturbed and perturbed agents after an episode.These measures support definitions of test-time resilience and test-time robustness.Source: §V · p. 16

• Test-time resilience is the minimum number of perturbations required to cause the maximum reduction in return at time t.Source: A. Test-time Resilience and Robustness · p. 17
• Test-time robustness is measured as the maximum achievable adversarial regret.Source: A. Test-time Resilience and Robustness · p. 17
• The resilience procedure can approximate the state-action value function through policy imitation when it is unavailable.Source: A. Test-time Resilience and Robustness · p. 17
• It trains an adversarial agent against the target policy, reports optimal adversarial return and maximum adversarial regret, and applies the policy across episodes.Source: A. Test-time Resilience and Robustness · p. 17
• Mean test-time resilience is the average adversarial return over episodes, whereas robustness replaces this with average adversarial regret.Source: A. Test-time Resilience and Robustness · p. 17
• Existing metrics include performance gap, percentage of target action, and time to failure, but cover only part of DRL algorithms.Source: B. Metrics for Attack Performance · p. 17

Tools and Platforms

• DRL can be implemented with multiple toolkits or combinations of toolkits for testing algorithms and attacks in simulated environments.Source: C. Attacking DRL: Tools and Platforms · p. 17
• OpenAI Gym supplies gaming environments and can be combined with TensorFlow for neural-network action selection from states.Source: C. Attacking DRL: Tools and Platforms · p. 17
• OpenAI Baselines, RLCoach, and Horizon provide implementations or integrated mechanisms for implementing and testing DRL algorithms.Source: C. Attacking DRL: Tools and Platforms · p. 17

Open Challenges

• Existing defenses generally protect only against the attacks for which they were designed, leaving vulnerability to proactive adversaries.Source: A. Universally Robust Algorithms · p. 17
• Current DRL algorithms typically learn one task at a time and require retraining from scratch for different games.Source: B. Multitask Learning · p. 17
• The paper identifies inadequate robustness benchmarks and calls for standardized measures covering diverse adversarial attacks and defenses.Source: C. Metrics for Robustness and Resilience · p. 18

System Design

• DRL system design needs standards for unsupervised learning and reward functions because agents may learn irrelevant features or intermediary errors.Source: D. System Design and Transferability · p. 18
• Ensemble defenses may exponentially increase model complexity and reduce real-time performance, making computationally efficient DRL protection challenging.Source: E. Ensemble of Defenses · p. 18

Privacy

• Model-extraction attacks threaten learned-model integrity through illegal duplication; proposed responses include increasing attack costs, watermarking policies, and constrained randomization.Source: F. Model Privacy · p. 18

Explainability

• Current DRL explainability techniques do not specifically target testers, developers, and the general public as distinct audiences.Source: G. Explainable and Transparent DRL · p. 18

Transferability

• Transfer learning can reduce target-model training data and time, but DRL research remains limited to some specific algorithms.Source: H. Transfer Learning for DRL · p. 18

Roadmap

• Secure DRL must jointly pursue task efficiency, computational efficiency, adversarial robustness, and sample efficiency, requiring tradeoffs among them.Source: §I · p. 18
• The proposed roadmap prioritizes sample-efficient inherently robust algorithms, human-oriented explainability, and metrics for robustness and resilience.Source: §I · p. 18