Source-linked AI summary
Learning from the Test: Self-Referential Differential Testing for Deep RL Agents
Junda He, Jieke Shi, Zhou Yang, Mingfei Cheng, David Lo
TL;DR
DRL testing often emphasizes catastrophic failures while lacking effective ways to assess policy optimality. Delta uses safety-testing trajectories to train offline-RL challengers for differential testing, and BCQ-based challengers uncovered 2,518 optimality issues per environment and 50.2% more than baselines.
Problem
Existing DRL testing largely overlooks policy optimality, despite its importance alongside safety and the difficulty of establishing an optimality oracle.
Method
Delta combines safety testing with trajectory collection and offline-RL challenger training, then flags cases where challengers achieve higher cumulative rewards than the AUT.
Results
2,518 optimality issues per environment were uncovered by BCQ-trained challengers, identifying 50.2% more optimality bugs than baseline methods across five environments.
Takeaways & Limitations
Safety-testing datasets can train competent challengers, with BCQ proving most effective for identifying optimality issues within Delta.
Takeaways & Limitations
Results may not generalize broadly, and underexplored state-space regions can leave challengers undertrained and produce false negatives.
Abstract
from arXiv · showhide
Deep Reinforcement Learning (DRL) has achieved significant success in complex decision-making problems. As DRL systems are increasingly deployed in real-world applications, ensuring their quality and reliability is paramount. Current works primarily focus on detecting safety-critical failures, often neglecting policy optimality, which can lead to reduced efficiency, user distrust, and economic losses. This oversight, compounded by the inherent "testing oracle problem" for optimality, leaves a significant gap in comprehensively evaluating DRL systems. To address this gap, we propose Delta (Differential Testing for DRL Agents), a novel and comprehensive framework that automatically identifies both safety-critical and optimality bugs in DRL agents. Delta employs a two-phase approach: (1) Safety Testing, where the Agent Under Test (AUT) is evaluated for catastrophic failures while collecting data from its decision-making policy, and (2) Optimality Testing, where this collected data from the prior phase is used to train a challenger agent via Offline Reinforcement Learning. Differential testing is then performed by comparing the challenger agent against the AUT; instances where the challenger achieves higher cumulative rewards indicate optimality issues in the AUT. We demonstrate Delta's effectiveness across five environments. We investigate the effectiveness of three offline RL algorithms (BC, BCQ, and CQL) in generating challenger agents. Experimental results demonstrate that safety testing datasets are valuable for training competent DRL agents. Challenger agents trained with BCQ proved most effective for identifying optimality issues within the framework of Delta. Across the five environments, Delta uncovered an average of 2,518 optimality issues, outperforming the baseline methods by 50.2%.
1 Introduction
Delta addresses the difficulty of evaluating DRL policy optimality without a known oracle by reusing safety-testing data to train a challenger agent for differential testing. Its two-phase framework evaluates both catastrophic failures and suboptimal behavior across diverse environments and offline RL algorithms.
- Motivation: Current DRL testing primarily detects catastrophic safety failures, leaving policy optimality insufficiently evaluated.Examples include DRL-controlled robots colliding with obstacles.
- Motivation: Unknown or analytically intractable optimal solutions create a testing-oracle problem for quantitatively assessing DRL policy optimality.Complex and dynamic environments often lack a ground-truth policy achieving theoretical maximum cumulative reward.
- Approach: Delta trains a challenger agent from the AUT’s safety-testing interaction data and compares cumulative rewards to expose optimality issues.The challenger serves as a self-referential testing oracle, avoiding the need to know the true optimum.
- Approach: Safety-testing data can support competent challenger agents because it contains edge cases and successful decisions from well-trained AUTs.This data provides a strong training starting point while reducing the need for inefficient exploration from scratch.
- Evaluation: Delta integrates safety testing and differential optimality testing into one pipeline for holistic DRL-agent evaluation.The framework was evaluated across CartPole, MountainCar, Acrobot, Hopper, and Walker2D using BC, BCQ, and CQL to generate challengers.
2 Preliminaries
This section introduces reinforcement learning through Markov Decision Processes and defines core concepts such as states, actions, transitions, trajectories, episodes, and policies. It also presents Offline RL as learning reward-maximizing policies from pre-collected trajectory datasets without further environmental interaction.
- Reinforcement Learning and MDPs: Reinforcement learning addresses learning to control a dynamical system, commonly formalized as an MDP tuple ⟨S,S_0,A,T,R,γ⟩.The tuple comprises states, possible starting states, actions, transition probabilities, rewards, and a discount factor.
- Reinforcement Learning and MDPs: An MDP state represents the agent’s situation, while actions affect the current state and T(s′|s,a) gives transition probabilities between states.S is the state set, S_0 contains possible starting states, and A is the action set.
- Core RL Concepts: The Markov property makes the future state depend only on the current state and action, and a trajectory is a sequence of state, action, and reward tuples.An episode is a complete trajectory from a start state to a terminal state.
- Offline Reinforcement Learning: Offline RL learns policies exclusively from pre-collected trajectory datasets, avoiding further interaction with the environment.This is valuable when environmental interaction is expensive, time-consuming, or risky.
- Offline Reinforcement Learning: The Offline RL objective is to learn a policy π:S→A using only dataset D while maximizing expected cumulative discounted reward.D is generated by a behavior policy, which may use random exploration, rules, expert demonstrations, or previously trained agents.
3 Approach
Delta is a black-box, two-phase framework that uses safety-testing data to train an offline-RL challenger, then differentially fuzzes the AUT to detect optimality issues. It flags an issue when the challenger achieves higher cumulative reward than the AUT from the same initial state, under a configurable threshold.
- Assumptions: The methodology assumes black-box access to deterministic AUT policies and deterministic environment transitions, with observations limited to interaction tuples.It is designed for environments modeled as Markov Decision Processes.
- Two-Phase Testing: Delta first performs safety testing, collecting the AUT’s observed behaviors as trajectory data for subsequent offline-RL training.The implementation uses CureFuzz, though Delta can integrate with other safety-testing frameworks.
- Two-Phase Testing: The collected dataset trains a challenger agent via offline reinforcement learning, which then serves as the testing oracle in differential fuzzing.Delta directly reuses existing safety-testing logs and avoids additional environment interactions during challenger training.
- Optimality Oracle: An optimality issue occurs when the AUT achieves lower cumulative reward than the challenger from the same initial state.The reward function is treated as the reference objective, and experiments use the default threshold δ=0.
- Differential Fuzzing: Delta uses energy-guided iterative fuzzing: it mutates high-energy initial-state seeds and evaluates resulting AUT and challenger behaviors.Seed diversity, regret, and inconsistency guide whether generated scenarios are retained and prioritized.
4 Experiment Setting
Delta is evaluated across five established DRL environments spanning classical control and MuJoCo, with environment-specific AUT algorithms and offline RL challengers. The experiments verify AUT performance before safety testing and use specified replication-package hyperparameters and hardware.
- Environments: 5 environments—CartPole, MountainCar, Acrobot, Hopper, and Walker2D—are used to assess Delta across classical control and MuJoCo settings.The first three are classical control environments, while Hopper and Walker2D are MuJoCo environments.
- AUT Implementation: DDQN trains AUTs in the three discrete-action environments, while SAC trains AUTs in Hopper and Walker2D.The implementations come from the stable and verified d3rlpy open-source library.
- AUT Validation: All AUTs were confirmed to have high performance before safety testing, using average cumulative reward and 95% confidence interval margins of error over 100 runs.Detailed results are reported in Table 1.
- Challenger Implementation: Three prominent offline RL methods are evaluated for challenger training, including BC, which learns a policy by mimicking state-action behavior.The passage introduces offline RL as the challenger-training approach but truncates the descriptions of the remaining methods.
- Experimental Configuration: MuJoCo hyperparameters come from the D4RL replication package, classical-control hyperparameters from Scope-RL, and experiments run in Python 3.9 on Ubuntu 22.04 with four NVIDIA RTX A5000 GPUs.The server also has an AMD EPYC 7643 48-core Processor and 504 GB RAM.
5 Results · RQ1: Which offline RL algorithm is most effective for training challenger agents to identify optimality bugs?
Delta’s RQ1 evaluation trains BC, BCQ, and CQL challengers from safety-testing datasets and compares their ability to detect optimality bugs. BCQ is most effective across all five environments, detecting an average of 2,517.9 issues per environment and becoming Delta’s default oracle.
- 5 Results: Safety testing ran five independent times per environment, recording every AUT trajectory into datasets used for challenger training.Each run lasted two hours, including a 30-minute sampling phase; five distinct datasets were produced per environment.
- RQ1: Which offline RL algorithm is most effective for training challenger agents to identify optimality bugs?: The evaluation compared Number of Bugs, Number of Distinct Bugs, and Distance across BC, BCQ, and CQL challengers.Number of Distinct Bugs counts unique discretized initial-state grid cells containing detected bugs, while Distance measures bug sparseness.
- RQ1: Which offline RL algorithm is most effective for training challenger agents to identify optimality bugs?: BCQ-trained challengers consistently detected the most Number of Bugs and Number of Distinct Bugs in every environment.Results were averaged over five fuzzing executions, with margins of error reported at 95% confidence.
- RQ1: Which offline RL algorithm is most effective for training challenger agents to identify optimality bugs?: CQL challengers could achieve the highest cumulative rewards yet remain less effective than BCQ challengers at detecting optimality bugs.This comparison indicates that reward performance and bug-detection effectiveness do not necessarily coincide under Delta.
- RQ1: Which offline RL algorithm is most effective for training challenger agents to identify optimality bugs?: 2,517.9 issues per environment was the average yield of BCQ-trained challengers, making BCQ the most effective algorithm for Delta’s optimality-bug detection.The paper consequently establishes BCQ-trained challengers as Delta’s default oracle.
RQ2: How effective is Delta at detecting optimality bugs compared to baseline methods?
Delta significantly outperforms MDPFuzz, CureFuzz, and GMT on nearly all optimality-bug detection metrics, identifying 50.2% more optimality bugs on average. Its strongest overall advantage is consistently finding the most total and unique bugs, although GMT achieves higher Distance in CartPole.
- Evaluation setup: All methods used the same BCQ-trained challenger, a two-hour budget, and five repeated runs for average-performance comparison.The repeated runs supported statistical reliability, with results reported as averages.
- Distance: Delta achieved the highest or joint-highest Distance score in four of five environments: MountainCar, Acrobot, Hopper, and Walker2D.Distance measures the spread of discovered bugs.
- Distance: In CartPole, GMT achieved higher Distance than Delta (0.084 vs. 0.071), while Delta found more distinct bugs (81.0 vs. 43.6).GMT’s bugs were farther apart on average, whereas Delta found a denser cluster of unique bugs.
- Overall effectiveness: Delta consistently found the most total and unique bugs, making it the most effective framework overall.Delta did not lead in every individual metric, but it consistently dominated bug-count measures.
- RQ2 Answer: 50.2% more optimality bugs were identified by Delta on average than by the baseline methods.The evaluation compared Delta with MDPFuzz, CureFuzz, and GMT.
RQ3: To what extent does each component of the energy function (i.e., diversity, regret, and inconsistency) contribute to Delta’s optimality bug detection performance?
The ablation study shows that Diversity, Regret, and Inconsistency are all vital to Delta’s optimality-bug detection performance. Regret is the single most crucial component for maximizing the total number of bugs found.
- RQ3: Regret removal causes the largest decline in total bugs found, including a drop from 5450.0 to 2238.4 in Acrobot.This corresponds to an approximately 59% decrease.
- RQ3: Without Regret, Acrobot’s distinct bugs fall from 81.0 to 39.4, more than halving the detected unique bugs.This finding reinforces Regret’s importance beyond total bug quantity.
- RQ3: Removing Inconsistency noticeably decreases both total and distinct bugs across all evaluated environments.The ablation results therefore indicate that Inconsistency contributes materially to Delta’s performance.
- RQ3: All three energy components play synergistic and distinct roles in Delta’s optimality-bug detection.The study identifies Diversity, Regret, and Inconsistency as integral rather than optional components.
RQ4: How useful are the identified optimality bugs to improve the performance of DRL agents?
Delta’s identified optimality bugs are useful for improving the Agent Under Test (AUT): superior challenger trajectories are curated into a dataset and used to fine-tune the AUT. The resulting AUT-Improved agent improves performance and reduces safety failures by up to 100% across environments.
- Improvement Method: Delta curates superior trajectories generated by the challenger into a dataset, Dsuperior, for improving the AUT.The dataset is formed from trajectories associated with identified optimality bugs.
- Improvement Method: Conservative Q-Learning (CQL) fine-tunes the AUT using only static data, producing the AUT-Improved agent.The CQL objective updates the existing policy while keeping fine-tuning stable and robust.
- Evaluation: AUT-Improved is evaluated in the original training environment without modifying transition dynamics, the reward function, or the action space.Mean cumulative reward is the primary comparison metric.
- Results: Up to 100%: independent CureFuzz re-evaluation shows that AUT-Improved reduces safety failures across all environments.The paper characterizes AUT-Improved as substantially more robust than the original AUT.
6 Discussion
Delta extends its optimality oracle to non-deterministic environments through statistical hypothesis testing and remains effective there. The discussion also identifies rigorous safety testing as a precondition and clarifies how challenger results support evaluation and downstream policy improvement.
- Non-deterministic environments: Delta reports an optimality bug when the challenger’s rewards are statistically significantly higher than the AUT’s.The oracle uses a one-sided Mann–Whitney U test with alternative hypothesis H1: R_ca > R_aut and default significance level α = 0.05.
- Limitations and interpretation: Delta should be applied only after developers are satisfied that safety testing has been rigorous enough to produce a diverse dataset D.Insufficient exploration of the AUT’s behavior undermines challenger quality and represents a more serious testing problem than missed optimality issues.
- Limitations and interpretation: A superior challenger exposes optimality issues, whereas failure to outperform a near-optimal AUT confirms its high performance rather than indicating oracle failure.The oracle’s purpose is to assess optimality, so validating strong AUT performance is as valuable as finding defects.
- Repair vs. Replace: Fine-tuning and direct replacement are complementary downstream choices enabled by Delta’s bug detection.When the AUT’s weights are inaccessible, fine-tuning is not feasible and the challenger becomes a candidate for direct deployment.
7 Threats to Validity
The paper addresses threats from randomized testing, limited generalizability, and the granularity used to count optimality bugs. It mitigates randomness through repeated experiments and statistical testing, and complements initial-state bug counts with behavioral-pattern reporting.
- Internal and external validity: Randomized testing threatens internal validity, so each experiment is repeated five times with statistical tests for significance and effect size.The implementations of CureFuzz, MDPFuzz, and GMT reuse their officially released replication packages.
- Internal and external validity: External validity is threatened because the analysis results may not generalize.The passage states that mitigation is discussed but does not provide the remainder of that discussion.
- Construct validity: Optimality bugs are counted by initial state because initial states are concrete, reproducible inputs for debugging and retraining.This granularity does not directly capture the diversity of underlying agent behaviors, so behavioral-pattern results are additionally reported.
8 Related Work · 9 Conclusion and Future Work
Related work spans safety, multi-agent, adversarial, and search-based testing of DRL agents, while Delta combines safety testing with offline-RL differential testing to detect both safety-critical and optimality bugs. Evaluations across five environments demonstrate Delta’s effectiveness and identify BCQ as the strongest challenger approach for finding optimality issues.
- 8 Related Work: MDPFuzz pioneered black-box fuzz testing for deep-learning models addressing Markov Decision Processes, including DRL agents.CureFuzz extends this direction with curiosity-driven scenario novelty to reveal diverse crash-triggering cases.
- 8 Related Work: MASTest targets multi-agent DRL systems by incorporating both individual diversity and team diversity.The supplied passage identifies this as a testing approach for multi-agent systems.
- 8 Related Work: AdvTest evaluates competitive game agents through constraint-guided adversarial agent training.It is presented alongside multi-agent DRL testing work.
- 8 Related Work: STARLA uses a genetic algorithm to find faulty episodes and actions leading to crashes.This is a search-based testing method for DRL agents.
- 8 Related Work: Tappler et al. propose search-based testing for RL agents with stochastic policies using depth-first backtracking search.The supplied passage describes this method as targeting RL-agent testing.
- 9 Conclusion and Future Work: Delta conducts safety testing on the AUT while collecting interaction trajectories, then trains a challenger with Offline Reinforcement Learning for differential comparison.A challenger achieving higher cumulative rewards flags an optimality issue in the AUT.