Source-linked AI summary
Evaluating RL Explainability Methods by How Much They Help Fix Bugs in Agents
Ram Rachum, Yotam Amitai, Bálint Gyevnár, Reuth Mirsky, Cameron Allen
TL;DR
Existing XRL evaluations do not directly measure how explanations help developers diagnose and repair malfunctioning agents. EvalXRL proposes a closed-loop benchmark using an LLM coding agent and repair performance, but this preliminary paper reports no benchmark results.
Problem
Existing XRL evaluations use functionally grounded metrics and human-grounded proxies rather than directly measuring explanations’ usefulness for diagnosing failures.
Method
EvalXRL has an LLM coding agent iteratively use different XRL methods to diagnose held-out RL malfunctions and repair agents, scoring repair across environment–malfunction–method tuples.
Results
The paper presents a planned benchmark and reports no empirical comparison or repair-score results.
Takeaways & Limitations
EvalXRL is proposed as a common, low-cost measure of downstream explanation utility for RL-agent diagnosis and repair.
Takeaways & Limitations
EvalXRL measures actionability for an LLM coder, so its results may not generalize to human practitioners.
Abstract
from arXiv · showhide
This preliminary paper outlines a planned evaluation benchmark for Explainable Reinforcement Learning (XRL) methods. Current evaluations rely on functionally-grounded metrics like faithfulness and compactness, and on human-grounded proxies like subjective ratings or prediction accuracy. We suggest evaluating XRL methods by how effectively their generated explanations help to diagnose and fix malfunctioning reinforcement learning (RL) agents. We propose EvalXRL, a benchmark in which a Large Language Model (LLM) coding agent uses different XRL methods to diagnose a held-out malfunction in an RL agent, and then repair it. Our proposed benchmark iterates across (environment $\times$ malfunction $\times$ XRL method) tuples and uses the reward signal of the RL agents to form a final score for each XRL method. The coding agent may use the method interactively: invoke the XRL method, process its output, form new hypotheses on what is broken, and invoke the method again with parameters adjusted for testing these hypotheses. This closed-loop structure may be described as a simplified version of the scientific method. Some XRL methods provide self-evaluations that follow this pattern; we propose the first head-to-head comparison of multiple XRL methods in closed-loop usage.
1 Introduction
The paper proposes evaluating XRL explanations by whether they help developers diagnose and repair broken RL agents. EvalXRL operationalizes this idea through continuous reward-based scoring, interactive closed-loop method use, and an LLM coding agent, while remaining a proposed rather than validated benchmark.
- Evaluation paradigm: The proposed evaluation paradigm defines understanding a mechanism as being able to fix it when it breaks, focusing on developers’ need to diagnose failing RL agents.The authors describe this functional view as partial but measurable and use “debugging” and “diagnosis” synonymously.
- Scoring: EvalXRL connects an RL agent’s reward signal directly to evaluation output, defining method success by the performance of an agent repaired using that method’s information.The authors argue that this unmediated connection may reduce proxy-driven bias in scores.
- EvalXRL benchmark: EvalXRL evaluates XRL methods by giving a developer a deliberately broken RL agent, asking for repair, and measuring the repaired agent’s continuous reward or held-out score.The benchmark holds controlled malfunctions fixed across methods to support like-for-like comparisons.
- Closed-loop diagnosis: The benchmark permits repeated, parameter-adjusted XRL invocations in a closed loop as developers form and test new hypotheses about what is broken.The paper presents this as a first head-to-head comparison of multiple XRL methods in closed-loop usage.
- Implementation and limitations: The default developer is an LLM coding agent, but its effectiveness in fixing bugs may not accurately transfer to human use, and EvalXRL is presented as a proposed design rather than a validated leaderboard.The authors state that pilot testing of hypotheses H1–H3 is planned for a follow-up paper.
2 Related Work
Prior XRL research has compared methods using prediction, identification, and human-centered evaluation tasks, while repair has received less direct attention. EvalXRL addresses this gap with a head-to-head benchmark in which an LLM coding agent uses XRL explanations in a closed loop to repair deliberately broken RL agents.
- Cross-method XRL comparisons: Gyevnar and Towers (2025) distinguish debugging from teaming, but their debugging metrics are prediction-based and do not include task completion.Their five debugging metrics are next-action, goal, sub-goal, counterfactual policy, and time taken; task completion appears only on the teaming side.
- Cross-method XRL comparisons: Recent cross-method XRL comparisons evaluate objective accuracy, mechanisms, or behavior under distribution shift rather than downstream repair.Examples include HIGHLIGHTS versus reward decomposition, four mechanisms on Pacman, and counterfactual versus critical-state trajectories under distribution shift.
- Repair as the XAI use case: EvalXRL adapts known-bug debugging tests and controlled-deficit agent studies to score XRL methods by downstream repair success.This extends supervised-XAI debugging tests and the Frogger-agent user-study paradigm toward repair-task evaluation.
- Dialogic and hypothesis-driven XAI: EvalXRL follows dialogic, hypothesis-driven explanation work but reverses the roles: an LLM coder drives the closed loop, enabling scalable end-to-end repair scoring.The related lineage frames explanation as abductive reasoning, follow-up questioning, re-explanation, and Popperian falsification.
- Gap: The proposed gap is a head-to-head, application-grounded XRL benchmark using repair-task completion of deliberately broken RL agents and an LLM coding agent as scalable surrogate user.This benchmark targets the combination of method comparison, deliberate malfunctions, repair outcomes, and scalable LLM-based evaluation.
3 Proposed Framework
EvalXRL defines a benchmark in which a coder repairs malfunctioning RL agents using packaged XRL methods, across environments and deliberately injected faults. It compares methods against no-method and cheat-oracle controls using normalized repair performance, baseline lift, and tool-call efficiency.
- Framework and setup: The coder may be an LLM coding agent or human, while each XRL method is packaged as a dossier of papers, code, documentation, and analysis tools.A malfunction is injected as a unified diff into training code or the environment.
- Framework and setup: The Docker harness isolates the coder’s workspace from an immutable evaluation copy and provides coding, scoring, submission, policy-serving, and method-specific tools.Both containers are cut off from the internet.
- Environments and malfunctions: The planned environments include Treasure Grid, Datacenter Cooling, and Traffic Light Control, spanning discrete and continuous control with real-world repair analogues.Treasure Grid uses heterogeneous rewards, while Datacenter Cooling combines continuous observations and actions with energy, safety, and equipment-wear objectives.
- Environments and malfunctions: Candidate faults include reward clipping, reward hacking, myopia, reward imbalance, distributional shift, sensor drift, and problematic traffic-signal behavior.The benchmark targets behavioral consequences, including goal misgeneralization, rather than code inspection alone.
- Controls and XRL methods: Every cell compares substantive XRL methods with a no-method baseline and a cheat oracle, requiring methods to beat the strongest non-explanation baseline under matched affordances.Planned methods include reward decomposition, counterfactual analysis, action explanations, saliency maps, decision-tree extraction, programmatic policies, and behavior summarization.
- Evaluation outcomes: The primary outcome is continuous post-repair task performance normalized to [0, 1], with headline results reported as lift over the same-cell no-method baseline.The un-clipped normalized score is additionally reported, while tool calls provide a secondary efficiency measure and root-cause identification remains qualitative.
4 Hypotheses
The benchmark tests whether method–malfunction interactions have nontrivial structure, whether a cheat oracle provides a high but imperfect empirical ceiling, and whether XRL methods can sometimes reduce repair performance. It also uses the oracle–optimal-repair gap to distinguish diagnosis from engineering difficulty and controls discovery claims across cells at q = 0.05.
- H1 (interaction structure): H1 predicts nontrivial interaction structure: methods and malfunctions will form clusters with similar repair-success profiles across method × malfunction cells.The clusters are not specified in advance; the test concerns whether such structure exists.
- H2 (oracle ceiling): H2 predicts that the cheat oracle will establish a high empirical ceiling without always achieving a perfect score, because implementing fixes may be harder than diagnosing malfunctions.The oracle–optimal-repair gap measures how much difficulty comes from diagnosis versus engineering; exceeding the oracle in a cell would be an interesting finding.
- H3 (methods can hurt): H3 predicts that at least one method–malfunction cell will score significantly below the no-method baseline, tested with a one-tailed paired comparison for bounded outcomes.Because this is an existence-of-discovery claim, the benchmark applies Benjamini–Hochberg false-discovery-rate control at q = 0.05 across cells.
5 Open Design Questions
The section identifies open design questions about EvalXRL’s validity, packaging, scaffolding, malfunction distribution, access model, and explanatory scope. It also proposes extensions involving multiple methods, composed environments, and evaluation beyond reinforcement learning.
- Validity: EvalXRL measures actionable information for an LLM coder rather than a human, motivating a small (N∼12) semi-structured human calibration study.LLMs may interpret structured explanations differently from humans interpreting visualizations; success with one audience need not transfer to the other.
- Interaction: Closed-loop explanation is an open design choice because prior evidence is mixed: HIGHLIGHTS produced better initial hypotheses, while ASQ-IT gained an advantage after iterative verification.Users may also struggle to choose starting parameters when editing inputs iteratively.
- Evaluation setup: Scores measure a method dossier—README, papers, source, and examples—so published results are properties of (method × packaging), not the underlying method alone.Different prose, examples, and visualization helpers can change repair performance without changing the method itself.
- Evaluation setup: Scaffold selection trades measurement cleanliness against coding support: a minimalist shell attributes more variance to the LLM and XRL method, while richer scaffolds add specialized capabilities.The minimalist option aligns with trying simpler tools before more elaborate ones.
- Validity: Synthetic malfunctions make EvalXRL evidence specific to its catalog rather than real-world bugs, motivating future sampling from RL incident reports or library issue trackers.Real deployments may include data-pipeline errors, version drift, RNG seed leaks, and emergent reward-hacking.
- Scope: EvalXRL’s repair score captures only one axis of explanation utility, while methods may also support trust calibration, accountability, mental models, and stakeholder communication.Conversely, repair gains could arise by short-circuiting rather than improving broader explanatory understanding.
- Extensions: Proposed extensions include joint use of methods with Shapley attribution, coupled multi-domain environments, and transfer of the repair-scored framework to malfunctioning open-source LLMs.The LLM extension changes the substrate and method space while retaining the coder, dossier-versus-baseline, repair metric, and two-container harness.
6 Conclusion
The paper proposes EvalXRL as a benchmark for measuring how well XRL explanations help coders diagnose and repair malfunctioning RL agents. It aims to provide a common, low-cost measure of explanations’ downstream utility, complementing expensive human studies that cannot be run at scale.
- Conclusion: EvalXRL evaluates XRL methods by how well they help a coder diagnose and repair RL agents.The benchmark targets explanations’ practical utility for fixing agent malfunctions.
- Conclusion: The benchmark is intended to offer the XRL community a common, low-cost way to measure explanations’ downstream utility.This claim is conditional on the paper’s hypotheses holding.
- Conclusion: EvalXRL is positioned to complement more expensive but more ecologically valid human studies that the field cannot run at scale.
- Conclusion: As a workshop-stage proposal, the paper seeks community feedback on whether its malfunction taxonomy is sufficiently comprehensive.The supplied passage begins this feedback request but is truncated before specifying the full question.