PaperlayerView exact revision on arXiv

Source-linked AI summary

Explaining Reinforcement Learning Decisions in Self-adaptive Systems

Jasmina Gajcin, Juan C. Rosero, Ivana Dusparic

arXiv:2608.14620v1cs.LGcs.AI

TL;DR

RL explanations are often difficult for non-experts to understand, while existing counterfactual tools are limited in scope. EARL unifies counterfactual methods for realistic RL systems; in CitiBikes, dataset-based methods were faster but RL-based methods produced more plausible explanations.

  • Problem

    Existing explainable RL methods often target developers and require technical expertise, despite users needing explanations for trust, verification, and collaboration.

  • Method

    EARL is a unified, extensible Python library that generates counterfactual explanations by exploring what-if scenarios across RL agents and complex environments.

  • Results

    In CitiBikes, dataset-based methods generated counterfactuals faster, whereas RL-based methods produced more plausible and feature-similar counterfactuals.

  • Takeaways & Limitations

    EARL enables counterfactual explanations in smaller-scale and controlled RL scenarios through a unified framework containing dataset-based and RL-based approaches.

  • Takeaways & Limitations

    The evaluation uses quantitative metrics but does not measure explanation usefulness to end users.

Abstract

from arXiv · show

Reinforcement Learning (RL) has been extensively used in autonomous and self-* systems, but RL policies, especially deep RL ones relying on neural networks, lack transparency and are difficult to understand. This can lead to diminished user trust, and makes for a more challenging verification of systems. To address this challenge, this paper introduces Explanations using Alternative Realities for Reinforcement Learning (EARL), a Python library to produce counterfactual explanations in RL settings. This library allows the user to produce explanations by exploring What-if scenarios to clarify agent behavior by comparing possible outcomes. Counterfactual explanations have been shown to be intuitive and user-friendly in psychology research, but have only recently been explored in RL, with existing implementations usually limited to toy examples and benchmarks. EARL supports counterfactual explanation generation in realistic RL-based self-adaptive systems. To demonstrate its applicability, we demonstrate its use in a simulation of CitiBikes, a self-adaptive bike-sharing system, and we provide evaluations showing how it performs in real applications.

1 Introduction

Self-adaptive systems use RL to adapt, but neural-network policies are opaque and difficult to interpret, creating a need for explanations that foster trust and facilitate verification. The paper introduces EARL, a Python library for generating user-friendly counterfactual explanations in complex RL environments.

  • RL enables self-adaptive systems to modify behavior through learned policies, but neural-network state-action mappings are often opaque and difficult to interpret.
  • XRL methods address the need for explanations, yet many target developers through low-level visualizations or global approximations requiring technical expertise.
  • Counterfactual explanations provide high-level, user-friendly “What if” scenarios that show how outcomes would change under different conditions.
  • EARL is a Python library that generates counterfactual RL explanations, supports multiple methods, and lets users explore what-if scenarios in complex environments.

2 Related Work

Related work establishes RL as suitable for self-adaptive systems but highlights the interpretability challenges of RL decisions, especially with neural networks. It also motivates counterfactual explanations as intuitive alternatives while noting that their use in RL remains limited.

  • RL in Self-Adaptive Systems: RL supports complex, multi-objective policies that adapt to uncertain environments, enabling applications such as data center cooling, drone navigation, and self-adaptive servers.However, difficult-to-interpret decisions limit RL’s applicability to high-risk real-life problems.
  • Explainable RL: Explainability supports verification and debugging, user trust, and potential fairness requirements under legal frameworks such as the EU’s GDPR.XRL methods explain agent decisions and are categorized as global or local approaches.
  • Counterfactual Explanations: Counterfactual explanations clarify why an outcome occurred by presenting an alternative scenario with a different result, making them intuitive and user-friendly.Their causal framing also helps users assign blame.
  • Counterfactual Explanations: Although counterfactual methods are extensively studied in supervised learning, their use in RL remains limited.The related-work discussion organizes the topic across RL applications, explainable RL, and counterfactual explanations.

3 Counterfactual Generation Methods in RL

The section reviews counterfactual generation methods implemented in the artifact, contrasting dataset-based domain transfer with RL-specific searches over reachable executions. RACCER variants generate alternative decisions by modifying action sequences through heuristic search, future exploration, or past reconstruction.

  • The artifact implements an overview of counterfactual generation methods for reinforcement learning.
  • GANterfactual-RL generates counterfactual states from agent-transition datasets by translating states between action-defined domains.It uses a StarGAN-based generator and discriminator, but does not model the environment’s sequential or stochastic nature.
  • RACCER searches an agent’s execution environment for reachable counterfactuals and evaluates candidates using reachability, fidelity, and stochastic uncertainty objectives.The approach requires access to the agent’s execution environment.
  • The heuristic-tree-search variant combines reachability, fidelity, and stochastic uncertainty in a loss function, balancing exploration and exploitation before filtering for the target action.It searches action sequences and retains counterfactual states x′ that lead to the target action a′.
  • RACCER-Advance replaces tree search with NSGA-II and searches future action sequences from the current state for an alternate decision.It supports questions about what could change in the future for a different decision.
  • RACCER-Rewind modifies past state-action sequences and uses NSGA-II to find alternative histories leading to a different outcome.It addresses what needed to be different in the past while optimizing the same three objectives as RACCER-Advance.

4 Library Structure

EARL uses a modular, extensible architecture comprising explanation methods, model wrappers, and evaluation interfaces. These components interact through uniform interfaces to generate and assess counterfactual explanations across RL agents and implementations.

  • Architecture: EARL’s modular architecture comprises explanation methods, model wrappers, and an evaluation interface, with basic Deep Q-learning and PPO implementations.The library is designed for extensibility across multiple models and implementations.
  • Explanation methods: The Explanation Method module exposes GANterfactual, RACCER-HTS, RACCER-Rewind, and RACCER-Advance through a common interface.Each algorithm is independent, allowing users to select strategies without modifying the rest of the library.
  • Model wrappers: Model wrappers provide a common interface for RL agents, enabling explanation methods to retrieve chosen actions and related decision metrics.Wrappers support pre-trained models or EARL’s default agent methods and facilitate interaction with different RL models.
  • Evaluation: The Evaluation interface compares explanation methods through metric-based evaluation and runtime logging, using coverage, generation time, similarity, plausibility, and diversity.Evaluation is designed to assess explanation quality and usefulness independently of the underlying RL algorithm.
  • Learning algorithms: EARL’s default DQN and PPO implementations use Stable Baselines3 and can automatically train models with standard hyperparameters when no pre-trained model is provided.This supports immediate experimentation in a specified environment and compatibility with explanation and evaluation components.

5 Utilizing EARL for Explainable RL-based Bike Sharing System

The section demonstrates EARL on a simulated CitiBikes reinforcement-learning environment, covering the environment design, black-box policy training, and integration steps needed for counterfactual explanations. The implementation standardizes observations and actions while enabling resets to arbitrary intermediate states for what-if analysis.

  • 5.1 RL environment: CitiBikes models a five-station bike-sharing network with location-, time-, and weather-dependent demand that creates station imbalances requiring repositioning.S2 and S5 receive more returns than rentals, while S3 and S4 frequently face shortages.
  • 5.1 RL environment: The environment represents each state with 38 features and uses a three-part multi-discrete action: source station, destination, and transferred bike count.The agent can transfer up to 10 bikes and receives penalties for shortages and bike transfers.
  • 5.2 Black-box policy training: A black-box policy π is trained with PPO to provide a sensible policy for counterfactual explanation, while EARL remains model-agnostic and applicable to any RL policy.The stated goal is explainability rather than state-of-the-art CitiBikes performance.
  • 5.3 EARL integration: The CitiBikes environment applies the same preprocessing pipeline during training and explanation through a custom generate_obs method called after steps and during resets.The method builds a unified state representation from station, decision, and shared features.
  • 5.3 EARL integration: EARL integration enforces a MultiDiscrete action space with source station, target station, and bike count dimensions, and adds set_nonstoch_state to reset arbitrary intermediate states.Resetting the same prior state allows the agent to try different actions during counterfactual generation.

6 Evaluation

EARL is evaluated in the CitiBikes simulation by generating counterfactual explanations for informative states and assessing them across scalability, feature similarity, plausibility, and diversity. The evaluation asks why an action was chosen by considering counterfactual states where the action changes.

  • Evaluation: EARL evaluates four implemented counterfactual-explanation methods on 100 informative CitiBikes states selected from 306 execution states.The selected states show a large difference between the best and worst action.
  • Evaluation: The evaluation seeks to explain why action a was chosen in state x by finding counterfactual state x′ where a different action would be selected.Because the action space is large, it considers all counterfactuals that change action a rather than fixing a target action.
  • Evaluation: EARL is assessed using four key parameters: scalability, feature similarity, plausibility, and diversity.These parameters are part of the evaluation framework used to assess the generated explanations.

1. Scalability:

The scalability evaluation examines how well counterfactual-generation approaches scale to high-dimensional tasks. It measures explanation coverage, pre-training time, and average counterfactual generation time.

  • Scalability: Scalability is assessed by how well the approaches perform in high-dimensional tasks.The evaluation focuses on the approaches’ ability to scale as task dimensionality increases.
  • Scalability: Coverage measures the percentage of input facts for which an explanation was successfully generated.This metric captures the proportion of input facts receiving successful explanations.
  • Scalability: The evaluation also measures pre-training time and average counterfactual generation time.Together, these metrics capture preparation cost and explanation-generation cost.

3. Plausibility:

The plausibility analysis addresses whether explanations capture multiple factors behind an outcome by evaluating the diversity of generated counterfactuals.

  • Plausibility:: The method generates multiple counterfactuals that reflect different factors causing a given outcome.This targets diversity within the solution set for each factual query.
  • Plausibility:: Diversity is measured by counting generated counterfactuals per factual query and calculating pair-wise feature distance within each solution set.These measures assess both the number and feature-level separation of alternatives.

4. Diversity:

In CitiBikes, all methods generate one counterfactual per factual state, but RACCER-Rewind produces greater feature diversity while other methods yield nearly identical explanations. The approaches also differ in plausibility, sparsity, and scalability trade-offs.

  • Scalability: All approaches achieve a 99% success rate in generating a counterfactual for each factual query, but generation speed reflects a pre-training-versus-runtime trade-off.GANterfactual-RL requires around 16h of pre-training but generates counterfactuals in under a millisecond; RL-based methods require no pre-training and traverse the agent’s execution path.
  • Proximity and sparsity: RACCER performs slightly better on proximity and achieves the best sparsity, followed by RACCER-Advance and RACCER-Rewind; GANterfactual-RL has a sparsity of 0.6879.A sparsity of 0.6879 indicates that the counterfactual and original instance differ in nearly 68% of their features.
  • Plausibility: RL-based methods produce only plausible instances, whereas GANterfactual-RL produces plausible counterfactuals in 82% of cases.The difference arises because RL-based methods search the environment’s state space, while GANterfactual-RL generates states without considering potential causal relationships between features.
  • Diversity: RACCER-Rewind achieves higher feature diversity, while the other algorithms have diversity scores close to zero.This suggests RACCER-Rewind explores a wider range of alternative scenarios, whereas the other methods produce very similar explanations.
  • Diversity: All approaches generate only one counterfactual per factual state.

7 Conclusions and Future Work

The paper presents and evaluates a unified Python library for counterfactual explanations in a real-world CitiBikes scenario. It identifies plausibility, scalability, and end-user usefulness as key challenges guiding future work.

  • Conclusions: The library applies four counterfactual generation methods in a CitiBikes environment and compares them using four evaluation metrics.Dataset-based methods generate counterfactuals quickly but require extensive pre-training, whereas RL-based methods require substantially more generation time.
  • Conclusions: Dataset-based methods primarily struggle with plausibility, while RL-based methods face scalability challenges in real-world domains.The library provides the first unified implementation of both approaches for smaller-scale scenarios and controlled environments.
  • Future Work: Future work should add plausibility constraints to dataset-based methods and investigate optimization strategies and state space reduction for more scalable RL approaches.These directions are intended to address the main challenges identified for the two method families.
  • Future Work: The evaluation uses quantitative metrics but does not measure explanation usefulness to end users, motivating future user studies.Such studies should assess whether counterfactual explanations are useful for supporting users’ understanding of model behavior.
Loading 2608.14620v1…