PaperlayerView exact revision on arXiv

Source-linked AI summary

Objective Metrics for Human-Subjects Evaluation in Explainable Reinforcement Learning

Balint Gyevnar, Mark Towers

arXiv:2501.19256v1cs.AIcs.HCcs.RO

TL;DR

XRL evaluations often omit human participants or rely on subjective reports that do not establish whether explanations work in practice. This paper curates objective behavioural metrics for debugging and human-agent teaming, while arguing that subjective measures remain useful alongside them. It also identifies experimental quality and the lack of standardised benchmarks as important constraints on comparison.

  • Problem

    XRL commonly neglects human-centred evaluation and relies on subjective metrics that capture users’ opinions rather than practical explanation effectiveness.

    Source: §1 · p. 2

  • Method

    The paper curates, describes, and compares objective human-behaviour metrics for debugging agent behaviour and supporting human-agent teaming.

    Source: §1 · p. 2, §2 · p. 2

  • Results

    The paper presents metric methodologies for predicting actions, goals, and sub-goals, evaluating counterfactual understanding, and measuring task completion, conflicts, and time in teaming.

    Source: §2.2 · p. 4, §2.3 · p. 6

  • Takeaways & Limitations

    Objective and subjective measures can be used together to assess behavioural effectiveness, user views, and possible relationships between them.

    Source: §3 · p. 6

    Metric effectiveness depends on experimental setup, and the lack of standardised benchmarks prevents comparisons between research even when objective metrics are used.

    Source: §3 · p. 6

Abstract

from arXiv · show

Explanation is a fundamentally human process. Understanding the goal and audience of the explanation is vital, yet existing work on explainable reinforcement learning (XRL) routinely does not consult humans in their evaluations. Even when they do, they routinely resort to subjective metrics, such as confidence or understanding, that can only inform researchers of users' opinions, not their practical effectiveness for a given problem. This paper calls on researchers to use objective human metrics for explanation evaluations based on observable and actionable behaviour to build more reproducible, comparable, and epistemically grounded research. To this end, we curate, describe, and compare several objective evaluation methodologies for applying explanations to debugging agent behaviour and supporting human-agent teaming, illustrating our proposed methods using a novel grid-based environment. We discuss how subjective and objective metrics complement each other to provide holistic validation and how future work needs to utilise standardised benchmarks for testing to enable greater comparisons between research.

1 Explainable Agents

Because explanations are intended for people, XRL evaluations should account for the explanation’s goal and audience rather than relying mainly on technical or subjective measures. The paper argues that observable changes in human behaviour provide a complementary basis for judging explanation effectiveness.

1 Explainable Agents

  • XRL commonly prioritises fidelity and performance while neglecting human-centred evaluation, and subjective quality measures alone are insufficient.Subjective metrics can reflect preferences but may be brittle because of survey design, evaluator demographics, and question wording.Source: §1 · p. 2
  • Actionability is defined as an explanation’s ability to affect or change human behaviour, making it a central target for evaluation.The paper contrasts actionable effectiveness with measures that capture only users’ opinions.Source: §1 · p. 2
  • Objective human metrics should measure behaviour against clearly defined goals for the system, audience, and explanations.This framing is intended to support practically applicable, comparable, epistemically grounded, and more reproducible XRL research.Source: §1 · p. 2
  • The paper curates and compares objective metrics for two explanation goals: debugging agent behaviour and supporting human-agent teaming.It also examines how metric choices affect the explanations that can be generated and evaluated.Source: §1 · p. 2

2 Objective Measures for XRL

The paper illustrates objective XRL evaluation in a mini-world and organizes metrics around debugging and human-agent teaming. These metrics assess whether people can interpret agent behaviour, predict goals or actions, coordinate effectively, and complete tasks efficiently.

2.1 Illustrative Example

  • The mini-world provides a shared grid-based scenario in which the system controls a soldier and, for teaming evaluations, a human can control the villager.The environment includes passable tiles, obstacles, farmers, soldiers, skeletons, and task-specific agent roles.Source: §2.1 · pp. 2–3, Figure 1 · p. 3

2.2 Assessing Explanations for Debugging

  • Next Action Prediction measures whether users can select an agent’s action from alternatives after receiving an explanation.It can be extended to subsequent actions, but long action sequences require many explanations and may offer limited real-world applicability.Source: §2.2 · pp. 3–4
  • Goal Prediction tests whether users can identify an agent’s top-level objective from alternative natural-language goals.It encourages explanations about higher-level behaviour but limits inclusion of the complete goal description because that would reveal the answer.Source: §2.2 · p. 4
  • Sub-Goal Prediction targets intermediate objectives in long episodes, although researchers must annotate states because no objective method identifies the agent’s true sub-goal.Its advantage is exposing short-term beliefs without presupposing the next action or final goal.Source: §2.2 · p. 4
  • Counterfactual Policy evaluations ask users to identify state changes that would produce a different or specified action, using selections or interactive state modification.Their effectiveness for this application is currently unknown, and explanations that reveal the answer can eliminate the intended cognitive requirement.Source: §2.2 · pp. 4–5

2.3 Assessing Explanations in Human-Agent Teaming

  • Teaming effectiveness can be evaluated through task completion, inter-agent conflicts, and time taken during coordinated work.Task completion measures achieved work, conflict metrics count communication failures, duplicated efforts, or blocking, and completion time provides a cognitive-complexity proxy.Source: §2.3 · pp. 5–6
  • In the teaming illustration, explanations coordinate the villager and soldier so their actions support farming, defence, and higher final episode scores.Without coordination, agents may remain together defensively while preventing the villager from farming.Source: §2.3 · pp. 5–6

3 Discussion

The paper frames objective human testing as a foundation for practical, comparable, and reproducible XRL research, while retaining subjective measures for holistic validation. It also identifies experimental quality, environment complexity, and standardized benchmarks as boundaries on what these evaluations can establish.

  • Objective human testing aims to make XRL research more practical, comparable, epistemically grounded, and reproducible across studies.The paper curates metrics for debugging and human-agent teaming while discussing their collection methods and effects on testable explanation mechanisms.Source: §3 · p. 6
  • Subjective questions complement objective metrics by revealing user beliefs and possible correlations with debugging accuracy, team speed, or team accuracy.The paper cautions that users’ beliefs should match explanation performance because incorrect beliefs may mislead users more than having no explanation.Source: §3 · p. 6
  • Objective metrics may vary in effectiveness with experimental setup quality, motivating stronger control of variables and insights from cognitive psychology.The authors argue that cognitive science can help ensure metrics measure the explanation properties they claim to capture.Source: §3 · p. 6
  • A “No Explanation” condition is recommended as a baseline for detecting random or biased user performance and separating descriptions from explanations.The paper also notes that a natural-language description of an agent’s next action can serve as a possible no-explanation comparison.Source: §3 · p. 6
  • Toy and gridworld environments are useful first steps, but evaluations should progress toward complex settings closer to real deployments.The paper connects more realistic environments with testing whether explanation methods scale to complex agent behaviour and decision-making.Source: §3 · p. 6
  • Without standardized benchmarks, even objective metrics cannot guarantee comparisons between research, motivating common evaluations and use cases.Shared benchmarks could improve understanding of explanation mechanisms’ strengths and weaknesses.Source: §3 · p. 6
Loading 2501.19256v1…