Source-linked AI summary
Automated Rationale Generation: A Technique for Explainable AI and its Effects on Human Perceptions
Upol Ehsan, Pradyumna Tambwekar, Larry Chan, Brent Harrison, Mark Riedl
TL;DR
Sequential environments make human-understandable explanation difficult because decisions are temporally connected and shaped by multiple complex factors. This paper introduces automated rationale generation, trains rationale generators from human explanation data in Frogger, and finds that intended rationale-style differences align with users’ perceptions.
Problem
Sequential environments pose challenges for human-understandable explanations because temporally connected decision-making and other complex factors shape behavior.
Method
The paper collects human explanations through remote think-aloud protocols and trains neural translation models to generate concise localized or detailed holistic rationales from exposed agent state and action representations in Frogger.
Results
The intended differences between rationale types aligned with users’ perceived differences, while evaluation established their plausibility and identified components influencing perceptions and preferences.
Takeaways & Limitations
Automated rationale generation can help autonomous agents communicate the motivations for their actions in a human-centered and understandable manner.
Takeaways & Limitations
The current system lacks interactivity, so users cannot contest a rationale or ask the agent to explain differently.
Abstract
from arXiv · showhide
Automated rationale generation is an approach for real-time explanation generation whereby a computational model learns to translate an autonomous agent's internal state and action data representations into natural language. Training on human explanation data can enable agents to learn to generate human-like explanations for their behavior. In this paper, using the context of an agent that plays Frogger, we describe (a) how to collect a corpus of explanations, (b) how to train a neural rationale generator to produce different styles of rationales, and (c) how people perceive these rationales. We conducted two user studies. The first study establishes the plausibility of each type of generated rationale and situates their user perceptions along the dimensions of confidence, humanlike-ness, adequate justification, and understandability. The second study further explores user preferences between the generated rationales with regard to confidence in the autonomous agent, communicating failure and unexpected behavior. Overall, we find alignment between the intended differences in features of the generated rationales and the perceived differences by users. Moreover, context permitting, participants preferred detailed rationales to form a stable mental model of the agent's behavior.
INTRODUCTION
The paper introduces automated rationale generation for sequential environments, where past decisions influence future actions and explanations must connect local decisions to future consequences. It presents human-data collection and neural translation methods for generating distinct rationale styles, then evaluates their perceived quality and differences.
- Motivation: Sequential environments require explanations that connect current actions to future decisions shaped by prior choices.The paper contrasts this with non-sequential tasks whose outputs depend only on current inputs.
- Concept: Automated rationale generation translates an agent’s internal state and action representations into natural-language explanations intended to sound human-like.Rationales are post-hoc explanations that need not reveal the agent’s true decision-making process.
- Contributions: The authors develop a methodology for collecting high-quality human explanation data through remote think-aloud protocols.The collected corpus supplies natural-language training data rather than synthetic explanations.
- Contributions: Neural translation models are configured to generate concise, localized rationales or detailed, holistic rationales.The intended distinction is based on how narrowly or broadly the rationale covers the relevant situation.
- Evaluation: The studies quantify rationale quality and user preferences while examining confidence, human-likeness, adequate justification, and understandability.Qualitative data is used to explain participants’ perceptions and preferences.
LEARNING TO GENERATE RATIONALES
The paper builds a Frogger-based pipeline that pairs game states and actions with players’ spoken rationales, then trains a neural network to generate plausible explanations. The design supports rationale collection, participant editing, and agent-agnostic translation from serialized state-action representations.
- Rationale Generation: A rationale is a human-like explanation that justifies an agent’s action without necessarily revealing its true decision-making process.It provides non-experts with accessible insight into why the agent acted.
- Rationale Generation: The pipeline collects player think-aloud data and trains an encoder-decoder network to generate plausible rationales for agent actions.The two stages link game states and actions to corresponding natural-language explanations.
- Experimental Setting: Frogger provides a simple Markovian sequential environment for testing the rationale-generation pipeline.The technique is agnostic to the agent type or training method when state and action representations can be exposed and serialized.
- Data Collection Interface: During data collection, the game pauses after each action so participants can verbalize the preceding rationale while speech-to-text records it.Participants can adjust the default pause, inspect the transcription, and edit it before continuing.
- Data Collection Interface: The review phase lets players step through action-rationale pairs in situated and global context and edit them without requiring manual post-processing.Organic participant editing allows the collected data to be fed directly into the learning model.
Neural Translation Model
The model translates serialized game-state and action information into natural-language rationales using an encoder-decoder recurrent network. Input configurations are varied to generate distinct rationale styles.
- Neural Translation Model: An encoder-decoder network maps game-state representation X to a word-sequence rationale Y.The architecture is suited to sequential dependencies between inputs and outputs.
- Neural Translation Model: The encoder and decoder use GRU-based recurrent neural networks, with decoder attention weighting input components by their effect on the output.
- Neural Translation Model: The input combines a serialized grid representation with Frogger’s position, most recent action, and remaining lives.
- Neural Translation Model: The rationale generator uses either the agent’s limited observation window or the entire board to support different rationale styles.The limited-window setup is called focused-view; the full-board setup is an alternative configuration.
Focused-view Configuration
The focused-view configuration grounds rationale generation in a local 7 × 7 neighborhood around Frogger, while complete-view uses the entire board and noise to encourage broader contextual reasoning. The user study compares generated rationales with exemplary and random alternatives.
- Focused-view Configuration: Focused-view uses only a 7 × 7 grid window around Frogger to ground explanations in nearby context.The paper states that this local context typically suffices for optimal play and relevant explanations.
- Complete-view Configuration: Complete-view supplies the entire game board, allowing rationales to incorporate potentially influential long-term factors.
- Complete-view Configuration: Complete-view adds 20% dummy-value replacement per grid element to reduce spurious correlations and expose the model to all board sectors.
- User Study: Human judgments compare focused-view or complete-view candidates against exemplary human rationales and randomly selected corpus rationales.The study evaluates whether generated rationales are plausible relative to a low baseline and an exemplary upper bound.
- User Study: Participants’ target dimensions include confidence, human-likeness, adequate justification, and understandability.
Method
The studies combine human explanation collection with perception testing of generated rationales in Frogger. Participants rated candidate, exemplary, and random rationales across four human-factor dimensions.
- Training Data: The training corpus contains over 2000 Frogger action-explanation samples collected from 60 participants.The task took approximately 36 minutes on average, and paired game-state images with explanations trained the network.
- Perception Study: Each video presented one Frogger action with random, exemplary, and candidate rationales for evaluation.The candidate was generated by either the focused-view or complete-view configuration.
- Perception Study: Actions sampled different game regions and action directions, including cars, transitions, and logs.
- Measures: Participants rated confidence, human-likeness, adequate justification, and understandability, then explained their ratings in mandatory free text.
- Measures: The four dimensions were selected through preliminary testing, interviews, literature review, and adaptation of technology-acceptance constructs.
Quantitative Analysis
Quantitative analyses found that rationale style and evaluation dimension affected ratings, while experimental group did not. Both generated-rationale configurations outperformed the random baseline across all evaluated dimensions.
- Quantitative Analysis: Rationale style significantly affected ratings, χ2 (2) = 594.80, p < .001.
- Quantitative Analysis: Evaluation dimension significantly affected ratings, χ2 (2) = 66.86, p < .001.
- Quantitative Analysis: Experimental group had no significant main effect, χ2 (1) = 0.070, p = 0.79.
- Quantitative Analysis: 1.90: focused-view and complete-view rationales were judged better than the random baseline across all dimensions, b = 1.90, t (252) = 8.09, p < .001.
- Quantitative Analysis: Focused-view candidates differed more from exemplary rationales than complete-view candidates did, partly because exemplary rationales received lower ratings alongside complete-view candidates.
- Qualitative Analysis: Open-ended analysis identified five overlapping components: Contextual Accuracy, Intelligibility, Awareness, Relatability, and Strategic Detail.These components influenced multiple perception dimensions rather than mapping one-to-one onto them.
Confidence (1)
Participants’ confidence depended on rationales being contextually accurate, aware of surroundings, strategically detailed, and intelligible. Human-likeness and relatability also shaped perceptions, but intelligibility alone was insufficient for high confidence.
- Confidence increased when rationales recognized environmental conditions, adapted to them, and avoided redundant information.
- Awareness of upcoming dangers and strategic detail indicating long-term planning inspired higher confidence.
- Intelligibility alone did not produce high confidence, while unintelligible or incoherent rationales reduced confidence.
- Major environmental mischaracterizations, including nonexistent objects or incorrect time sequences, produced very low confidence.
- Human-likeness was influenced by intelligibility, relatability, and strategic detail, with errors and planning interpreted inconsistently.
Adequate Justification (3)
Adequate justification depended primarily on contextual accuracy and awareness of the agent’s surroundings. The preference study then directly compared rationale styles, finding significant preference for complete-view rationales across all three dimensions.
- Adequate Justification (3): Rationales with irrelevant details or low contextual accuracy were judged poor justifications.
- Adequate Justification (3): Complete-view rationales were preferred as justifications when they related the action to Frogger’s end goal.
- Adequate Justification (3): Awareness of surroundings increased justification ratings, while random rationales were downgraded for lacking such awareness.
- Adequate Justification (3): Contextual accuracy mattered more than rationale length for understandability, and focused-view rationales could provide better immediate context.
- PREFERENCE STUDY: FOCUSED– VS. COMPLETE–VIEW: The preference study directly compared focused-view and complete-view rationales rather than comparing each style with baselines.
- PREFERENCE STUDY: FOCUSED– VS. COMPLETE–VIEW: Preference for complete-view rationales was significant across all three dimensions, including confidence and responses to agent mistakes.
Most Important Difference (1)
Participants identified the scope of events considered in a rationale as the most important difference between focused-view and complete-view styles. Focused-view rationales were concise and immediate, whereas complete-view rationales incorporated broader context and longer-term planning.
- Participants consistently distinguished the rationales by their level of detail, clarity, and perceived short- versus long-term planning.
- Focused-view rationales emphasized the next move, while complete-view rationales considered the entire picture and broader event scope.
- Participants recognized the intended configuration difference without knowing which network generated each rationale.
- The findings validated that focused-view outputs were succinct and localized, while complete-view outputs were detailed and broader.
- Focused-view rationales addressed immediate actions, whereas complete-view rationales incorporated current circumstances, future moves, and associated dangers.
- These perceived differences shaped later judgments about confidence and communication preferences for failure and unexpected behavior.
Confidence (2)
Participants were more confident in rationales that conveyed long-term planning, environmental awareness, and humanlike judgment. Conciseness helped when actions were trivial, but broader context generally made complete-view rationales more confidence-inspiring.
- Confidence increased with strategic detail, long-term planning, environmental awareness, and humanlike expressions.
- Complete-view rationales inspired more confidence because participants perceived long-term planning as predictive and intelligent.
- Participants valued advance knowledge of the agent’s plans because it could allow mistakes to be caught earlier.
- Complete-view rationales were perceived as reflecting broader environmental understanding and more closely resembling human judgment.
Failure (3)
For failure communication, participants generally preferred detailed, complete-view rationales because they supported understanding the agent’s reasoning and building a mental model for diagnosing problems. However, focused-view rationales remained useful when step-by-step simplicity improved understandability.
- Failure: Complete-view rationales were preferred for failure because they provided adequate explanatory power, awareness, and relatability for developing an agent mental model.Participants wanted to understand how the robot’s mind might be working so they could effectively fix the issue.
- Failure: Detailed rationales often outweighed reduced readability, with participants accepting grammatical errors when the added details clarified the agent’s reasoning.This preference was strongest when users needed the full reasoning behind the agent’s movements.
- Failure: Simple rationales remained valuable because focusing on one step at a time could make it easier to understand what went wrong.The trade-off was that simplicity could also make the agent appear to have limited understanding capabilities.
- Failure: Participants preferred complete-view rationales because strategic detail helped them understand the steps and reasoning leading to a failure.They associated this detail with better ability to identify where a problem occurred and potentially avoid future mistakes.
- Failure: Failure and unexpected behavior both motivated mental-model building, but participants sought mental models for different underlying reasons.Failure involved fixing or preventing problems, whereas unexpected behavior involved understanding violated expectations.
DESIGN LESSONS AND IMPLICATIONS
The paper argues that rationale design should be context-sensitive: agents can vary input configurations and detail to match task demands, agent personas, and communication goals. It also identifies limitations involving interactivity, continuous environments, and one-time evaluations.
- DESIGN LESSONS AND IMPLICATIONS: Rationale styles should be adapted to context, with input configurations chosen to meet the needs of a task or agent persona.The paper gives a companion agent requiring high relatability for user engagement as one example.
- DESIGN LESSONS AND IMPLICATIONS: Failure and unexpected behavior may require complete-view or hybrid configurations when transparency and strategic detail are important.The appropriate level of detail depends on why information is needed and on the task’s characteristics.
- DESIGN LESSONS AND IMPLICATIONS: The approach lacks interactivity, so users cannot contest a rationale or request another explanation.The authors identify contestability as a future extension.
- DESIGN LESSONS AND IMPLICATIONS: The data-collection pipeline is designed for discrete-action games with natural explanation break points, leaving continuous-time and continuous-action environments unresolved.Future work must determine how to collect explanations without being too intrusive to participants.
- DESIGN LESSONS AND IMPLICATIONS: The study’s conclusions are based on one-time interactions, so future deployment should examine potential novelty effects longitudinally.The authors propose longer-term deployment to better control for novelty effects.
- DESIGN LESSONS AND IMPLICATIONS: The paper’s mixed-methods evaluation found that intended rationale differences aligned with users’ perceived differences and revealed dimensions shaping their preferences.The evaluation established plausibility while also providing contextual understanding of the components influencing perception.