Source-linked AI summary
Rationalization: A Neural Machine Translation Approach to Generating Natural Language Explanations
Upol Ehsan, Brent Harrison, Larry Chan, Mark O. Riedl
TL;DR
The paper addresses how autonomous systems can provide understandable explanations for sequential behavior. It introduces AI rationalization, translating internal state-action representations into natural language with neural machine translation, and evaluates the approach in Frogger. The reported results support accurate, context-appropriate rationalizations and higher human satisfaction than alternative explanations.
Problem
Autonomous systems make complex sequential decisions under uncertainty, creating a need for human-understandable explanations of unexpected or failed behavior.
Method
AI rationalization translates internal state-action representations into natural language using neural machine translation trained on human think-aloud data and state annotations.
Results
Neural machine translation models significantly outperformed random and majority baselines in accuracy and produced rationalizations appropriate to given situations.
Takeaways & Limitations
Rationalizations were more satisfying than alternative explanations and can make autonomous systems more relatable and human-like when their internal processes are non-intuitive.
Takeaways & Limitations
Satisfactory rationalizations require balancing adequate detail against information overload, and the effects of inaccurate rationalizations on confidence and rapport remain future questions.
Abstract
from arXiv · showhide
We introduce AI rationalization, an approach for generating explanations of autonomous system behavior as if a human had performed the behavior. We describe a rationalization technique that uses neural machine translation to translate internal state-action representations of an autonomous agent into natural language. We evaluate our technique in the Frogger game environment, training an autonomous game playing agent to rationalize its action choices using natural language. A natural language training corpus is collected from human players thinking out loud as they play the game. We motivate the use of rationalization as an approach to explanation generation and show the results of two experiments evaluating the effectiveness of rationalization. Results of these evaluations show that neural machine translation is able to accurately generate rationalizations that describe agent behavior, and that rationalizations are more satisfying to humans than other alternative methods of explanation.
Introduction
The paper introduces AI rationalization, generating human-like natural-language explanations for autonomous behavior by translating state-action representations. It evaluates this approach in Frogger and examines both rationalization accuracy and human satisfaction.
- AI rationalization produces explanations of agent behavior as if a human had performed the behavior.
- The proposed technique translates internal state-action representations into natural language using an encoder-decoder neural network.Training uses natural-language utterances collected from people performing the learning task.
- The Frogger evaluation measures whether generated rationalizations accurately describe the current environment context.Frogger is used because its learned behavior may be difficult for non-experts to understand.
- The paper reports experiments using semi-synthetic data to assess the translation technique’s accuracy.
- The paper analyzes how rationalization types affect human satisfaction to inform explainable-agent design.
Background and Related Work
The paper distinguishes natural-language explanation for sequential decision-making from interpretability, which supports inspection of algorithms or representations. It positions rationalization as a novel explainability approach for stochastic sequential domains.
- Prior explainable-AI work includes simulation-log transformation, tutoring systems, plan verbalization, and multiagent policy translation.
- This work differs by allowing generated rationalizations that need not truly represent the algorithm’s decision-making process.It applies explainable-AI techniques to sequential decision-making in stochastic domains.
AI Rationalization
AI rationalization treats explanation generation as translation from internal state-action information to human-like language. The workflow pairs states and actions with human utterances and trains an attention-based encoder-decoder network.
- Rationalization seeks human-like justifications rather than necessarily accurate accounts of the underlying decision process.
- The approach first creates a corpus of natural-language and state-action pairs, then trains an encoder-decoder network to translate them.
- The corpus records visited states, performed actions, and think-aloud utterances associated with critical states and actions.
- State and action representations can use any consistently string-convertible structure, without requiring human operators to interpret that structure.
- The encoder-decoder maps an input sequence X to an output sequence Y through an RNN encoder, context vector, RNN decoder, and attention mechanism.
Experiments
The experiments test whether encoder-decoder networks generate context-fitting rationalizations and whether humans prefer them to other explanations. Both hypotheses are evaluated in the Frogger arcade-game environment.
- The experiments test whether encoder-decoder networks accurately generate rationalizations fitting the learning environment’s situational context.
- The experiments also test whether humans find rationalizations more satisfying than other forms of explainability.
- Two evaluations are conducted in Frogger, chosen for its fast pace, rich state space, and learnability.
Rationalization Generation Study Methodology
The study evaluates neural machine translation for generating Frogger rationalizations from state-action information, using human think-aloud data and grammar-grounded examples across maps with different obstacle densities.
- Corpus Construction: Human players’ think-aloud utterances were collected and linked to gameplay actions, producing 225 action-rationalization trace pairs.Players transcribed their utterances and assigned them to specific actions.
- Training and Test Set Generation: A grammar generated synthetic rationalizations for state-action-state triples, with 20% of examples in each grammar-rule cluster reserved for testing.Each triple contained the initial state, action, and resulting state, including agent coordinates and grid layout.
- Experimental Conditions: The evaluation compared encoder-decoder predictions with random and majority-vote baselines across maps containing 25%, 50%, or 75% obstacles.All maps were designed to remain completable, providing varied environmental conditions for testing.
- Network Training: The encoder-decoder used a parallel state-action and language corpus to translate internal representations into natural-language rationalizations.The network was a two-layer attention-based LSTM encoder-decoder trained for 50 epochs.
- Evaluation Metric: Accuracy was defined as the percentage of predictions matching their associated test example, while outputs with BLEU below 0.7 were discarded.Multiple grammar rules tied at the highest BLEU score counted as a match when one matched the testing sentence’s generating rule.
- Results: The encoder-decoder consistently outperformed random and majority baselines, with statistically significant differences across all three maps.Accuracy was highest on the 75% map, followed by the 25% and 50% maps.
Rationalization Satisfaction Study Methodology
The study compared three agents’ explanation styles in a Frogger-based survival scenario and found that participants preferred rationalizations over action declarations and numerical outputs. Rationalizations were associated with explanatory power, relatability, playfulness, and appropriate detail.
- Study design: The rationalizing robot used neural translation, while comparison agents gave only actions or numerical utility values.Participants encountered the agents anonymously as Robot A, Robot B, and Robot C.
- Study design: Participants ranked explanations from three agents after watching six success and failure videos in a quasirandomized presentation order.The sample comprised 53 adults recruited through Amazon Mechanical Turk, with 91% reporting residence in the United States.
- Satisfaction results: The rationalizing robot received the most first-place ratings, the action-declaring robot the most second-place ratings, and the numerical robot the most third-place ratings.A Friedman test found significant differences among repeated-measures satisfaction ratings (χ² = 45.481, p < 0.001).
- Satisfaction results: All pairwise satisfaction comparisons were significant, with rationalizations preferred over both action declarations and numerical outputs.The reported p-values were 0.0059 for rationalization versus action declaration, and p < 0.001 for the other two comparisons.
- Qualitative analysis: Participants associated satisfactory rationalizations with explanatory power, relatability, ludic quality, and adequate detail.Explanatory power was the most frequently cited reason for the rationalizing robot’s superior placement; participants also described it as relatable and entertaining.
- Qualitative analysis: Satisfaction depended on balancing detail against information overload, since some participants found the rationalizing robot too talkative and preferred simpler expressions.The authors also reported that the human-centered communication style supported rapport and accessible explanations for non-experts.
Future Work
Future work will examine how rationalization types affect confidence, perceived intelligence, and tolerance to failure, including how much inaccuracy people accept before confidence and rapport decline.
- Future experiments: Future experiments will test how rationalization types affect confidence, perceived intelligence, and tolerance to failure.The proposed studies would extend the current experimental methodology.
- Future experiments: The authors plan to inject increasing errors into rationalizations to measure their effects on confidence and rapport.This is intended to characterize human preferences under inaccurate rationalizations.
Conclusions
The paper concludes that neural machine translation can produce rationalizations more accurately than baselines and more satisfactorily than alternative explanations. It presents rationalization as a way to make autonomous systems more relatable and human-like when their decisions are non-intuitive.
- Conclusions: Neural machine translation produced rationalizations with accuracies above baselines and greater human satisfaction than alternative explanation methods.These are the paper’s two reported conclusions about rationalization performance and evaluation.
- Conclusions: Rationalization can make autonomous systems more relatable and human-like when their internal decision processes are non-intuitive.The authors connect this potential to future real-world robotic systems in healthcare, accessibility, personal services, and military teamwork.
- Conclusions: The authors envision rationalization as a contribution toward making real-world commercial robotic systems more accessible across several application domains.The stated domains include healthcare, accessibility, personal services, and military teamwork.