Explanation in Artificial Intelligence: Insights from the Social Sciences

Tim Miller

Source-linked AI summary

Explanation in Artificial Intelligence: Insights from the Social Sciences

Tim Miller

arXiv:1706.07269v3Jun 22, 2017cs.AI

TL;DR

Explainable AI research often focuses on causal attribution, despite evidence that human explanations are contrastive, biased, and social. This paper reviews social-science research on explanation and identifies four findings that should inform explainable AI models.

Problem
Existing explainable AI work often provides causal attribution without necessarily producing explanations that lay users can interpret or that address implicit contrast cases.
Method
The paper reviews research from philosophy, psychology, and cognitive science on how people define, generate, select, present, and evaluate explanations.
Results
The review identifies four major findings: why-questions are contrastive, explanations are biasedly selected, explanations are social, and causal links matter more than probabilities.
Takeaways & Limitations
These four findings should be incorporated into explainable AI models where feasible to improve explanatory agents.
Takeaways & Limitations
The reviewed dialogue models abstract away from cognitive processes and interactions, making assumptions that do not account for all possible interactions.

Abstract

from arXiv · show

There has been a recent resurgence in the area of explainable artificial intelligence as researchers and practitioners seek to make their algorithms more understandable. Much of this research is focused on explicitly explaining decisions or actions to a human observer, and it should not be controversial to say that looking at how humans explain to each other can serve as a useful starting point for explanation in artificial intelligence. However, it is fair to say that most work in explainable artificial intelligence uses only the researchers' intuition of what constitutes a `good' explanation. There exists vast and valuable bodies of research in philosophy, psychology, and cognitive science of how people define, generate, select, evaluate, and present explanations, which argues that people employ certain cognitive biases and social expectations towards the explanation process. This paper argues that the field of explainable artificial intelligence should build on this existing research, and reviews relevant papers from philosophy, cognitive psychology/science, and social psychology, which study these topics. It draws out some important findings, and discusses ways that these can be infused with work on explainable artificial intelligence.

1. Introduction

The paper argues that explainable AI should treat explanation as a contextual human-agent interaction problem and build on mature research from philosophy, psychology, and cognitive science. Its review identifies four findings: explanations are contrastive, selectively biased, more effective when causal than probabilistic, and inherently social.

Motivation: Explainable AI should model human explanations because people apply cognitive biases and social expectations when generating and evaluating them.The paper argues these human patterns can improve interactions with explanatory AI.
Motivation: Most explainable AI research relies on researchers’ intuitions about what makes a ‘good’ explanation rather than established social-science frameworks.The paper warns that experts who understand decision models may not be best positioned to judge explanation usefulness.
Contribution: The review surveyed over 250 social-science publications on explanation and selected a smaller subset for presentation based on currency and relevance.It presents relevant theories, experimental evidence, and ideas for incorporating this work into explainable AI.
Scope: Explainable AI is fundamentally a human-agent interaction problem at the intersection of artificial intelligence, social science, and human-computer interaction.The article therefore focuses on philosophical, social-psychological, and cognitive-science views of explanation and their implications for AI design and interaction.
Major findings: Explanations are contrastive, selectively biased, causal rather than merely probabilistic, and social.People seek explanations for an event instead of a foil, select a few causes from many possibilities, find statistical generalisations unsatisfying without causal grounding, and present explanations relative to the explainee’s beliefs.
Major findings: Together, these findings show that explanations are contextual rather than simply presentations of associations and causes.Explainers select contextually relevant causes and may interact or argue with explainees about the explanation.

2. Philosophical Foundations — What Is Explanation?

The section frames explanation as conveying information about an event’s causal history from someone who possesses it to someone else, while distinguishing causal explanation from causal attribution and interpretability.

What Is Explanation?: Explanation conveys information about an event’s causal history from someone possessing that information to someone else.This definition follows Lewis’s account of explanation as an interpersonal act involving explanatory information.
What Is Explanation?: The foundational discussion defines causal explanation and distinguishes it from causal attribution and interpretability.These concepts are presented as related but distinct foundations for understanding explanation.

2.1. Definitions

The section distinguishes causal attribution, causal explanation, and interpretability, while framing causality through dependence and transference theories. It also defines explanation as cognitive inference plus a selected product and classifies explanatory questions as what, how, and why.

Definitions: Explanation-related terms are often conflated, so the section distinguishes causal attribution from causal explanation and briefly contrasts explanation with interpretability.
Definitions: Causality theories divide into dependence theories, centered on counterfactual states, and transference theories, centered on physical energy transfer.
Definitions: Counterfactual, interventionist, and probabilistic accounts define causes through absent events, controllable changes, or increased event probabilities.
Definitions: Halpern and Pearl’s formal model represents causation with structural causal models containing exogenous and endogenous variables, contexts, and actual, counterfactual, and minimality conditions.
Definitions: Explanation comprises a cognitive abductive process that identifies and selects causes for an event, alongside the resulting explanatory product.
Definitions: Explanatory questions fall into what, how, and why classes, with why-questions described as the most challenging from a reasoning perspective.

2.2. Why People Ask for Explanations

People primarily ask for explanations to facilitate learning, improving their understanding and supporting prediction and control. They also seek meaning and shared understanding, while explanations can serve social goals such as influencing beliefs, emotions, actions, or trust.

Learning and understanding: Explanation primarily facilitates learning by improving models of how events or properties arise for prediction and control.Heider describes explanations as helping people understand someone or something well enough to derive a stable model.
Learning and understanding: Explanations support inference learning by filtering causal beliefs and changing perceived likelihoods of claims.Lombrozo emphasizes that these effects arise from explanations as explanations, not solely from the causal information they reveal.
Social and pragmatic goals: People ask for explanations to find meaning by reconciling contradictions or inconsistencies in their knowledge structures.This is one of Malle’s two reasons for seeking everyday explanations.
Social and pragmatic goals: People also ask for explanations to manage social interaction by creating shared meaning and influencing beliefs, impressions, emotions, or actions.In AI, explanations can create a shared understanding of an agent’s decision between the agent and a human observer.
Social and pragmatic goals: Explanations can serve persuasion, learning, and blame-assignment functions, including generating human trust in an intelligent agent’s decision.The goals of explainer and explainee may differ in some social explanations.

2.3. Contrastive Explanation

The section argues that explanations are fundamentally contrastive: they answer why a target event occurred rather than a counterfactual foil. It emphasizes that understanding and selecting explanations depends on the fact–foil contrast.

2.3. Contrastive Explanation: Contrastive explanations answer “Why P rather than Q?”, where P is the target event and Q is a counterfactual contrast case that did not occur.
2.3. Contrastive Explanation: Most why-questions request contrastive explanations, even when the foil is unstated, because people infer possible foils from language and tone.A single question can admit many possible foils, such as leaving a door closed or opening a window instead.
2.3. Contrastive Explanation: An explanation is good only relative to the explainee’s understood foil, since the same cause may explain both the fact and the foil.“Because she was hot” distinguishes opening the door from leaving it closed, but not from turning on air conditioning.
2.3. Contrastive Explanation: Van Bouwel and Weber distinguish plain-fact questions from P-contrast, O-contrast, and T-contrast questions involving properties within objects, between objects, or across time.Plain fact asks why object a has property P; the three contrastive forms compare properties, objects, or times.
2.3. Contrastive Explanation: The paper presents contrastive explanation as a major finding supported by research on how people select and evaluate explanations using fact–foil contrasts.

2.4. Types and Levels of Explanation

Explanation types depend on the question being asked, including whether it concerns an actual event or hypothetical circumstances. For why-questions, Aristotle’s four causes provide a foundational framework, alongside later models that distinguish explanatory levels and categories.

Question-dependent explanation: Explanation type depends on the question, with actual-event and hypothetical-condition questions requiring different answers.The section focuses on a subset of philosophical work addressing why-questions.
Aristotle’s Four Causes: Aristotle’s Four Causes model identifies material, formal, efficient, and final elements for answering why-questions.These concern what something is made of, its defining properties, its proximal mechanisms, and its goals, respectively.
Aristotle’s Four Causes: A single why-question can receive material, formal, efficient, or final explanations, as illustrated by different answers to why a pen contains ink.The answers respectively invoke containment material, the pen’s category, a person filling it, and its writing purpose.
Related explanatory frameworks: Later frameworks distinguish physical, design, and intention stances, or computational, representational, and hardware levels of understanding.Dennett proposed the three stances, while Marr and Poggio defined three levels for computational problems.
Related explanatory frameworks: Kass and Leake categorize anomaly explanations as intentional, material, or social, with social explanations covering non-intentionally driven human behavior.Intentional and material explanations roughly correspond to Aristotle’s final and material categories, whereas social explanations do not map directly onto the other models.

2.5. Structure of Explanation

The section presents explanation as a structured, multilevel process: scientific explanations connect different levels while accounting for intermediate levels, and social explanations operate across conceptual, psychological, and linguistic layers.

Explanatory models: Earlier explanation models included Hempel and Oppenheim’s logically deductive account and Kelley’s statistical model based on co-variation.The passage notes that subsequent experimental research uncovered problems with these influential models.
Scientific explanation: Overton identifies five categories of properties or objects that can be explained in science, including theories, models, kinds, entities, and data.The categories organize scientific explanation by distinguishing principles, abstractions, universals, instances, and observations.
Scientific explanation: Explanations of phenomena at one level must refer to at least one other level and include all intermediate levels between them.An arthropod’s eight legs is explained through entities, kinds, models, and the underlying theory of arthropods.
Scientific explanation: A theory-data explanation is the most complex structure because it contains the longest chain of relationships between two levels.Figure 4 presents this general structure as the most relationship-intensive form of theory-based explanation.
Social explanation: Malle characterizes social explanation as comprising three layers: a conceptual framework, psychological processes, and language.These layers respectively concern assumptions about behavior and explanation, processes for constructing explanations, and linguistic structures used to give them.

2.6. Explanation and XAI

The section argues that XAI should treat explanation as contrastive, distinguish causal attribution from explanation, and match answers to the level at which questions are posed. Meaningful explanations also require agents to reason about explicit causal and explanatory models alongside their decision mechanisms.

Causal attribution and explanation: Causal attribution—extracting and displaying a causal chain—is not necessarily an explanation, because lay users may be unable to interpret most model chains.The section distinguishes presenting causes from providing an explanation a person can understand.
Contrastive explanation: Explanation is fundamentally contrastive: people request contrasts, while complete explanations impose too much cognitive burden.The section presents contrastive explanation as its central point and explains why it is cognitively preferable.
Contrastive explanation: XAI must handle implicit foils, since interpreting “Why P?” contrastively can require all causes; foils may instead be elicited or inferred from abnormality, gaze, or gestures.Contrastive questioning is an opportunity but also an HCI challenge when users do not state the comparison case.
Levels of explanation: XAI researchers should adopt levels-of-explanation models because answers to why-questions depend on the level at which those questions are posed.These frameworks help analyze which explanatory questions agents may receive, even when agents need not explain at every level.
Explanatory models: Meaningful explanations require agents to reason about their own causal models and use explanatory models alongside decision-making mechanisms.The section recommends abstract symbolic models that record relevant properties and support explanation generation.
Explanatory models: Overton’s model, or a similar framework, provides a necessary foundation for structuring and communicating explanations in AI.Applying such a structure requires explicit explanatory models for the system’s relevant categories, including how knowledge was learned.

3. Social Attribution — How Do People Explain Behaviour?

Social attribution concerns how people perceive and explain others’ behaviour through goals, intentions, traits, situational factors, group agency, and normative expectations rather than through behaviour’s actual causes. This research provides foundations for explainable AI, especially goal-based, collective, and norm-sensitive systems.

Social Attribution: Social attribution explains behaviour through perceived goals and intentions, including proximal intentions that support more distant end goals.Goals are defined as ends to which means contribute, while intentions are short-term goals adopted to achieve those ends.
Social Attribution: People tend to explain others’ behaviour through traits rather than situational factors, partly because beliefs and desires are not directly observable.This correspondence bias makes intentional behaviour especially relevant to social explanation.
Social Attribution: People attribute more intentionality to jointly acting groups than to individuals, because joint action is understood as deliberative and explicitly coordinated.By contrast, aggregate groups receive less intentionality attribution than individuals.
Social Attribution: Norm violations strongly affect intentionality attribution: participants judged harmful side-effects intentional 82% of the time, but helpful side-effects intentional only 23% of the time.Adults also distinguish whether an actor knew the relevant norm, whereas children do not.
Implications for Explainable AI: Social-attribution findings are relevant to AI because people attribute beliefs, desires, and intentions to artificial objects, while groups, norms, and abnormalities shape explanation expectations.Norm violations may suggest immoral machines and are especially likely to prompt requests for explanations because people commonly seek explanations for unusual events.
Implications for Explainable AI: Malle’s three-layer model is presented as the most mature social-attribution framework and a foundation for explaining goal-based deliberative systems.Reason, enabling-factor, and causal-history-of-reasons explanations can support BDI models, planning agents, counterfactuals, and explanations involving traits or optimization criteria.

4. Cognitive Processes — How Do People Select and Evaluate Explanations?

People distinguish among multiple possible causes of an event because context makes some explanations more relevant than others. The section examines causal connection, explanation selection, and explanation evaluation as distinct cognitive processes.

Explanation selection: Context determines which of multiple equally true causes is considered the most relevant explanation.Different observers may identify different causes of the same event, such as negligence, construction defects, or environmental conditions.
Cognitive processes: Mill’s early work treated causal connection and explanation selection as essentially arbitrary while describing cognitive biases that influence which explanations people use.He argued that statistical correlations help identify causes through the Method of Difference.
Cognitive processes: The section surveys three cognitive processes involved in explanation: identifying causes, selecting a subset of causes, and evaluating explanation quality.These processes are termed causal connection, explanation selection, and explanation evaluation, respectively.

4.1. Causal Connection, Explanation Selection, and Evaluation

Malle’s theory separates explanation into information and impression-management processes, further distinguishing information requirements, information access, pragmatic goals, and functional capacities. The section focuses on how explainers infer and select causes, and how explainees evaluate explanations.

Causal Connection, Explanation Selection, and Evaluation: Malle’s theory divides explanation into information processes for devising explanations and impression-management processes for governing explanatory social interaction.The theory further distinguishes the tools used to construct and give explanations from the explainer’s perspective or knowledge.
Causal Connection, Explanation Selection, and Evaluation: The four dimensions are information requirements, information access, pragmatic goals, and functional capacities.Requirements concern what makes an explanation adequate; access concerns available causal information; goals concern the explanation’s purpose; capacities constrain achievable goals.
Causal Connection, Explanation Selection, and Evaluation: People typically infer causes from observations and prior knowledge, then select some causes as explanations according to the explanation’s pragmatic goal.These processes concern the explainer’s information access and pragmatic goals.
Causal Connection, Explanation Selection, and Evaluation: Causal connection comprises abductive reasoning, which infers and tests explanatory hypotheses, and simulation, which uses counterfactuals to derive explanations.The processes overlap but differ in whether reasoning depends on assumptions about observations or simulation of counterfactual cases under known causal rules.
Causal Connection, Explanation Selection, and Evaluation: Explanation evaluation occurs when explainees judge whether an explanation is satisfactory, primarily by whether it enables understanding of the cause.Cognitive biases lead people to prefer certain explanation types over others.

4.2. Causal Connection: Abductive Reasoning

Research on abductive reasoning shows that explanations guide learning, generalization, categorization, and inference. These effects depend on explanatory features, causal modes, contextual relevance, and people’s preference for inherent properties.

Learning and generalization: Explanations constrain property generalization to features shared by source and target objects, linking abductive explanation with learning processes.Without feature-based explanations, greater overall feature similarity increased generalization; with explanations, generalization depended almost entirely on explained features being shared.
Learning and generalization: Explanation-based categorization can improve accuracy, but generalization patterns that support learning may also weaken learning through overgeneralization.Williams et al. compared participants who explained categorizations during training with those who thought aloud; the explanation group was more accurate, although the passage does not report the full result.
Functional and mechanistic explanation: People use different abductive modes for different phenomena: functional explanations fit dependence relations, whereas mechanistic explanations fit physical phenomena.Lombrozo’s experiments manipulated explanatory mode and found that functional interpretations affected counterfactual dependence judgments; the supplied passage truncates the remaining result.
Functional and mechanistic explanation: Explanations alter inference and category membership by changing the importance assigned to features, with mechanistic explanations increasing reliance on causal features and functional explanations producing the corresponding functional pattern.In the flower example, participants were more likely to infer bending from brom compounds after a mechanistic explanation, and the reverse occurred for functional causes.
Properties and categorization: Formal categorization explanations favor inherent k-properties over extrinsic t-properties, reflecting a cognitive bias toward stable, enduring features rather than historical factors.Participants rated k-property explanations better in formal modes, and Cimpian and Salomon argue that people preferentially explain phenomena using features concerning how an object is constituted.

4.3. Causal Connection: Counterfactuals and Mutability

People identify causes by selectively mutating events in counterfactual simulations rather than evaluating every possible past event. Mutability is shaped by abnormality, perspective, temporal proximity, intention, controllability, and social appropriateness.

Causal Connection: Counterfactuals and Mutability: People use heuristics to select events for counterfactual mutation, with abnormality, intention, time, and controllability serving as key criteria.This selective process is presented as systematic rather than arbitrary.
Causal Connection: Counterfactuals and Mutability: Abnormal events are commonly undone when people judge causality, according to Kahneman and Tversky’s simulation heuristic.Their studies used causal-chain vignettes involving a car accident and a fatality.
Causal Connection: Counterfactuals and Mutability: People’s perspective changes which events they undo: empathising with the teenage driver led participants to undo his events more often than Mr. Jones’s.The finding indicates that focus affects counterfactual mutation.
Causal Connection: Counterfactuals and Mutability: People undo more recent events than more temporally distal events when assessing causal chains.Miller and Gunasegaram demonstrated this temporal effect in a study involving teachers selecting exam questions.
Causal Connection: Counterfactuals and Mutability: Intentional, controllable events are overwhelmingly selected for undoing over uncontrollable events, regardless of sequence position or whether they are normal or abnormal.Girotto and colleagues also varied whether deliberative actions were constrained or unconstrained, while McCloy and Byrne examined the social appropriateness of controllable behaviour.

4.4. Explanation Selection

People select only the most relevant causes rather than explaining every cause, because causal chains can be too large to comprehend. Selection is strongly contrastive and shaped by abnormality, temporal proximity, intention, and differences between successful and failed actions.

Cognitive constraints: People select the most relevant causes instead of providing every cause, partly because complete causal chains are often too large to comprehend.Explanation selection therefore reflects cognitive constraints on understanding.
Contrastive explanation: People primarily select explanations by comparing the difference between a fact and its foil, so only causes distinguishing the two cases are explanatory.This contrastive approach treats explanatory relevance as determining which causes are selected and weighted.
Abnormality: Abnormal conditions are preferred over normal background conditions, and experiments support non-statistical measures as valid explanatory foils.For example, faulty seals are preferred to atmospheric oxygen because the former is unusual while the latter is present in every shuttle launch.
Temporal and intentional features: Explanation selection also favors proximal over distal causes and intentional over non-intentional actions, while functional reasons can outrank mechanistic explanations.Hilton et al. describe normal-versus-abnormal and intentional-versus-non-intentional contrasts in causal chains; Lombrozo extends functional preference beyond intentional action.
Action explanations: For failed or obstructed actions, people prefer a single cause, and false preconditions become more important than they are for unobstructed events.The relevant single cause may be a goal or precondition, with preconditions gaining importance when they obstruct the action.
Action explanations: For successful actions, people prefer goals as explanations but assign higher probability to conjoined goals and preconditions; unlikely preconditions can reverse this preference.Unlikely actions are judged harder to control, which contributes to the difference between preferred explanations and judged probabilities.

4.5. Explanation Evaluation

People evaluate explanations through coherence with prior beliefs, simplicity, generality, causal usefulness, and explanatory power rather than probability alone. Evaluation also depends on the explainer’s goal and the mode of explanation used.

Coherence, simplicity, and generality: Explanations are preferred when they cohere with prior beliefs, cite fewer causes, and explain more events.Thagard’s theory treats these properties as foundational conditions for acceptable explanations and aligns with human judgments.
Causality and probability: Statistical relationships alone are unsatisfying because people prefer causal explanations of events.The red-ball example illustrates that a statistical generalisation may be true without providing the desired causal explanation.
Causality and probability: People judge explanation quality through pragmatic usefulness and relevance, so the most probable or true cause is not necessarily best.Truth or likelihood is necessary but insufficient, while conversational expectations and causal usefulness influence evaluation.
Explanatory power: People assign higher value to beliefs that explain observations and lower value to beliefs that can themselves be explained by other observations.This pattern links the value assigned to beliefs with their explanatory power, including probability and personal relevance.
Goals and explanatory modes: The explainer’s goal affects evaluation, particularly because different goals favor different explanatory modes.Experiments varied tasks such as identifying causes or categorising organisms, corresponding to efficient and formal modes.

4.6. Cognitive Processes and XAI

Cognitive research suggests that XAI should select and generate explanations using human-relevant biases, including abnormality, intentionality, controllability, and perspective. It also indicates that simplicity, generality, and coherence should complement likelihood, while interaction can reduce users’ need for explanation.

Implications for XAI: Cognitive-process research therefore offers XAI principles for selecting explanations that align with users’ learning, expectations, causal judgments, and social perspectives.The section presents these cognitive processes as relevant to both explanation generation and interpretable behaviour.
Explanation selection: XAI has studied feature-based and information-gain explanation selection, but not using human cognitive biases to choose among competing causes.Existing approaches mainly select important model features through local explanations or information gain.
Learning and generalisation: Repeated interaction can reduce users’ need for explanation because they build better system models, generalise behaviour, and become less surprised by abnormal events.Explanation supports learning and generalisation, making familiarity a condition for requiring less explanation.
Causal and social cognition: Causal-search heuristics, abnormality, intentionality, functionality, and controllability can help explanatory agents find causes that human explainers expect.Abnormal events can trigger explanation and guide selection, though detecting them may be difficult in some models.
Perspective and counterfactuals: In multi-agent settings, explanation can focus on undoing actions of the agent whose behaviour is being explained, reflecting perspective-dependent counterfactual reasoning.For generated behaviour, agents may also select actions whose future consequences are more constrained and less open to counterfactual reversal.
Explanation evaluation: Likelihood is not everything: simplicity, generality, and coherence are at least equally important criteria for useful explanations and may deserve priority when generating trust.These criteria can prioritise among likely causes or guide action selection, even when a less likely action better serves known goals and has fewer causes.

5. Social Explanation — How Do People Communicate Explanations?

Social explanation treats explaining as an interaction between explainer and explainee governed by communicative expectations, not merely as selecting a causal account. Research suggests explanations should fit the explainee’s mental model, remain relevant and appropriately informative, support argumentation, and often be interactive across media.

Social Explanation: Explanation is an interaction between explainer and explainee governed by rules, extending beyond selecting an explanation.The roles may be played by different agents or by the same person or agent.
Relevance and Mental Models: Good explanations must be relevant to both the question and the explainee’s mental model because humans are model-based rather than proof-based.Structural causal models represent explanations as values that remove possibilities from the explainee’s epistemic state.
Conversational Maxims: Conversational maxims shape explanation: excess information increases cognitive load and dilutes important information, while disabling the maxims eliminates the dilution effect.Tetlock and colleagues found that the dilution effect disappeared when participants were told displayed information was selected at random.
Argumentation: Explanations also function argumentatively, with many statements serving as claim-backings that justify reported causes.Explanations therefore both report why something happened and defend that causal claim.
Interactive Explanation in AI: Interactive explanation models can apply across verbal and visual media, but truly self-explaining agents require interaction, communication maxims, and user models tailored to the explainee.Existing systems and models provide starting points, although research has not adequately explored Theory of Mind or socially interactive explanation for modern machine learning.

6. Conclusions

Explainable AI can benefit from social-science models of how people define, generate, select, present, and evaluate explanations. Applying these findings is not straightforward, but collaboration across relevant disciplines could improve explanatory agents in many cases.

Conclusions: Social-science research offers models for how people define, generate, select, present, and evaluate explanations that can inform explainable AI.The paper reviews relevant findings from social-science research and discusses how they can be used in explainable AI.
Conclusions: Explainable AI should incorporate four findings: why-questions are contrastive, explanations are biased selections, explanations are social, and causal links matter more than probabilities.The author notes that these ideas may not be feasible for every application, but could improve explanatory agents in many cases.
Conclusions: Adapting these models requires refinement, extension, and close collaboration among explainable AI, philosophy, psychology, cognitive science, and human-computer interaction researchers.The author notes that related projects are already underway and have produced impressive results.