TED: Teaching AI to Explain its Decisions

1 INTRODUCTION

• Regulatory calls for meaningful information raise the question of how explanations can be understandable and useful to their audience.Selbst and Powles characterize meaningful information as understandable, actionable, and flexible across technical approaches.Source: §1 · p. 1
• Complex machine-learning techniques can be difficult to explain even for experts, despite growing demand for explanations in high-stakes domains.That demand relates to oversight and respect for citizens’ due-process rights.Source: §1 · p. 1
• Rather than expose or probe a model’s inner workings, TED has a model jointly produce a decision and an explanation in a format that can vary with user needs.This contrasts with approaches that explain a model directly, use a simpler proxy, or probe it with related inputs.Source: §1 · p. 1
• TED proposes a framework for producing explanations that match consumers’ complexity and domain needs, with a simple instantiation and two illustrative examples.The paper also identifies open research opportunities for extending the framework.Source: §1 · p. 1

2 CHALLENGES TO PROVIDING AI EXPLANATIONS

• The paper identifies justification, a match to consumer complexity, and a match to the relevant domain as characteristics of meaningful explanations.A domain match uses terms relevant to the explanation’s consumer and domain.Source: §2 · p. 2
• Explanation consumers include decision makers, affected users, regulators, and system builders, whose motivations and tolerance for complexity differ.Affected users such as loan applicants need reasons for outcomes in simple terms without industry jargon or complex formulas.Source: §2 · p. 2
• There is no clear definition of a valid explanation outside logical proof; validity appears subjective to the consumer and circumstances.The diversity of consumers, their sophistication, and their domain knowledge makes a single meaningful explanation difficult to provide without information about the consumer.Source: §2 · p. 2

3 RELATED WORK

• Prior work includes interpreting existing or enhanced models, approximating deployed models with simpler models, and generating rationales from text or image inputs.Examples include logical expressions that match deployed decisions and textual rationales derived from input content.Source: §3 · p. 2
• Model interpretation and simpler proxies can help system builders, but may not meet non-experts’ needs or work when features are not human-comprehensible.The paper describes these approaches as precisely characterizing how a machine-learning decision was made.Source: §3 · p. 2
• Input-derived textual explanations are promising, but their generalization to other domains and usefulness across consumer groups remain unclear.The cited work includes selecting text sufficient to trigger a classification and generating descriptive, discriminative image captions.Source: §3 · p. 2
• TED is compatible with calls for user-centered explanations and more rigor in explainable-AI research.The paper cites Doshi-Velez et al. on societal, moral, and legal expectations, and Miller and colleagues and Doshi-Velez and Kim on user needs and rigor.Source: §3 · p. 3

4 TEACHING EXPLANATIONS

• TED offers a complementary approach that trains a model to produce decisions and explanations together, using explanations provided in the training data.The paper contrasts this with approaches that expose or probe a model’s inner workings.Source: §4 · p. 3

4.1 TED Framework and a Simple Instantiation

• The Cartesian-product instantiation combines each decision Y and explanation E into a new class YE, trains a classifier on that class, then decodes predictions into Y and E.E can take different forms and need not be expressed in terms of input features X.Source: §4.1 · p. 3, Figure 1 · p. 3

4.2 Example

• In the cancer-treatment example, adding explanations splits original treatment classes into explanation-specific classes, creating a two-level hierarchy that could support hierarchical classification.Treatment A is split into A1 and A2, while B becomes B3 and B4 and C becomes C5.Source: §4.2 · pp. 3–4, Figure 2 · p. 4

4.3 Advantages

• TED’s Cartesian-product approach adds encoder and decoder components without requiring a change to the existing classification algorithm.The paper says the approach is independent of the classification algorithm and can work with any supervised classifier.Source: §4.3 · p. 4
• Enumerating training explanations can support consumer auditability and may help surface bias or a need for more training data.The authors say biased decisions may be harder to justify and that instances with the same explanations would be expected to cluster in feature space.Source: §4.3 · p. 4

5 EVALUATION

• The evaluation uses a simple Cartesian-product instantiation with neural nets and random forests on two use cases; synthetic datasets are used because explanation-labeled datasets are not readily available.The authors caution that accuracies may be artificially high due to the data-generation method, and that a definitive answer is beyond the paper’s scope.Source: §5 · p. 4, §5.2 · p. 5
• Loan repayment explanation accuracy was 99.4%, versus 99.2% baseline Y accuracy; derived Y accuracy reached 99.6%, while tic-tac-toe move accuracy reached 97.4%.Table 1 summarizes accuracy for predicting Y and E in both examples.Source: Table 1 · p. 5, §5.2 · p. 5, §5.1 · p. 5

5.1 Tic-Tac-Toe

• Tic-tac-toe explanations followed sequential rules: complete three in a row if possible, block the opponent doing so next turn, create a two-in-a-row threat, or otherwise choose an empty square, preferring center, corners, then middles.Source: §5.1 · p. 5
• The tic-tac-toe classifier predicted the correct explanation 96.3% of the time across 36 possible explanations formed from four rules and nine moves.The paper notes that it rarely predicted the correct move without also predicting the correct rule.Source: §5.1 · p. 5

5.2 Loan Repayment

• Because the loan dataset lacked explanations, researchers generated them from two rules, modified labels where rules disagreed, and assigned eight explanation values.The two rules came from researchers at IBM Research and predicted Y with 72% accuracy as the challenge-winning entry.Source: §5.2 · p. 5

6 EXTENSIONS AND OPEN QUESTIONS

• Alternatives to the Cartesian product include multitask learning or jointly aligned feature embeddings for neighbor-based explanation prediction.The framework assumes training data with explanations and uses them to train a classifier predicting Y and E.Source: §6 · p. 5
• Cartesian-product TED increases the classifier’s class count when explanations are added, potentially stressing effectiveness or training time, though this was not observed in the two examples.The authors suggest extreme-classification techniques as potentially applicable.Source: §6 · p. 5
• Enumerated explanations may proliferate near-duplicates, so domain-specific languages or tooling could add structure; word embeddings may help structure free text.Source: §6 · p. 6
• The same base explanation could be expressed at different sophistication levels for consumers such as first-time borrowers, loan officers, or regulators.The paper describes this as a postprocessing step after the classifier predicts an explanation.Source: §6 · p. 6
• When full training-set explanations are unavailable, possibilities include external knowledge, few-shot learning from a subset, or active learning to guide annotation.The paper also proposes measuring how explanatory-system performance changes as more training explanations are provided.Source: §6 · p. 6

• TED replicates human domain users’ reasoning instead of probing a model’s reasoning process, addressing domain users more directly than model builders.The paper frames this paradigm as complementary to model-introspection approaches.Source: §7 · p. 6
• Social and behavioral science findings indicate that people prefer simpler, more general, and coherent explanations, even over more likely ones.Such explanations may help observers understand and accept explanations, which Miller connects to establishing trust.Source: §7 · p. 6
• Across two examples, explanations were highly accurate with no loss in prediction accuracy.Source: §7 · p. 6