Post Hoc Explanations of Language Models Can Improve Language Models

1 Introduction

• Human-annotated rationales are not scalable because they require substantial human involvement, and they can reduce performance on some sophisticated language-understanding tasks.The authors note that biases in human rationales may not align with performance goals.Source: §1 · p. 2
• AMPLIFY creates natural-language rationales from post hoc attribution scores, which identify input features that influence predictions and provide corrective cues to LLMs.For an error, the framework can highlight keywords according to explanations computed for the instance and its true label.Source: §1 · p. 2
• AMPLIFY computes explanations with a smaller proxy model, then combines selected samples, their true labels, and explanations into a few-shot prompt for the LLM.The proxy approach makes explanation generation more computationally feasible than calculating gradients for very large LLMs.Source: §1 · pp. 2–3
• AMPLIFY improves performance by about 10–25% across a wide range of tasks, including tasks where Chain-of-Thought prompting falls short.The authors also analyze how proxy-model, sample-selection, explanation-method, and rationale-template choices affect performance.Source: §1 · p. 3

In-context Learning

• In-context learning adapts a language model to a new task by providing a few task examples in the prompt, without conventional fine-tuning.Chain-of-Thought prompting adds human-annotated, step-by-step rationales, but is not scalable and performs poorly on some reasoning tasks.Source: §2 · p. 3

Post Hoc Explanations

• Post hoc explanation methods express feature attributions that capture each input feature’s influence on a model prediction.The passage distinguishes perturbation-based methods such as LIME and SHAP from gradient-based methods such as SmoothGrad and Integrated Gradients.Source: §2 · p. 3
• The authors state that the utility of post hoc explanation methods for improving LLM performance had not previously been studied, and present their work as a first attempt.Source: §2 · p. 3

3 Our Framework AMPLIFY

• The overall framework uses the selected samples, their true labels, and their explanations as rationales in a test-time prompt to the LLM.Source: §3 · pp. 3–4

Step (1): Proxy Model Selection

• The proxy model is substantially smaller than the target LLM so post hoc explanation generation is computationally viable.The paper considers using a pre-trained model directly or training a smaller model for the target task; the former requires no additional computation for training.Source: §3 · p. 4

Step (2): Few-shot Sample Selection

• The method selects validation examples misclassified by the LLM and ranks them by Misclassification Confidence Score to find the most egregious errors.The selected input and ground-truth-label pairs are intended to supply corrective signals in the prompt.Source: §3 · p. 4

Step (3): Rationale Generation

• For each selected sample, an explanation method assigns attribution scores to input tokens for the proxy model’s prediction of the provided label, then the method retains the top-k words.Token scores are averaged within each word; the paper experiments with multiple explanation methods.Source: §3 · p. 4

Step (4): Prompt Design for LLMs

• AMPLIFY turns the selected words into a rationale linking them to the ground-truth label, then concatenates input–rationale–label examples into a few-shot prompt.The prompt is combined with the test input for the LLM.Source: §3 · pp. 4–5

4 Experimental Evaluation

• The evaluation uses five Big-Bench-Hard tasks—Snarks, Causal Judgment, Ruin Names, Formal Fallacies, and Salient Translation Error Detection—plus CommonsenseQA and Coin Flip, comparing GPT-3 and GPT-3.5 with Answer-Only and human-rationale CoT baselines.AMPLIFY generates corrective rationales using GPT-2 and BERT as proxy models and tests Vanilla Gradients, Gradient x Input, and contrastive explanations.Source: §4 · p. 5
• The implementation used a five-keyword rationale template and, unless stated otherwise, a GPT-2 proxy fine-tuned for the target task; Gradient x Input was the default explanation method.The keywords were selected by highest attribution scores.Source: §4 · pp. 5–6

4.1 Empirical Analysis

• GPT-3.5 reached 91.6% accuracy on Snarks, 16% above AO and over 20% above CoT, while AMPLIFY also led on the other reported tasks.For Causal Judgment, AMPLIFY achieved 76.3% versus CoT at 63.1% and AO at 57.8%.Source: Table 1 · p. 6, §4.1 · p. 6
• With GPT-2 as proxy, AMPLIFY performance was often close with no target-task fine-tuning (E=0) to performance after 200 epochs (E=200).For GPT-3.5 on Snarks, E=0 reached 88.8%, exceeding the best CoT result reported in Table 1; the authors point to limited task data and GPT-2’s pretrained language capabilities.Source: Table 2 · p. 6, §4.1 · p. 7
• Selecting samples with the highest Misclassification Confidence Score yielded the best performance among the tested strategies; random selection performed worst across datasets.Lowest-MCS results were comparable to highest-MCS results, while faithful-explanation selection underperformed lowest-MCS on most datasets.Source: Table 3 · p. 7, §4.1 · p. 8
• Gradient x Input and its contrastive variant generally performed best, but LLM performance changed little across the tested gradient-based explanation methods.On Snarks, GPT-3.5 accuracy remained around 88.8–91.6% across methods.Source: Table 4 · p. 7, §4.1 · p. 9
• In qualitative examples, AMPLIFY captured details missed by AO and CoT, including the red wire’s role in a short circuit and the relevant body part in a CommonsenseQA question.In the CommonsenseQA example, CoT chose “brains” while AMPLIFY chose “two eyes.”Source: Figure 2 · p. 8, §4.1 · p. 9
• For Causal Judgment, a task-tailored rationale template raised GPT-3 accuracy from 60.5% to 63.1%, while punctuation changes had no observed effect.The comparison was with a task-independent rationale template.Source: §4.1 · p. 9

5 Conclusion

• AMPLIFY replaces human-annotated rationales with automated rationales obtained using post hoc explanation techniques.The framework uses a four-step approach with smaller, open-source models for efficient explanation computation.Source: §5 · p. 9
• AMPLIFY improves performance by 10–25% across diverse tasks and outperforms conventional CoT prompting that relies on human-annotated rationales.Source: §5 · p. 9
• The conclusion identifies post hoc explanation methods as valuable tools for enhancing LLM effectiveness.Source: §5 · p. 9

A.1 Proxy Model Task Performance

• Table 5 compares proxy-model performance on target tasks with and without fine-tuning.Source: Table 5 · p. 13

• GPT-3.5 gives an incorrect answer with both AO and CoT prompting in the CommonsenseQA example.Source: §A.2 · p. 13
• The likely issue is that AO and CoT miss that the question’s context concerns eyes rather than the body overall.Source: §A.2 · p. 13
• AMPLIFY answers correctly and identifies “eyes” as the key clue for its prediction.Source: §A.2 · p. 13

A.3 Hyper-parameter Analysis

• AMPLIFY has no scalability constraints as prompt size s and selected-token count k increase, unlike CoT, which requires more human effort to create chains.The passage defines s as few-shot prompt size and k as the number of most important tokens identified by post hoc explanation.Source: §A.3 · p. 13

A.4 Impact of BERT as Proxy Model on LLM Performance

• With BERT as proxy, fine-tuning generally provides marginal improvements, so it can often be skipped to reduce computational overhead.Source: §A.4 · p. 13

• AMPLIFY does not eliminate broader LLM concerns, including misinformation, social bias, hallucination, or the carbon footprint of heavy LLM use.Source: B Limitations and Broader Impacts · p. 13
• Because AMPLIFY relies on post hoc explanations, it may inherit their robustness, disagreement, and stability issues.Source: B Limitations and Broader Impacts · p. 13

C.1 Larger Proxy Model: GPT-2-Medium

• GPT-2-medium without fine-tuning lets the LLM surpass the CoT baseline, showing proxy-model fine-tuning is not mandatory.Source: §C.1 · p. 14

C.2 Multi-step Problem Solving : GSM8k

• On GSM8k, AMPLIFY outperforms AO but underperforms CoT, consistent with multi-step math reasoning exceeding what post hoc explanations are designed to explain.The authors describe AMPLIFY as an alternative, not a replacement for CoT, on complex-language-understanding tasks where human-annotated chains are especially challenging to obtain.Source: §C.2 · p. 14
• In the CommonsenseQA example, standard prompting and CoT answer incorrectly, while AMPLIFY highlights the eye-related context.Source: Figure 3 · p. 15

C.3 Analysis on Other Big-Bench Hard Tasks

• AMPLIFY achieved similar gains on Disambiguation QA, Word Sorting, and Hyperbaton, with performance reported as the percentage of test samples solved accurately.Table 10 presents the performance comparison for these tasks.Source: §C.3 · p. 15, Table 10 · p. 16
• Word sorting showed only minimal improvement, which the authors attribute to its need for lexical-property understanding rather than linguistic semantics.Source: §C.3 · p. 15
• Table 10 compares performance on Disambiguation QA, Word Sorting, and Hyperbaton using the percentage of test samples solved accurately.Source: Table 10 · p. 16