PaperlayerView exact revision on arXiv

Source-linked AI summary

The Unreliability of Explanations in Few-shot Prompting for Textual Reasoning

Xi Ye, Greg Durrett

arXiv:2205.03401v2cs.CL

TL;DR

The paper asks whether explanations improve few-shot in-context learning for textual reasoning and whether generated explanations reliably support predictions. Across four LLMs and three datasets, explanations substantially help mainly text-davinci-002, while reliability-based calibration improves InstructGPT across all datasets.

  • Problem

    The paper examines whether explanations improve few-shot in-context learning for textual reasoning and reliably explain prompted LLM predictions.

  • Method

    The authors test four LLMs on three QA and NLI datasets, assess explanation factuality and consistency, and train lightweight calibrators from reliability features.

  • Results

    Explanations substantially improve accuracy only for text-davinci-002, while automatically calibrated explanations improve InstructGPT’s in-context performance across all three datasets.

  • Takeaways & Limitations

    Explanation factuality can indicate prediction correctness, enabling post-hoc calibration even when generated explanations are not factually grounded.

  • Takeaways & Limitations

    Because fluent explanations may be ungrounded and convincing, deploying them can deceive users about incorrect model responses.

Abstract

from arXiv · show

Does prompting a large language model (LLM) like GPT-3 with explanations improve in-context learning? We study this question on two NLP tasks that involve reasoning over text, namely question answering and natural language inference. We test the performance of four LLMs on three textual reasoning datasets using prompts that include explanations in multiple different styles. For these tasks, we find that including explanations in the prompts for OPT, GPT-3 (davinci), and InstructGPT (text-davinci-001) only yields small to moderate accuracy improvements over standard few-show learning. However, text-davinci-002 is able to benefit more substantially. We further show that explanations generated by the LLMs may not entail the models' predictions nor be factually grounded in the input, even on simple tasks with extractive explanations. However, these flawed explanations can still be useful as a way to verify LLMs' predictions post-hoc. Through analysis in our three settings, we show that explanations judged by humans to be good--logically consistent with the input and the prediction--more likely cooccur with accurate predictions. Following these observations, we train calibrators using automatically extracted scores that assess the reliability of explanations, allowing us to improve performance post-hoc across all of our datasets.

1 Introduction

The paper investigates whether self-generated explanations improve few-shot in-context learning for textual reasoning and finds that their reliability is limited. It nevertheless uses explanation-reliability features to calibrate predictions and improve their trustworthiness.

  • Introduction: LLM-generated explanations often align with predictions but may not be factually grounded in the input, even on a simple synthetic dataset.The paper evaluates explanation factuality as grounding in the input and consistency as entailment of the final prediction.
  • Introduction: Automatically extracted features that approximate explanation reliability allow calibrators to improve predictions post-hoc by permitting null answers when explanations are unreliable.The paper notes that no automated method perfectly assesses reliability, so calibration uses approximate features.
  • Introduction: Explanations do not always substantially improve few-shot in-context learning on textual reasoning tasks, which comprise question answering and natural language inference across three datasets and four LLMs.The study examines OPT, GPT-3 (davinci), InstructGPT, and text-davinci-002.
  • Introduction: Explanation factuality can indicate prediction correctness, because unreliable explanations correspond to predictions that are less likely to be correct.This relationship motivates using explanation reliability for post-hoc calibration.

2 Does Prompting with Explanations Improve In-Context Learning?

Across three English textual-reasoning datasets, explanations produce small-to-moderate gains for OPT, GPT-3, and InstructGPT, whereas text-davinci-002 benefits substantially across all tasks. The study compares standard few-shot prompting with explain-then-predict and predict-then-explain prompts using four LLMs.

  • 2.1 Datasets: The tasks are reading-comprehension question answering and natural language inference, evaluated on three English-language datasets with test sets of 250 examples each.The datasets are SYNTH, ADVHOTPOT, and E-SNLI; SYNTH provides a controlled multi-hop setting, ADVHOTPOT is adversarially balanced, and E-SNLI classifies entailment, contradiction, or neutrality.
  • 2.2 Baselines: The comparison uses Few-Shot prompting without explanations against explain-then-predict and predict-then-explain paradigms, where explanations are respectively generated before or after the label.Predict-then-explain explanations do not influence labels under greedy inference, although explanations in the prompt still affect predictions.
  • 2.4 Results: Explanations yield small-to-moderate gains for OPT, GPT-3, and InstructGPT, while text-davinci-002 benefits greatly across all three tasks and consistently favors explain-then-predict.For InstructGPT, explain-then-predict raises SYNTH from 54.8 to 58.5 and ADVHOTPOT from 56.8 to 59.4; on E-SNLI, predict-then-explain beats Few-Shot by 2.6.
  • 2.3 Setup: The experiments use roughly the maximum prompt-length shots—16 for SYNTH, 6 for ADVHOTPOT, and 32 for E-SNLI—and four LLMs: OPT, GPT-3, InstructGPT, and text-davinci-002.The study primarily tests InstructGPT because it was the most capable available model and still had substantial room to improve on the target datasets.
  • 2.4 Results: Explain-then-predict typically degrades performance relative to predict-then-explain on SYNTH and E-SNLI, while the latter exceeds Few-Shot on E-SNLI by 2.6.The main exception is SYNTH for InstructGPT, where explain-then-predict improves performance.
  • 2.4 Results: The authors caution against strong scientific claims about text-davinci-002 because its differences from earlier models are undocumented and the model lacks transparency.Comparing GPT-3 and InstructGPT also suggests that moving to instruction-tuned models alone does not explain text-davinci-002’s behavior.
  • 2.4 Results: The gains do not generalize strongly across LLMs, even on the synthetic dataset, unlike prior benefits concentrated on program-like tasks such as integer addition and program execution.The authors attribute the difference to their tasks emphasizing textual reasoning grounded in provided contexts.

3 Can LLMs Generate Factual and Consistent Explanations?

The study evaluates explanation reliability through factuality—grounding in the input—and consistency—entailing the model’s prediction. Despite performance gains from explanations, generated explanations can be unreliable, while factuality and consistency often help identify prediction accuracy.

  • Factuality measures whether an explanation is faithfully grounded in the input without hallucinations, whereas consistency measures whether it entails the prediction.
  • The evaluation uses automatic judgments on SYNTH and manual annotations for InstructGPT explanations on ADVHOTPOT and E-SNLI, with consistency reported only for E-SNLI.Results use explanations and predictions from one training-shot set, and only P-E is reported for E-SNLI because E-P performs substantially worse.
  • Results: Although explanations improve InstructGPT’s performance across three tasks, the generated explanations remain unreliable and can disconnect from the model’s predictions.The section reports this result in Table 2, while E-SNLI consistency is limited because explanations often require external commonsense knowledge.
  • On ADVHOTPOT, factuality matches InstructGPT’s prediction 80.0% of the time, substantially exceeding prediction accuracy, while nonfactual SYNTH explanations usually indicate incorrect predictions across all four LLMs.Accuracy and factuality or consistency are typically correlated, especially factuality; on SYNTH, Accuracy = Factuality across all four LLMs.

4 Calibrating In-Context Learning using Explanations

Explanation-based calibration uses factuality signals from generated explanations to adjust prediction probabilities, improving accuracy and AUC over uncalibrated and probability-only baselines. It remains effective with limited extra data that cannot fit in the in-context prompt.

  • SYNTH calibration: 74.8% accuracy: rejecting nonfactual explanation-answer pairs raises SYNTH performance from 52.4% to 74.8%.The procedure checks the top five InstructGPT candidate answers and retains the first pair whose explanation is factual.
  • Framework: The calibrator applies a linear transformation to prediction probabilities augmented with an explanation factuality factor, unlike classical probability-only calibration.It is trained from additional examples using predicted probabilities, explanation factors, and target probabilities; factuality is approximated through lexical overlap between explanations and inputs.
  • Data-scarce setting: Explanation calibration exploits extra labeled examples that do not fit in the prompt, whereas nearest-neighbor prompting helps with more data but is ineffective in the extreme data-scarce regime.With 128 shots, FEW-SHOT(NN) improves over FEW-SHOT by only 3.3%, and finetuned ROBERTA reaches 54.9%, below GPT-3-based models.
  • Calibration results: 68.5% accuracy: P-E+EXPLCAL is 12% above FEW-SHOT and 5% and 3% above FEW-SHOT+PROBCAL and P-E+PROBCAL, respectively.Explanation-based calibration outperforms probability-only calibration and remains more effective as training data grows, while probability-based calibrators nearly saturate at 96 examples.
  • Results: 68.8 AUC: E-P+EXPLCAL surpasses FEW-SHOT by 7 points and E-P by 4 points, while achieving higher coverage-accuracy than its uncalibrated counterpart.The AUC upper bound is constrained by answer accuracy, and the explanation-based calibrator can work with as few as 32 examples.

5 Related Work

This work builds on research in in-context learning and explanation-based few-shot prompting, but examines free-form explanations for textual reasoning in QA and NLI. It also focuses on the nature of explanations generated by LLMs and distinguishes this setting from prior explanation-use methods.

  • The study extends in-context learning research on wayward behaviors and mitigation strategies by focusing specifically on the use of explanations (Brown et al., 2020; Min et al., 2022; Webson and Pavlick, 2022; Zhao et al., 2021).
  • Unlike concurrent work on explanation-based few-shot learning, especially in symbolic reasoning, this study examines more free-form explanations for textual reasoning in QA and NLI over provided contexts (Nye et al., 2021; Wei et al., 2022; Marasović et al., 2022; Chowdhery et al., 2022; Lampinen et al., 2022; Wiegreffe et al., 2022).It also focuses on the nature of explanations generated by LLMs.
  • Prior work has used textual explanations and highlights to improve models, including pipelines that predict from generated explanations and joint models trained with explanations as additional supervision (Zaidan et al., 2007; Wiegreffe et al., 2021; Zhou and Tan, 2021; Chen et al., 2022; Hancock et al.

6 Discussion & Conclusion

The paper concludes that explanations do not reliably improve in-context learning and may be nonfactual, yet their quality can still help assess prediction correctness. It cautions that convincing but ungrounded explanations pose deployment risks and that lexical overlap is only a weak correctness signal.

  • Caveats and Risks of Explanations from Large Language Models: LLM explanations may not reflect internal reasoning and can be factually ungrounded, despite grammatical and convincing presentation that may deceive users in practice.These findings motivate caution when deploying explanation-generating systems.
  • Caveats and Risks of Explanations from Large Language Models: Explanations that do not improve accuracy can still support calibration, although lexical overlap provides only a weak signal of explanation correctness.Stronger entailment models could assess explanations across tasks without fine-tuning, and a language model could be trained specifically for verification.
  • Conclusion: Across four LLMs, two QA datasets, and one NLI dataset, simply adding explanations does not always improve in-context learning, while nonfactual explanations often accompany wrong predictions.This relationship can be leveraged to assess prediction correctness.

Checklist · A Details of Prompts

The checklist reports that the paper addresses scope, limitations, societal impacts, ethics compliance, reproducibility, asset licensing, and data considerations. Appendix A documents prompt formats for SYNTH, ADVHOTPOT, and E-SNLI, including explanation placement.

  • Checklist: The authors affirm that the abstract and introduction accurately reflect the paper’s contributions and scope, and that limitations are discussed in Section 6.
  • Checklist: The checklist confirms discussion of potential negative societal impacts and compliance with ethics review guidelines.
  • Checklist: For the experiments, the authors report providing reproducibility materials, training details, error bars, and compute-resource information.They use the GPT-3 Instruct-series API, specifically text-davinci-001.
  • Checklist: The checklist states that existing assets are cited and licensed, while the new Synthetic dataset is included in the supplementary material.
  • Checklist: The authors report no discussion of consent, personally identifiable information, or offensive content because these checklist items were marked [No].Crowdsourcing and human-subjects checklist items were marked [N/A], including participant instructions, risks, approvals, and compensation.
  • A Details of Prompts: Appendix A shows prompt examples for SYNTH, ADVHOTPOT, and E-SNLI in Figures 6, 7, and 8, respectively.The prompts follow Brown et al. (2020); explanation-based E-P and P-E approaches insert explanations before or after the input with conjunction words.

B Details of the SYNTH Dataset

The SYNTH dataset is a controlled synthetic multi-hop QA benchmark built from templated two-sentence reasoning chains with randomized entities, verbs, and professions. Its construction prevents reasoning shortcuts while retaining a regular structure that remains difficult for models.

  • Each context contains four reasoning chains, each comprising two templated sentences of the form “A [verb] B. B is [profession].”.
  • The dataset randomizes A and B from 50 names, while selecting verbs and professions from separate pools of 30 each.
  • The design eliminates reasoning shortcuts, yet a ROBERTA model requires roughly 500 examples to reach near-100% test accuracy.

C Details of the ADVHOTPOT Dataset · D Details of Reliability Annotations

The paper constructs a compact ADVHOTPOT benchmark with adversarial contexts and manually validated answer quality, then annotates generated predictions and explanations for reliability. Annotation agreement is high for both answer correctness and explanation factuality.

  • C Details of the ADVHOTPOT Dataset: ADVHOTPOT contexts combine two ground-truth supporting paragraphs with two adversarial paragraphs, while retaining only sentences relevant to answering each question [Yang et al., 2018; Jiang and Bansal, 2019].The preprocessing reduces context length to better fit in-context learning.
  • C Details of the ADVHOTPOT Dataset: The test set contains 250 examples, balanced between 125 cases where few-shot GPT-3 predictions were correct and 125 where they were incorrect.The examples were sampled after few-shot inference on 1,000 cases.
  • C Details of the ADVHOTPOT Dataset: Because F1 can misrepresent answer quality, the authors manually assessed answer correctness, including semantically equivalent answers such as “United States” and “US” [Bulian et al., 2022].This manual assessment addresses cases where exact-answer overlap yields zero F1 despite correctness.
  • C Details of the ADVHOTPOT Dataset: Correctness annotations for 100 overlapping examples achieved high inter-annotator agreement, with Cohen’s Kappa of 0.84.The overlap was between annotations produced by the authors.
  • D Details of Reliability Annotations: For the 250 ADVHOTPOT test examples, the authors manually inspected predictions and explanations generated with one training-shot set and annotated factuality and consistency.These annotations support evaluating explanation reliability.
  • D Details of Reliability Annotations: Factuality annotations for 100 examples under the E-P paradigm achieved Cohen’s Kappa of 0.85 between overlapping author annotations.This indicates high agreement for the explanation-factuality judgments.

E Calibrating P-E on ADVHOTPOT · F Additional Output Examples · G Details of Automatically Assessing Consistency and Factuality on SYNTH

On ADVHOTPOT, explanation-based calibration improves P-E’s prediction quality, while additional examples expose unreliable and nonfactual explanations. On SYNTH, consistency and factuality are assessed by matching generated explanations to structured question templates and context facts.

  • E Calibrating P-E on ADVHOTPOT: 62.6 AUC versus 58.4: P-E+EXPLCAL improves P-E and outperforms FEW-SHOT and FEW-SHOT (NN) on ADVHOTPOT.Table 5 reports explanation-based calibration across data conditions, and the experiment follows the setup from Section 4.4.
  • F Additional Output Examples: On SYNTH, GPT-3 almost always generates nonfactual explanations when answering incorrectly with either P-E or E-P.These failure cases are illustrated in Figure 9.
  • F Additional Output Examples: For ADVHOTPOT, Figure 10 provides examples of unreliable explanations generated using P-E.
  • F Additional Output Examples: Using E-P on E-SNLI, GPT-3 sometimes ignores premise information when explaining predictions, especially in examples shown in the figure’s bottom section.Figure 11 contrasts completions for examples with different ground-truth labels.
  • G Details of Automatically Assessing Consistency and Factuality on SYNTH: On SYNTH, regular expressions extract variables from well-formed question, explanation, and answer templates for automatic assessment.Questions use “Who V1 P1?”, explanations use “N1 is P2 and N2 V2 N3,” and answers are names such as N4.
  • G Details of Automatically Assessing Consistency and Factuality on SYNTH: An explanation is consistent when N2 = N4, N1 = N3, P2 = P1, and V2 = V1, ensuring it matches the question and entails the answer.
  • G Details of Automatically Assessing Consistency and Factuality on SYNTH: An explanation is factual only when both “N1 is P2” and “N2 V2 N3” appear exactly in the context.

H Results of Using Explanations in an Alternative Style on SYNTH · I Results of Adding “Step by Step” Trigger in Prompts

Reversing explanation sentence order on SYNTH generally worsened performance, while adding “Let’s think step by step.” did not significantly improve E-P and often degraded it. The reversed style nevertheless produced highly consistent explanations whose factuality usually tracked prediction correctness.

  • H Results of Using Explanations in an Alternative Style on SYNTH: Reversed explanations generally underperformed the original style, with no improvement from explanations for GPT-3 or InstructGPT.The alternative format reverses the two extracted sentences so that the reasoning chain begins with A [verb] B and then states B’s profession.
  • H Results of Using Explanations in an Alternative Style on SYNTH: P-E consistently outperformed E-P under the reversed explanation style for GPT-3, InstructGPT, and text-davinci-002.
  • H Results of Using Explanations in an Alternative Style on SYNTH: Reversed explanations almost always remained consistent when models were prompted in either the E-P or P-E paradigm.
  • H Results of Using Explanations in an Alternative Style on SYNTH: Under the reversed style, explanation factuality almost always indicated whether predictions were correct.
  • H Results of Using an Alternative Style on SYNTH: The two tested prompts were judged the most natural explanation styles, while smaller formatting changes had only minor observed effects.
  • I Results of Adding “Step by Step” Trigger in Prompts: The step-by-step experiment prepended “Let’s think step by step.” to E-P exemplar explanations and targeted the multi-step SYNTH and ADVHOTPOT tasks.It excluded text-davinci-002 because explanations had already substantially improved its performance and omitted OPT because its performance was too low.
  • I Results of Adding “Step by Step” Trigger in Prompts: Adding the step-by-step trigger produced no statistically significant E-P improvement for GPT-3 or InstructGPT and typically degraded performance.

J Information about Cost of Running Experiments

Experiments were costed using August 2022 GPT-3 API pricing and 250-example samples, which the paper reports as sufficient to distinguish approaches based on significance tests and confidence intervals.

  • Cost estimation: The experiments use the largest available GPT-3 models as of August 2022 at $0.06 per 1,000 tokens, with 250 examples per result.Each example uses roughly 1,400 tokens under FEW-SHOT and 2,000 tokens under E-P.
  • Cost estimation: The authors subsample 250-example sets to reduce cost rather than evaluating on full datasets.Significance tests and reported confidence intervals indicate that this sample size can distinguish the approaches’ performance.
Loading 2205.03401v2…