PaperlayerView exact revision on arXiv

Source-linked AI summary

When Do Explanations Help In-Context Learning? A Comparative Study of Natural Language Explanation Types and Faithfulness

Mahdi Dhaini, Adam Dejl, Juraj Vladika, Volkan Özer, Barbara Plank, Gjergji Kasneci

arXiv:2608.16627v1cs.CLcs.AI

TL;DR

It is unclear how explanation source and faithfulness-based selection affect the utility of rationales in explanation-augmented ICL. The paper compares these choices across six benchmarks and four instruction-tuned models, finding frequent classification gains but more variable math-reasoning effects and metric-dependent faithfulness benefits. It also identifies substantial disagreement between faithfulness metrics as a limitation for selecting self-NLEs.

  • Problem

    Systematic evidence is limited on how human, self-generated, and externally LLM-generated explanations, particularly their faithfulness, affect downstream utility as behavior-shaping ICL inputs.

  • Method

    The paper compares three NLE sources and random or faithfulness-based selection across six benchmarks and four instruction-tuned models, including rationale-misalignment tests.

  • Results

    On classification-style benchmarks, adding NLEs often improves accuracy over explanation-free few-shot prompting; external LLM-NLEs and available human NLEs are generally more consistent than self-NLEs, while math effects vary more.

  • Takeaways & Limitations

    External LLM-NLEs and human rationales are practical starting points, whereas self-NLE selection requires metric- and task-sensitive evaluation because faithfulness-based choices do not consistently improve utility.

  • Takeaways & Limitations

    The two faithfulness metrics disagree substantially, with an average disagreement rate of approximately 0.5 across datasets and models.

Abstract

from arXiv · show

Natural language explanations (NLEs) are increasingly used as inputs, for example, as few-shot rationales that influence model behavior in in-context learning (ICL). However, it remains unclear how different types of NLEs compare in their effects on downstream model performance in explanation-augmented prompting. Therefore, we provide a comparative evaluation across six benchmarks and four instruction-tuned models, studying how NLE source (human-written when available, self-generated explanations, generated by an external LLM) and NLE selection (random vs faithfulness-based filtering) affect downstream utility of NLEs when used in ICL settings. Our extensive evaluation shows that, on classification-style benchmarks, adding NLEs to few-shot prompts often improves accuracy over few-shot prompting without explanations; among NLE sources, externally generated LLM-NLEs often provide strong downstream utility and remain competitive with human rationales where both are available, whereas self-NLEs are more sensitive to the selection strategy. On math reasoning, the effects are more model- and source-dependent. We further show that faithfulness-based selection of self-NLEs yields small average gains overall, but can improve or reduce performance depending on the metric, task, and model. Different faithfulness metrics can disagree substantially, affecting which self-NLE examples are selected and their downstream predictive utility. Robustness tests with randomly swapped and out-of-distribution rationales indicate partial robustness, suggesting that semantic alignment contributes to performance gains. Overall, our results provide insights for selecting and reporting explanations that influence model behavior in practical prompting pipelines.

1 Introduction

The paper studies when natural-language explanations improve in-context learning, comparing explanation sources, faithfulness-based selection, and rationale alignment across tasks and models. It finds that explanation utility varies by source, metric, task, and model, while semantic alignment matters for performance.

  • The paper addresses limited systematic evidence about how explanation source and faithfulness affect downstream utility when explanations shape model behavior.
  • The evaluation combines error-based sample selection, explanation-quality scoring, prompt construction, and robustness tests with swapped or out-of-distribution rationales.
  • The study compares human, self-generated, and externally LLM-generated explanations as in-context rationales across six benchmarks and four models.
  • Externally generated explanations typically yield the largest gains on classification-style benchmarks, whereas math-reasoning benefits are smaller and more model- and source-dependent.
  • Faithfulness-based selection can help but depends on the metric and dataset, because faithfulness metrics may disagree and alter selected self-explanations and their utility.
  • Misalignment stress tests indicate that semantic alignment between explanations and examples is important.

2 Background and Related Work

Prior work positions NLEs as human-readable explanations used for interpretability, supervision, synthetic rationale generation, and performance improvement. This paper extends that literature by systematically comparing explanation sources and selection strategies as behavior-shaping inputs in ICL.

  • Explainable NLP research has produced datasets containing explanations for labeled samples across diverse tasks, with human rationales traditionally treated as a reference standard.
  • LLM-generated NLEs offer human-readable justifications, but prior work questions whether self-explanations faithfully reflect language-model reasoning processes.
  • Prior studies have investigated whether incorporating human rationales into model inputs can improve language-model performance.
  • The paper presents a comparative study of self-generated, human-annotated, and externally LLM-generated NLEs as in-context rationales across model scales.
  • It further examines how faithfulness measurement and random versus faithfulness-based filtering change which explanations are included and how much they help.

3 Our Approach - Comparative Study Setup

The comparative setup selects few-shot examples, generates explanations from multiple sources, filters rationales using random or faithfulness-based strategies, and constructs explanation-augmented prompts. Additional swaps and cross-domain rationales test robustness to misalignment.

  • 3.1 Few-shot Samples Selection: The framework selects few-shot samples, generates NLEs, resamples rationale-example pairs, and constructs the final ICL prompt.
  • 3.1 Few-shot Samples Selection: Few-shot samples are selected from examples the evaluated model misclassified under zero-shot prompting.
  • 3.2 Natural Language Explanation Generation: The study compares human-annotated, self-generated, and externally LLM-generated explanations as rationales.
  • 3.2 Natural Language Explanation Generation: Self-NLEs are generated by the evaluated model through post-hoc chain-of-thought, with error examples regenerated conditioned on the gold label.
  • 3.3 NLE Selection: Most-faithful selection ranks self-NLEs with fm1 or fm2 and uses the top-n rationale-example pairs, while human and LLM explanations reuse those pairs for comparison.
  • 3.3 NLE Selection: Robustness settings randomly swap rationales or replace them with explanations from another dataset, with out-of-distribution swaps applied only to LLM-NLEs.

4 Experimental Setup

The experiments evaluate four instruction-tuned autoregressive models of varying sizes on six datasets, using zero-shot, few-shot, and zero-shot chain-of-thought baselines. Faithfulness is assessed with two automated metrics, including a continuous composite and a binary counterfactual-consistency measure.

  • 4.2 Datasets: The six-dataset evaluation includes ECQA and e-SNLI plus Big-Bench-Hard tasks such as SNARKS, Causal Judgment, and Boolean Expressions.
  • 4.1 Models: The evaluation uses GPT-4o-mini, Llama-3.1-8B, Llama-3.3-70B, and Mistral-7B-Instruct-v0.3 to compare model sizes and scales.
  • 4.3 Evaluating Self-NLEs Faithfulness: Faithfulness measures whether a self-NLE reflects and is consistent with the model’s answer-generating behavior.
  • 4.3 Evaluating Self-NLEs Faithfulness: fm1 averages QAG, Counterfactual Stability, and Contextual Faithfulness scores, each ranging from 0 to 1, into a final score in the same range.
  • 4.3 Evaluating Self-NLEs Faithfulness: fm2 labels a self-NLE faithful when the model changes its prediction to an intended alternative after a counterfactual explanation edit.
  • 4.4 Baselines: The baselines include zero-shot prompting, few-shot input-output examples without rationales, and zero-shot chain-of-thought prompting.

5 Results and Analysis

Across classification-style benchmarks, explanation-augmented few-shot prompts often improve accuracy, with LLM- and human-generated NLEs generally more reliable than self-NLEs. Faithfulness-based self-NLE selection produces small average gains but varies by task, model, and metric, while mismatched rationales cause degradation yet do not eliminate model accuracy.

  • Comparing NLE types: LLM- and human-NLEs often improve classification accuracy over few-shot prompting without explanations, while self-NLEs help with greater variability.LLM-NLEs are consistently competitive across datasets and often yield the strongest performance.
  • Comparing NLE types: Human-NLEs are slightly stronger than LLM-NLEs on e-snli, whereas LLM-NLEs generally lead on other evaluated classification-style comparisons.The human-rationale advantage is reported specifically for e-snli; on ecqa, the broader LLM-NLE pattern holds.
  • Comparing selection setups: LLM- and human-NLEs usually retain higher and more stable predictive utility than self-NLEs across selection settings.Exceptions include selected gsm8k and cj comparisons where self-NLEs outperform LLM-NLEs.
  • Faithfulness and self-NLE utility: Faithfulness-based selection of self-NLEs yields small average gains, but effects vary across datasets, models, and metrics; lowest-faithfulness selection averages −0.035 and −0.041 for fm1 and fm2.On gsm8k, fm1 selection improves average performance by 0.073, driven largely by a +0.350 gain for Llama-70B.
  • Robustness to misaligned rationales: OOD rationale swaps consistently reduce performance, while in-domain random swaps have mixed effects and can leave models partially robust.Models retain non-trivial accuracy under OOD rationales, despite consistent harm from semantic mismatch.

6 Discussion

The discussion finds that explanation-augmented ICL is broadly useful but highly dependent on explanation source, selection strategy, task, and model. Faithfulness filtering can help, yet metric disagreement and rationale misalignment complicate reliable deployment.

  • Which NLE to Use?: External LLM-generated and human-written NLEs generally provide stronger and more stable utility than self-NLEs, while self-NLEs are more selection-sensitive.Human rationales remain advantageous for fine-grained semantic alignment on e-snli, whereas external LLM explanations are broadly competitive.
  • Disagreement Between NLE Faithfulness Metrics: Approximately 0.5 average disagreement between faithfulness metrics means that about half of NLEs receive conflicting faithful-versus-unfaithful labels.The disagreement direction is dataset-dependent but consistent across models, indicating that the metrics capture different operational aspects of faithfulness.
  • Faithfulness-Based NLE Selection Can Help, but Not Always: Faithfulness-based self-NLE selection yields small average gains, but its effects vary across metrics, datasets, and models.On gsm8k, fm1-based high-faithfulness selection improves average performance, while fm2 produces almost no average change and can help one model while harming another.
  • Insights on Robustness and Misalignment: Randomly swapped and out-of-distribution rationales generally reduce performance, although models retain partial robustness to rationale misalignment.Out-of-distribution swaps are consistently harmful across datasets, supporting preference for explanations semantically aligned with each example.
  • NLEs vs. CoT Prompting: NLE-augmented prompting and chain-of-thought prompting are complementary: CoT is particularly effective for multi-step math, while aligned NLE exemplars perform strongly on classification tasks.Structured NLE exemplars mostly outperform CoT in several classification model–dataset settings while consistently improving over standard few-shot prompting.
  • Impact of Model Size on Explanation Sensitivity: Relative explanation-augmented accuracy gains are larger for smaller models, whereas Llama-70B is more robust across strategies with less variance.The findings suggest that external reasoning support is more valuable for smaller models, while well-selected explanations still improve models at different scales.

7 Conclusion

The paper comparatively evaluates NLE sources and faithfulness-based selection for explanation-augmented ICL across benchmarks and instruction-tuned models. It finds that NLEs often improve classification accuracy, while faithfulness selection offers small but variable gains and metric choice can change outcomes.

  • Conclusion: Across six benchmarks and four instruction-tuned models, the study compares self-generated, externally LLM-generated, and available human-written NLEs as few-shot ICL rationales.It examines NLE source and selection strategy as design choices for explanation-augmented prompting.
  • Conclusion: On classification-style benchmarks, adding NLEs often improves accuracy over few-shot prompting without explanations, with external and human NLEs generally more useful than self-NLEs.On gsm8k, NLE utility varies more substantially with model, source, and selection strategy.
  • Conclusion: Faithfulness-based self-NLE selection produces small average gains, but can improve or reduce downstream performance depending on the metric, task, and model.Different faithfulness metrics can produce different selected examples and performance outcomes.
  • Conclusion: Randomly swapped and out-of-distribution rationales indicate partial robustness to rationale misalignment, while semantic alignment remains relevant to explanation utility.The conclusion reports these robustness findings as part of the paper’s evaluation of explanation-augmented ICL.

A Limitations

The study’s conclusions are bounded by fixed computational and empirical choices, limited human-rationale coverage, proxy faithfulness metrics, and small evaluation-subset differences across settings.

  • Empirical and Computational Scope: Four instruction-tuned models, six benchmarks, and a fixed 6-shot setup may limit statistical power and generalizability.The authors identify other model families, larger prompt budgets, and different prompting strategies as boundaries.
  • Empirical and Computational Scope: The OOD rationale analysis covers only LLM-NLEs, while stronger CoT variants such as Auto-CoT remain outside the comparison.The authors leave broader OOD coverage and computationally intensive CoT variants for future work.
  • Human-explanations coverage: Human-versus-LLM comparisons are dataset-specific because only ecqa and e-snli provide human rationales.The authors call for validation across more datasets, domains, and annotators.
  • Faithfulness of NLEs: Faithfulness results concern metric-based self-NLE faithfulness rather than definitive evidence about hidden causal reasoning.The study uses fm1 and fm2, each with stated design constraints, including fm1’s heuristic weighting and fm2’s binary counterfactual dependence.
  • Evaluation-Subset Variation Across Selection Settings: Very small performance differences across selection settings may partly reflect slight changes in the evaluated instances.Different selection strategies can exclude different few-shot examples and thereby alter the remaining evaluation subsets.

B Implementation Details

The experiments use reproducible sampling and repeated runs, and adopt error-sampling because it performed better than success-sampling in their comparison.

  • Reproducibility and evaluation setup: All setups use six few-shot samples, average results over five runs, and fix a random seed for reproducible selection.Model calls otherwise use the default API temperature.
  • Sample selection strategy: Error-sampling is adopted because it outperformed success-sampling in the authors’ experiments.The rationale is that models learn more from previously misclassified examples.

C.1 Datasets

The study evaluates explanation-augmented ICL across six datasets, using disjoint few-shot pools, multiple rationale perturbations, faithfulness procedures, and several explanation sources.

  • Dataset and evaluation construction: Few-shot examples are selected from each dataset’s common pool and excluded from evaluation to prevent prompt–test overlap.The setup uses six selected examples and evaluates the remaining instances.
  • OOD rationale selection: The OOD setting draws LLM-NLEs from mismatched datasets, with dataset-specific source pairings across ecqa, e-snli, snarks, boolean, cj, and gsm8k.Examples include e-snli explanations for ecqa and ecqa explanations for e-snli.
  • Robustness settings: Random-rationale robustness tests start from the fm1-based most-faithful condition and replace rationales while retaining selected examples.The OOD analysis is implemented only for LLM-generated rationales.
  • Faithfulness evaluation: Counterfactual faithfulness prompts edit explanations toward alternative answers, then test whether the edited explanations support those answers.The procedure uses minimal paragraph edits and evaluates answers from the resulting counterfactual explanations.
  • Explanation source comparison: NLE length shows no clear utility relationship: self-NLEs are often longer, while shorter o3-mini and human explanations are often competitive or strongest.The reported pattern is not universal, indicating that length alone does not explain predictive utility.
  • Explanation source comparison: GPT-4o-mini and o3-mini NLEs have broadly comparable random-selection utility, with each explainer achieving higher mean accuracy on three of six benchmarks.The higher-cost o3-mini explainer does not consistently produce more useful NLEs.

L Disagreement Between NLE Faithfulness Metrics

The paper compares two faithfulness metrics through aggregate disagreement reporting and model-specific binary heatmaps, while also situating these comparisons alongside random-selection performance tables.

  • Metric disagreement: Table 9 reports the total count, rate, and direction of disagreements between faithfulness metrics fm1 and fm2 across datasets and models.The table is designed to quantify how often and how the metrics diverge.
  • Metric disagreement: Figures 3–6 encode each ecqa test sample by column and each faithfulness metric by row, using green for True and blue for False.The heatmaps cover 4o-minimodel, Llama-70B, Llama-8B, and Mistral-7B.
  • Random-selection comparison: Figure 7 compares NLE types under random selection using dataset panels, mean accuracy across five runs, and standard-deviation error bars averaged over four models.Baselines are ZS, FS-R, and CoT; NLE conditions include self-, human-, LLM-4o, and LLM-o3 explanations where available.
  • Random-selection comparison: Table 10 organizes error-sample random-selection results by model, reporting self-explanations and two LLM explanation variants.Unavailable human explanations are marked with dashes, and row maxima are bolded.

M.2 Self-NLEs Results Across Setups

This section reports self-NLE results across models, datasets, prompting baselines, and random, faithful, and lowest-faithful selection strategies. Results include mean performance, dataset-level changes relative to random selection, and run-to-run variability.

  • Results setup: Performance is reported for four models across five datasets under eight prompting and explanation-selection conditions.The conditions include few-shot random or faithful baselines, random selection, faithful selection with fm1 or fm2, and lowest-faithful selection with fm1 or fm2.
  • Variability and comparison: The evaluation reports averages over five runs and separately reports standard deviations for each model, dataset, source, and selection strategy.The highest value in each row is identified in the results tables, and statistical significance is assessed in the detailed setup.
  • NLE sources: The tables compare self-explanations with human and LLM-generated explanations under random, faithful, and lowest-faithful selection.Human explanations are unavailable for datasets without human rationales.
  • Selection effects: Dataset-level changes are measured as ∆accuracy relative to random selection for most-faithful and lowest-faithful settings.Absolute accuracy under each alternative setup is shown in parentheses.

N Examples of Faithful-Random and OOD Model Outputs

These examples illustrate explanation-augmented prompts with six few-shot examples, including random rationales generated by the evaluation model or another model. The displayed cases include both correct and incorrect predictions across Boolean, ECQA, and OOD-rationale settings.

  • Random Rationales: Random-rationale prompts place an instruction and six explanation-bearing few-shot examples before the test question.The Boolean examples use explanations generated by 4o-mini, while other examples use o3-mini or the evaluation model itself.
  • Random Rationales: The Boolean prompt example with 4o-mini is correct despite being labeled an error case under the LLM-explanation strategy.Its rationales explain Boolean-operator evaluations and provide final answers.
  • Prompt format: The examples show strict answer templates combining a concise explanation with a single answer choice or Boolean label.Several prompts explicitly require the format “Explanation” followed by “Answer.”
  • Self-explanation: Self-explanation examples use the evaluation model to generate rationales for Boolean and ECQA demonstrations.The captions identify Llama-8B and Llama-70B evaluation settings, with correct predictions shown in the illustrated cases.
  • OOD Rationales: OOD-rationale prompts reuse explanations from another dataset, and the illustrated Boolean and ECQA cases show that predictions can be either wrong or correct.Snarks explanations are used for Boolean, while e-SNLI explanations are used for ECQA.

O Examples of Different NLEs

The examples compare self-generated, human-written, and externally LLM-generated explanations for individual inputs. They reveal differences in explanation detail and reasoning style, alongside occasional disagreement in predicted labels.

  • ECQA and e-SNLI: Figures compare six explanation setups for ECQA and e-SNLI, spanning three self-explanation models, human rationales, and two external LLM generators.The comparison is intended to show differences in style, detail, and reasoning approach.
  • Sarcasm reasoning: The sarcasm examples use irony, contradiction, and the phrase “we all know” to justify selecting statement (a).The explanations differ in length, with some presenting multiple steps and others giving a shorter contrast with statement (b).
  • Snarks: The Snarks comparison likewise presents self- and externally generated explanations without a human explanation in the listed setups.The figure description frames the comparison around model size, strategy, and generation source.
  • Boolean reasoning: The Boolean examples illustrate explanations that evaluate negation and operator precedence step by step before producing a final label.Some explanations explicitly revisit an intermediate mistake or emphasize the order of operations.
Loading 2608.16627v1…