PaperlayerView exact revision on arXiv

Source-linked AI summary

A Removal Based Approach to Improve LLM Faithfulness at Test-Time

Qinglan Luo, S M A Nahian, John Guttag, S. Mazdak Abulnaga, Katie Matton

arXiv:2609.04343v1cs.AIcs.CL

TL;DR

Incomplete explanations omit factors that influence model answers, while existing approaches do not fully address this problem. The removal wrapper removes input concepts not credited in an explanation and re-queries the model; across two datasets and multiple model families, it improves faithfulness over prompting baselines.

  • Problem

    Incomplete explanations omit factors that influence answers, and existing approaches do not fully address explanation incompleteness.

    Source: §1 · pp. 1–2

  • Method

    The removal wrapper uses an auxiliary model to remove input concepts not credited in the explanation, then re-queries the target model on the reduced input.

    Source: §1 · pp. 2–3

  • Results

    Across two datasets and multiple model families, the wrapper improves faithfulness under several independent metrics versus standard prompting and prompting that encourages faithfulness.

    Source: §1 · p. 3, §5 · p. 10

  • Takeaways & Limitations

    Because it requires no access to model internals and targets completeness, the wrapper can be applied through APIs and paired with approaches targeting soundness.

    Source: §5 · p. 10

    The removal method addresses incompleteness, not unsoundness; it does not correct explanations that credit concepts with no causal effect on the answer.

    Source: §5 · p. 9

Abstract

from arXiv · show

Large language models (LLMs) are increasingly used for consequential decisions, making their explanations an important tool for auditing model behavior. Unfortunately, these explanations can be unfaithful, failing to reflect the actual reasoning underlying the model's decisions. We consider a setting in which an LLM provides both an answer and an explanation in response to a question. We identify two distinct dimensions of unfaithful explanations: incompleteness, meaning that the explanation omits factors that influence the answer, and unsoundness, meaning that the explanation cites factors that did not influence the model's answer. Existing approaches to improving LLM faithfulness include training-time methods, which require access to model weights and extensive computational resources, and test-time methods that largely focus on addressing unsoundness. We introduce a test-time approach that directly targets incompleteness. We remove from the input the concepts not credited in the model's explanation and re-query the model on the reduced input. This eliminates unmentioned influences while preserving the influence of mentioned concepts. Across two datasets, multiple model families, and two independent faithfulness metrics, our approach improves explanation faithfulness compared to both standard prompting and prompting to encourage faithfulness. Our method is model-agnostic and can be applied at inference time without modifying model parameters, providing a flexible mechanism for reducing hidden influences and improving the reliability and safety of LLM-assisted decision making.

1 Introduction

The paper focuses on incomplete explanations, where decision-influencing factors are omitted, and introduces a test-time removal wrapper to align answers with explanations.

1 Introduction

  • Hidden concepts can silently alter model behavior while bypassing human monitoring, raising concerns for AI safety and reliability.Examples include social stereotypes and prompt hints that influence answers without being credited in explanations.Source: §1 · p. 2
  • Train-time approaches alter model weights and can be infeasible or resource-intensive, while existing test-time methods do not fully address incompleteness.Factored Decomposition and Formal Symbolic Reasoning can still depend on hidden concepts.Source: §1 · p. 2
  • The approach aims to prevent hidden concepts from influencing decisions, rather than requiring those concepts to be revealed in explanations.The goal is to align the model’s answer with its explanation.Source: §1 · p. 2
  • The removal wrapper targets incompleteness by removing input concepts not credited in the explanation, then re-querying the target model.An auxiliary language model identifies uncredited concepts and edits the input.Source: §1 · p. 2
  • Across two benchmarks, two model families, and two independent metrics, the wrapper improves faithfulness over standard prompting and prompting that encourages faithfulness.The paper also reports additional gains when combining the wrapper with existing test-time methods.Source: §1 · p. 3

2 The Removal Wrapper

The removal wrapper edits context-based questions to exclude concepts not credited by the model, then obtains a fresh answer and explanation from the reduced input.

2 The Removal Wrapper

  • Context-based questions contain information decomposable into multiple human-interpretable, semantically distinct concepts.The goal is for the answer to depend only on concepts credited in the explanation.Source: §2 · p. 3
  • The explanation’s credit set comprises input concepts cited as influencing the answer; its complement comprises concepts the explanation implies are irrelevant.A hidden concept influences the answer but is not credited in the explanation.Source: §2 · p. 3
  • The wrapper seeks completeness by eliminating hidden concepts’ causal influence, potentially changing the answer rather than changing the explanation.The method is presented as a way to improve explanation completeness.Source: §2 · pp. 3–4
  • The auxiliary model identifies question concepts, determines which are credited, and removes the uncredited concepts while preserving remaining information.The target model is then queried on the reduced input in a new context, without the original question or first response.Source: §2 · p. 4
  • Completeness is guaranteed only if the auxiliary model identifies concepts and credits correctly and the model’s credit set remains consistent across queries.The required consistency is equality between the original and reduced-input explanation credit sets.Source: §2 · p. 4

3 Experiments: Hint-Augmented QA

On hint-augmented multiple-choice QA, the removal wrapper performs best on all three faithfulness metrics across Qwen and Gemma, outperforming the prompt-only baseline.

3.1 Dataset

  • The benchmark pairs factual multiple-choice questions with hints favoring uniformly random options, providing a controlled setting where the planted source of incompleteness is known.Prior work found that models may select the hint-implied answer without verbalizing use of the hint.Source: §3.1 · p. 4

3.2 Faithfulness Metrics

  • The three measures assess hint naming, correlation between concepts’ causal and explanation-implied effects, and whether explanations improve answer prediction.Causal effect is measured by changing a concept and comparing answer distributions; NSG compares prediction accuracy with the full response versus the answer alone.Source: §3.2 · p. 5

3.3 Experimental setup

  • The experiments use Qwen3.5-9B and Gemma-4-12B-it, with a prompt instructing targets to ignore user opinions as the baseline.For remaining experiments, they use 100 Qwen and 86 Gemma hint-flip questions from the 2,000-question set.Source: §3.3 · p. 5

3.4 Results and Discussion

  • The wrapper removes hints that explanations do not verbalize, making the hint-verbalization score 1 by construction.The table caption reports that the deterministic score was also verified experimentally.Source: §3.4 · p. 5, Table 1 · p. 6
  • The removal wrapper performs best on all three metrics across both target models, outperforming the prompt-only baseline.Table 1 reports the hint-augmented QA results for Base, Prompting, and Wrapper.Source: §3.4 · p. 5, Table 1 · p. 6
  • r rose from 0.274 to 0.873 on Qwen and from 0.335 to 0.924 on Gemma with the wrapper.These are causal concept faithfulness correlations between causal effect and explanation-implied effect.Source: §3.4 · p. 6
  • NSG rose from −0.083 to 0.211 on Qwen and from −0.190 to 0.125 on Gemma with the wrapper.The prompt-only baseline remains negative on Qwen (−0.025), though it helps on Gemma.Source: §3.4 · p. 6

4 Experiments: the BBQ dataset

On BBQ, combined prompting or removal improves faithfulness, with the strongest method varying by target model and metric. Removal reduces hidden concepts most, but that effect depends on consistency across repeated queries.

4.1 Dataset

  • BBQ has 2,592 questions varying one demographic attribute; neutral-label rewriting corrected option/person confusion in counterfactual construction.Questions cover age, gender, race, religion, nationality, disability, or socioeconomic status, and ask about something unstated in the context.Source: §4.1 · p. 6

4.2 Faithfulness Metrics

  • BBQ evaluation uses causal concept faithfulness and NSG; hint verbalization is inapplicable, while Figure 4 plots CE against EE and uses Pearson r.Figure 4 shows higher r after removal, alongside the elimination of hidden concepts in the plot's bottom-right region.Source: §4.2 · p. 6, Figure 4 · p. 7

4.3 Experimental Setup.

  • Evaluation compares five methods on 100 of 2,592 questions, using 30 responses per original question and each concept-edited counterfactual.Target models are Qwen3.5-9B, Qwen3.5-27B, and Gemma-4-12B-it; Qwen3.5-27B is the auxiliary model and NSG predictor. Baselines include Base, faithfulness Prompting, and Modular prompting; Combined pairs removal with modular prompting.Source: §4.3 · pp. 7–8

4.4 Results and Discussion

  • For causal concept faithfulness, Combined scores highest on Qwen3.5-9B and Qwen3.5-27B (r=0.890 and 0.843), while Removal improves Gemma-4-12B from base 0.622 to 0.819.Combined scores 0.801 on Gemma-4-12B, close behind Removal.Source: §4.4 · p. 8
  • Base NSG is 0.006, 0.054, and 0.102; Combined leads on Qwen3.5-9B/27B (0.130, 0.180), while Removal leads Gemma-4-12B (0.134).All NSG values are small and the intervals overlap.Source: §4.4 · p. 8

4.5 Faithfulness Type Analysis

  • For Qwen3.5-27B, removal halves hidden counts (12 to 6); Combined has the fewest unsound concepts (12) and highest faithfulness (r=0.84).Removal follows with r=0.80.Source: §4.5 · pp. 8–9, Figure 6 · p. 9
  • Removal's hidden-concept effectiveness depends on repeated attribution consistency: Qwen3.5-9B's varying removals introduce a similar number of new hidden concepts, leaving the count unchanged.The wrapper removes the hidden concepts it targets, but varying reductions offset that decrease on Qwen3.5-9B.Source: §4.5 · p. 9

5 Limitations & Conclusion

The removal wrapper improves explanation faithfulness across two datasets and multiple model families, but addresses incompleteness rather than unsoundness and depends on reliable concept extraction and independence assumptions.

Limitations

  • The method addresses incomplete explanations, not unsound ones; combining it with approaches such as modular prompting [9] can improve overall faithfulness.It does not correct explanations that credit a concept with no causal effect on the answer.Source: §5 · p. 9
  • Effectiveness depends on an auxiliary model correctly and consistently extracting, crediting, and removing or editing input concepts.Extraction errors or inconsistency across similar inputs reduce effectiveness.Source: §5 · p. 9
  • The method treats input concepts as independent, although removing one can change the meaning of the remaining text.Future work could use a causal graph to guide which concepts to remove.Source: §5 · p. 9

Conclusion

  • Across two datasets and multiple model families, the removal wrapper improves explanation faithfulness; pairing it with modular prompting [9] further improves faithfulness in the experiments.It operates at test time without access to model internals.Source: §5 · p. 10

A.1 Measuring faithfulness

The appendix describes two faithfulness measures used in prior work and contrasts them with training-time approaches that require access to model weights.

A.1 Measuring faithfulness

  • The evaluation uses measures comparing concepts’ causal effects with explanation credit and testing whether explanations predict answers on perturbed inputs.It also mentions editing the chain of thought and checking whether the answer changes; the authors use the first two measures because their method constrains answer influences, not CoT use.Source: §A.1 · p. 11

A.2 Improving faithfulness at train-time

A.3 Improving faithfulness at test-time

Prior test-time approaches either constrain answers, decompose questions, or modify explanations; the removal wrapper instead keeps the explanation and edits the question.

A.3 Improving faithfulness at test-time

  • Formal Symbolic Reasoning [10] constrains answers to follow explanations, and Modular Prompting [9] separates sub-questions into contexts, yet the model sees the full input when choosing them or its program, allowing hidden concepts to influence both.Chuang et al. (2026) iteratively refine the prompt and explanation to raise a measured faithfulness score; Alon et al. (2026) estimate which input parts drove the answer.Source: §A.3 · p. 11
  • The removal wrapper keeps the model’s explanation unchanged and adapts the question so the answer more closely conforms to that explanation.This contrasts with approaches that work on the explanation instead.Source: §A.3 · p. 12

B Hint-Augmented QA Experiment Details

The appendix specifies the hint-augmented QA experiment’s faithfulness assumptions, metric adjustments, acknowledgment judgment, simulatability prompts, and regulative prompting baseline.

B.1 Hint verbalization

  • The hint-verbalization proof assumes deterministic target responses, correct auxiliary identification of hint credit, and a nonzero metric denominator; under these conditions the wrapper achieves HVF(M) = 1.The denominator condition means there are questions whose answer is influenced by the hint.Source: §B.1 · p. 12

B.2 Causal concept faithfulness

  • For causal concept faithfulness, the appendix uses total variation distance instead of KL divergence, measures it empirically, and correlates it with explanation error across questions.The questions have two default concepts: the factual question and the hint.Source: §B.2 · p. 12

B.3 Hint acknowledgment

  • Hint acknowledgment requires an explanation to rely explicitly on the user’s suggestion as support; merely mentioning or rejecting it does not count.The prompt asks the evaluator for a one-sentence justification followed by a YES or NO decision.Source: §B.3 · pp. 12–14

B.4 Normalized simulatability gain

  • The normalized simulatability gain setup compares predictions on a related question with and without the target model’s explanation.The prompts ask predictors to model the target AI’s answer rather than answer the question themselves.Source: §B.4 · pp. 14–15

B.5 Regulative Prompt Baseline

  • The regulative prompting baseline instructs the target model to ignore the user’s opinion, guess, or preference and answer from the question and its own knowledge.Source: §B.5 · p. 15

C.1 BBQ Data Pre-Processing

This section describes the BBQ data and prompt adaptations for concept editing and crediting, counterfactual prediction, modular question answering, and faithfulness evaluation.

C.1 BBQ Data Pre-Processing

  • Neutral labels (Person 1/2 or collective equivalents) prevent concept edits from changing the option identity associated with the same underlying choice.An auxiliary model performs the rewrite.Source: §C.1 · p. 15

C.2 Causal Concept Faithfulness

  • The experiment uses total variation distance to keep CE in [0, 1] and computes empirical CE directly because open-source models avoid prior paid-model sampling constraints [6].Source: §C.2 · p. 16

C.3 BBQ Prompts

  • The BBQ prompts adapt the causal-concept framework [6] to neutral labels and identify explicit, distinct concepts about individuals rather than inferred attributes or question-dependent scaffolding.The auxiliary model for these steps is Qwen3.5-27B.Source: §C.3 · pp. 16–17
  • Removal and value-change prompts edit only the selected concept, preserve other question information, and require the original answer choices to remain unchanged.Removal instructions address fused attributes and require checking that the removed value is no longer stated while kept concepts remain unchanged.Source: §C.3 · pp. 20, 21, 23, 24
  • The BBQ edit examples change selected concepts while retaining others, then ask annotators to judge whether the edited context remains coherent.The shown edits include a case judged incoherent when a charity donor was placed at a restaurant opening.Source: §C.3 · pp. 24–26
  • The concept-crediting prompt asks an auxiliary model to judge whether the target model’s explanation treats each listed concept as a reason for its answer.The examples distinguish concepts explicitly cited as reasons from concepts that are merely present in the question.Source: §C.3 · pp. 26–27
  • The NSG predictor simulates the reference model’s answer to a counterfactual question both without and with the reference explanation.The with-explanation prompt explicitly tells the predictor to use the reference system’s explanation to infer its behavior.Source: §C.3 · pp. 28–29
  • Modular prompting decomposes a multiple-choice question into the fewest self-contained subquestions, including only the facts each subquestion needs.Subquestions must be independently answerable and can refer to earlier answers through placeholders.Source: §C.3 · p. 29
  • Each modular subquestion is answered in a fresh context that sees only that question, and the final answer defers entirely to the subanswers.The recomposition prompt directs the model to map verified subanswers to answer choices without re-reasoning from scratch.Source: §C.3 · p. 30
  • The prompting baseline prepends an instruction to cite every factor that influences the answer and no factor that does not.It also requires the final answer to follow only from the stated reasons.Source: §C.3 · p. 31

D Per-concept z-score plots

Figures 6–8 plot per-concept causal and explanation effects for three target models. The plots use within-question z-scores, which show relative ordering rather than absolute effect size.

D Per-concept z-score plots

  • Figures 6–8 show per-concept CEz versus EEz for Qwen3.5-27B, Qwen3.5-9B, and Gemma-4-12B, respectively.Source: D Per-concept z-score plots · p. 31
  • Within-question z-scores show relative causal-effect ordering, not absolute magnitude; tiny within-item variance can make small effects appear far from the faithful line.Source: D Per-concept z-score plots · p. 31
  • Figure 7 uses one panel per method, with a fitted EEz = r CEz line and 90% confidence interval against the faithful EEz = CEz line.Source: Figure 7 · p. 32

E Why removal helps least on Qwen3.5-9B

Removal helps least on Qwen3.5-9B: different completions lead the wrapper to remove different concepts, leaving a similar number of new hidden concepts behind. Modular prompting is less sensitive to these varying reductions and lowers the count from 15 to 6.

E Why removal helps least on Qwen3.5-9B

  • 11.4% average completion disagreement on Qwen3.5-9B, versus 7.7% on Qwen3.5-27B and 6.2% on Gemma-4-12B, accompanies less consistent concept crediting.Because the wrapper removes concepts left uncredited by each completion’s explanation, it removes different sets on different runs; those varying reductions leave a similar number of new hidden concepts on Qwen3.5-9B.Source: E Why removal helps least on Qwen3.5-9B · p. 32
  • 15 to 16 hidden concepts on Qwen3.5-9B with removal alone, versus 15 to 6 with the modular-prompting combination.The wrapper halves hidden concepts on Qwen3.5-27B (12 to 6) and Gemma-4-12B (19 to 10).Source: E Why removal helps least on Qwen3.5-9B · p. 32
  • Modular prompting is less sensitive to small differences between reductions, so the combined method avoids the wrapper’s Qwen3.5-9B pattern of leaving a similar number of new hidden concepts behind.The combined method reduces the count from 15 to 6.Source: E Why removal helps least on Qwen3.5-9B · p. 32
Loading 2609.04343v1…