PaperlayerView exact revision on arXiv

Source-linked AI summary

How the Audit Rule Shapes Faithful Factor Explanations in LLMs

Taolin Zhang, Hanyu Wang, Jiuheng Wan, Tingyuan Hu, Chengyu Wang

arXiv:2610.01514v1cs.CL

TL;DR

The paper asks how to elicit truthful factor-level reports when verification is limited and reports can affect which factors are audited. It formalizes the setting as a verification game and finds that report-dependent auditing encourages suppression, while a report-independent audit component supports truthful reporting; synthetic and real-model tests follow this pattern, especially when incentives are explicit.

  • Problem

    Factor-level faithfulness has often been studied with models as passive targets, while truthful elicitation typically assumes exogenous verification or one-dimensional reports, leaving open how truthful factor reports fare when a limited audit budget depends on the report.

    Source: §1 · p. 1

  • Method

    The paper models reporting and auditing as a verification game, instantiates it with the Counterfactual Brier Score, and considers report-independent audits and a mixed rule with a report-independent floor.

    Source: §1 · p. 1, §3.1 · p. 3, §3.4 · p. 6

  • Results

    Synthetic agents match the predicted suppression under report-dependent auditing and truthful reporting with a report-independent floor; real LLMs respond strongly to explicit incentives, with weaker directionally similar behavior in multi-round interaction.

    Source: §4.6 · pp. 8–9

  • Takeaways & Limitations

    Under partial verification, explanation systems should include a report-independent audit component so under-reporting cannot reduce scrutiny.

    Source: §5 · p. 9

    The behavioral evidence is directional rather than equilibrium evidence, and experiments use API models with deterministic decoding, so openweight or sampling-based models may behave differently.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Large language models are often asked which input factors influenced their outputs. For structured inputs, such reports can be checked by counterfactual perturbation, but each factor must be queried multiple times to estimate its effect, so verification is usually budget-limited. We study how this limited-budget setting changes the incentive to report factor-level influence truthfully. We formalize the interaction as a verification game and show that proper scoring alone is not enough when auditing depends on the report: report-dependent auditing creates a suppression incentive, because factors reported as important are more likely to be checked and penalized for estimation noise. In contrast, report-independent auditing, or a mixed rule with a small report-independent floor, removes this channel and makes truthful reporting preferable to full suppression. We instantiate the framework with the Counterfactual Brier Score (CBS) and evaluate its predictions on four NLP benchmarks. A synthetic rational agent matches the theoretical prediction exactly, and real LLMs follow the same incentives when they are made explicit. The main design implication is simple: under partial verification, factor-level explanation systems should include a report-independent audit component so that under-reporting cannot be used to avoid scrutiny.

1 Introduction

The paper studies truthful factor-level reports when verification is limited and the report can affect which factors are audited. It argues that audit design, not proper scoring alone, determines whether truthful reporting is incentivized, and tests the predicted effects with synthetic and real agents.

1 Introduction

  • The paper frames factor-level faithfulness as truthful importance reports checked by removing or replacing individual input factors and observing output changes.This interventional signal avoids the ambiguity of evaluating free-form chain-of-thought explanations.Source: §1 · p. 1
  • The gap is that factor-faithfulness work often treats models as passive targets, while truthful-elicitation work typically assumes exogenous verification or one-dimensional reports.Here, the report can affect which factors are checked under a limited audit budget.Source: §1 · p. 1
  • The verification game makes truthful elicitation a joint property of the score and audit rule; CBS is strictly proper under exogenous verification when estimates are unbiased.Under budgeted endogenous auditing, properness alone is not enough.Source: §1 · pp. 1–2
  • Report-dependent auditing creates a suppression incentive because reporting a factor as important increases its chance of being audited and penalized for estimation noise.A completeness bonus instead induces inflation of non-causal factors; a report-independent rule or small report-independent floor removes the under-reporting channel.Source: §1 · p. 2
  • A synthetic rational agent matches the predictions, while real LLMs switch almost deterministically between suppression and truthful reporting when audit incentives are explicit.The tests use five commercial LLMs across four NLP benchmarks.Source: §1 · p. 2
  • The paper’s contributions combine the audit-policy result, CBS for multi-dimensional factor importance under exogenous unbiased verification, and empirical tests across models and benchmarks.The experiments include synthetic best responses, single-shot strategic choice, and multi-round behavioral checks.Source: §1 · p. 2

2 Related Work

The related work connects factor-level explanation faithfulness with truthful elicitation and verification under limited audit budgets. The paper distinguishes its setting by combining vector-valued reports, partial noisy verification, and an audit allocation controlled by the report.

Factor-level explanation faithfulness

  • Factor-level faithfulness studies test whether stated reasons align with model behavior, including through counterfactual perturbations [Han et al., 2026; Radhakrishnan et al., 2023].The paper narrows its question to making factor-level self-reports incentive-compatible when only a subset of factors can be verified.Source: §2 · p. 2

Scoring rules and mechanism design

  • Unlike scalar confidence calibration [Guo et al., 2017; Xiong et al., 2024] or text elicitation with available ground truth [Wu and Hartline, 2024], this setting elicits vector reports with partial, noisy verification.This connects the problem to mechanism design with verification, where incentives depend on both scoring and verification policy.Source: §2 · p. 2

Adjacent literatures

  • Decision markets use scalar probabilities and observed outcomes, whereas this work checks vector reports against noisy finite-sample estimates.Report-sensitive spot-checking [Zarkoob et al., 2020] also uses scalar reports and an observed TA signal.Source: §2 · pp. 2–3

Strategic reporting and verification

3 Method

The method models factor-level explanation as a verification game in which the model reports influence scores and the verifier audits counterfactual effects under a limited budget. It shows that audit design shapes whether truthful reporting is optimal.

3.1 Setup and Notation

  • For each structured-input factor, influence is its probability of changing the answer under a specified perturbation protocol, and the target report is the resulting influence vector.This is protocol-relative sensitivity, not access to internal representations.Source: §3.1 · p. 3

3 Method

  • The verifier commits to a scoring rule and audit policy, choosing them to make the target report optimal even when only a subset of factors can be audited.Source: §3.1 · pp. 3–4, §3 · p. 3

3.2 The Counterfactual Brier Score

  • The Counterfactual Brier Score applies a coordinate-wise quadratic score to counterfactual audit outcomes and is strictly proper with fixed audits and unbiased estimates.Single-factor perturbations cannot detect cross-term interactions; full Shapley evaluation requires O(2^m) evaluations.Source: §3.2 · p. 4

3.3 Audit Design: What Can Go Wrong?

  • Under proportional auditing, reporting an interior-influence factor positively has negative expected payoff, encouraging suppression; a completeness bonus instead can reward inflation of non-causal factors.The coordinate-wise propositions do not establish full suppression as the global optimum; the stronger claim is checked numerically.Source: §3.3 · p. 5

3.4 Resolution: Report-Independent Verification

  • Report-independent audits preserve strict propriety, while a mixed rule’s report-independent floor makes truthful reporting outperform full suppression above a threshold.The theorem compares truth with full suppression, not every partially distorted report; in practice, a 10% floor beats full suppression on almost all evaluation instances.Source: §3.4 · p. 6

4 Experiments

The experiments test audit-rule predictions with synthetic best responses, strategic choices by real LLMs, and multi-round interactions. Across these settings, report-independent auditing supports more faithful factor reporting, while explicit incentives elicit stronger strategic adaptation than policy descriptions alone.

4 Experiments

  • The experiments compare synthetic best responses, single-shot real-LLM choices under explicit incentives, and multi-round behavioral adaptation under repeated feedback.The setup describes these as tests of the theoretical predictions.Source: §4 · p. 6

4.1 Setup

  • The evaluation uses 200 problems per dataset and three counterfactual replacements per factor, with the audit budget reallocating checks rather than increasing expected API cost.It covers five API-accessed LLMs and BBQ, GSM8K, HotpotQA, and StrategyQA.Source: §4.1 · pp. 6–7

4.2 Rational-Agent Best Responses

  • 100% truthful reporting under report-independent auditing and about 98% with a 10% floor contrast with 63.1% truthful and 36.9% suppressed factors under proportional auditing.The mixed rule recovers truthful reporting across alternative audit rules; top-k auditing still suppresses a sizable fraction.Source: §4.2 · p. 7, Figure 1 · p. 7

4.3 Score-Agnosticity of the Audit Rule

  • Across three separable strictly proper scores, proportional auditing induces suppression, report-independent auditing restores full truthfulness, and a 10% floor recovers about 98% truthfulness.This supports an audit-allocation explanation rather than one specific to CBS.Source: §4.3 · p. 8, Table 2 · p. 8

4.4 Real-LLM Strategic Choice

  • With explicit reward tables, LLMs overwhelmingly choose suppression under proportional auditing and truthful reports under report-independent auditing; adaptation is much weaker without them.The experiment offered truthful, suppress-all, inflate-all, uniform, and interpolated reports.Source: §4.4 · p. 8, Figure 2 · p. 8

4.5 Behavioral Plausibility Check

  • In multi-round games, proportional auditing shifts behavior toward more suppression, larger distortion, and lower normalized CBS; the evidence is directional, not equilibrium behavior.The pattern is clearer on the two strongest models with cumulative feedback.Source: §4.5 · p. 8, Table 3 · p. 9

4.6 Summary of Empirical Evidence

  • Across five LLMs and four datasets, report-independent auditing yields lower distortion, higher F1 and normalized CBS, and less suppression.The multi-round results are directional behavioral evidence, not equilibrium behavior.Source: Figure 3 · p. 9, §4.6 · p. 9

4.7 Additional Results

  • Four representative case studies show close alignment in faithful cases and suppression or inflation in unfaithful cases; a deployment vignette illustrates how a report-independent floor can reveal demographic suppression.These examples provide qualitative support for the proposed mechanism.Source: §4.7 · p. 9

5 Conclusion

The conclusion argues that partial verification makes truthful factor-level reporting depend on audit design as well as scoring. It recommends a report-independent audit component, citing results consistent with this account across five LLMs and four benchmarks.

5 Conclusion

  • Under budgeted, report-dependent auditing, proper scoring alone does not ensure truthful reports; a small report-independent floor removes the suppression incentive without increasing the verification budget.The floor keeps total verification budget fixed.Source: §5 · p. 9
  • Across five LLMs and four benchmarks, empirical results are consistent with the account, and current models follow the incentives when the mechanism is explicit.The conclusion says models do not reliably derive these incentives on their own.Source: §5 · p. 9
  • The practical implication is to include a report-independent audit component in factor-level explanation systems under partial verification.Source: §5 · p. 9

Limitations

The behavioral experiments provide directional rather than equilibrium evidence, and the paper cautions that current LLMs do not consistently optimize the mechanism without explicit guidance. Results may differ for openweight models or sampling-based generation.

Limitations

  • Behavioral results are directional, not equilibrium evidence; current LLMs do not consistently optimize the mechanism without explicit guidance.The multi-round results should be read in that light; experiments use commercial API models with deterministic decoding, so openweight models or sampling-based generation may behave differently.Source: Limitations · p. 9

Ethical Considerations

The ethical discussion characterizes the benchmark data and intended use, while the appendix proofs establish strict propriety under unbiased verification and a suppression incentive under proportional auditing.

Ethical Considerations

  • The work uses publicly available benchmarks and commercial API models; BBQ demographic descriptors are synthetic stimuli, not real personal information.The framework is intended to make explanations easier to audit, although a known audit rule could itself become an optimization target.Source: Ethical Considerations · pp. 9–10

A.1 Proof of Theorem 1

  • For any report other than truth, the proof establishes a strictly positive objective difference under unbiasedness.This is the strict-positivity step in the proof of Theorem 1.Source: §A.1 · p. 12

A.2 Proof of Proposition 2

  • Under proportional auditing, suppressing a positively reported factor improves utility, and full suppression beats truth when at least two factors have interior influence.Each interior factor contributes strictly negative utility under proportional auditing, while full suppression yields zero.Source: §A.2 · p. 13

A.3 Proof of Proposition 3

The proofs show that a verification bonus can make positive reports of non-causal factors preferable to silence, whereas report-independent auditing uniquely rewards truthful reports when every factor has positive audit probability.

A.3 Proof of Proposition 3

  • For a non-causal factor, adding a bonus per verified reported factor can make a positive report strictly better than silence.The proof also establishes that the payoff maximum is attained at an interior report.Source: §A.3 · p. 13

A.4 Proof of Theorem 4

  • With report-independent audit probabilities, each coordinate’s strictly convex objective is minimized at its true influence, making truth the unique best response.The condition σ_j > 0 is necessary because a never-audited factor cannot be identified.Source: §A.4 · p. 13

A.5 Proof of Theorem 5

The mixed audit rule makes truthful reporting preferable to full suppression above a threshold and converges to report-independent auditing as its floor weight approaches one. The appendix also describes the experiments’ implementation, prompts, and benchmark perturbations.

A.5 Proof of Theorem 5

  • Above the influence-dependent threshold α*, truthful reporting strictly dominates full suppression; as α approaches 1, truthful reporting becomes the unique best response.At α = 1, the mixed rule converges to report-independent auditing.Source: §A.5 · p. 14

B.1 Implementation Details

  • The base evaluation uses 200 examples per benchmark and three counterfactual replacements per factor, totaling about 70,000 calls across five models.The estimate is roughly 14,000 counterfactual API calls per model across the benchmarks.Source: §B.1 · p. 14
  • Factor-importance reports use structured JSON prompts with scores from 0 to 1, and answer matching uses normalized exact string match.The verifier estimates influence from the empirical fraction of counterfactual replacements that flip the answer.Source: §B.1 · p. 14, §B.2 · p. 14

B.2 Prompt Templates

  • The prompt suite includes instructions to inflate all factor scores to 0.85–0.95 or deflate them to 0.05–0.15.A separate perturbation-oracle prompt is kept minimal to reduce contamination from earlier instructions.Source: §B.2 · pp. 14–15
  • The single-shot strategic-choice prompt asks models to select the highest-reward report under proportional or report-independent auditing.The prompt describes proportional audits as more likely for factors assigned higher importance, while report-independent audits use fixed probabilities.Source: §B.2 · p. 15
  • The multi-round game gives cumulative-score feedback and asks models to adapt their factor reports over successive rounds.Feedback includes reports, audits, revealed effects, and scores.Source: §B.2 · pp. 15–16

B.3 Datasets and Perturbations

  • The four benchmarks use task-specific counterfactuals: attribute swaps for BBQ, numerical substitutions for GSM8K, paragraph edits for HotpotQA, and fact removals for StrategyQA.StrategyQA produces the noisiest and most subjective perturbation setting among the four domains.Source: §B.3 · p. 16

B.4 Sanity Checks on the Scoring Instrument

The checks find that CBS detects both inflation and deflation, with more balanced sensitivity than the compared alternatives. Across domains, measured faithfulness varies, and correct answers do not guarantee faithful factor reports.

Cross-domain faithfulness

  • Normalized CBS is highest on GSM8K, where factor effects are relatively crisp, and lowest on StrategyQA, where perturbation effects are noisier and more subjective.Variation across models is domain-specific, so individual gaps warrant cautious interpretation.Source: §B.4 · p. 16

Adversarial robustness

  • CBS decreases under both inflated and deflated reports, with a larger drop for inflation in the tested BBQ subset.Figure 4 reports mean normalized CBS for 150 trials per condition, with error bars of ±1 SEM.Source: Figure 4 · p. 16, §B.4 · p. 16

Scoring-rule comparison

  • Among proper scores, CBS has the most balanced sensitivity to inflation and deflation; diagnostic baselines are less balanced and lack the same incentive guarantees.The other proper scores preserve the main audit-design result in synthetic best-response analysis.Source: §B.4 · p. 17

Continuous vs. binary elicitation

  • Continuous reports outperform binary support reports, especially for factors with intermediate rather than purely binary influence.This supports using graded reports in the main experiments.Source: §B.4 · p. 17

When are models unfaithful?

  • In BBQ, identity-related factors are often under-reported; in HotpotQA, importance can be assigned to non-causal distractors, and answer correctness is only weakly related to faithful reporting.Source: §B.4 · p. 17

B.5 Estimation Robustness

Robustness checks find that audit-design patterns persist across factor schemas and rounds, while measurement estimates depend on perturbation budgets, noise, and templates. A report-independent floor keeps suppressed factors auditable without increasing the overall audit budget.

Sample complexity

  • By n ≥ 5, estimates are very close to the n = 10 reference; gains diminish quickly after n = 2, supporting n = 3 as a lower-cost practical default.The largest gain appears between n = 1 and n = 2, and variance also decreases as the budget increases.Source: §B.5 · p. 17

Robustness under noise

  • As verification noise increases, the gap between truthful and random reports shrinks; CBS retains positive discrimination under moderate noise but loses resolution under heavy corruption.The decline is smooth and qualitatively similar across domains.Source: §B.5 · p. 17

Template sensitivity

  • Aggregate normalized CBS changes little across alternative counterfactual templates, although per-factor estimates show noticeable template sensitivity.The aggregate pattern suggests the metric-level conclusions are not driven by a single template; the per-factor variation is consistent with protocol-relative faithfulness.Source: §B.5 · p. 17

Factor-schema sensitivity

  • Across three factor schemas, proportional auditing induces suppression, while report-independent auditing restores truthful reporting; a 10% floor repairs suppression in all three.Source: §B.5 · pp. 17–18

Long-horizon behavior

  • On Qwen3.7-Max and DeepSeek-V4-Pro, the gap between proportional and report-independent auditing appears early and remains stable across 20 rounds.The authors suggest stronger models track the audit incentive quickly once cumulative feedback is available.Source: §B.5 · p. 18

Deployment vignette

  • In a logged BBQ example, a report-independent floor keeps an unreported identity factor auditable, revealing its influence and penalizing the misreport without increasing the overall audit budget.Source: §B.5 · p. 18

B.6 Case Studies

The cases illustrate both faithful and misleading factor reports, including deliberate suppression and mistaken introspection. Counterfactual verification distinguishes behaviorally accurate explanations from self-reports that hide relevant dependencies.

Case 1: Faithful report avoids a stereotype

  • In a BBQ case, only question framing affects the answer, and the model assigns almost all reported importance to that factor, which verification confirms as near-faithful.The high CBS score verifies consistency with behavior rather than relying on the self-report alone.Source: §B.6 · pp. 18–19, Table 8 · p. 20

Case 2: Demographic suppression and evidence inflation

  • In a BBQ gender case, the model’s answer depends on gender, qualifications, and framing, but its report suppresses the first two and inflates non-causal factors.A report-independent floor keeps the hidden identity factor auditable when proportional auditing may miss it.Source: §B.6 · p. 19, Table 9 · p. 20

Case 3: Perfect multi-hop attribution

  • In a HotpotQA multi-hop case, the model identifies exactly the two supporting paragraphs and assigns zero importance to distractors, yielding a behaviorally accurate attribution.Source: §B.6 · p. 19, Table 10 · p. 20

Case 4: Mistaken introspection hides broad dependence

  • In a HotpotQA case, perturbation shows dependence on all ten paragraphs, while the model reports only one, reflecting mistaken introspection as well as unfaithfulness.A report-independent audit rule keeps unreported coordinates auditable.Source: §B.6 · pp. 19–20, Table 11 · p. 20

Related papers

Loading 2610.01514v1…