PaperlayerView exact revision on arXiv

Source-linked AI summary

Would this change your answer? Evaluating Explanations of LLM Behavior In The Wild with Counterfactual Experiments

Adam Karvonen, Euan Ong, Subhash Kantamneni, Samuel Marks

arXiv:2608.16747v1cs.LGcs.AI

TL;DR

Evaluating explanations is difficult because model behaviors’ true causes are usually unknown, and prior counterfactual-simulatability evaluations have often used narrow settings. CHIVE investigates unexpected behaviors with counterfactual prompt edits; interpretability tools show no uplift, while training on CHIVE data generalizes to held-out investigations, including an out-of-distribution prompt source.

  • Problem

    Because model behaviors’ true causes are usually unknown, explanations cannot be checked directly, and counterfactual-simulatability evaluations have often relied on narrow settings.

    Source: §1 · p. 1

  • Method

    CHIVE discovers unexpected model behaviors by sampling responses and investigating them with counterfactual prompt edits.

    Source: §1 · p. 1

  • Results

    Training on CHIVE data improved counterfactual predictions on held-out investigations, including an out-of-distribution prompt source, while the evaluated interpretability tools showed no uplift.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    Counterfactual investigations offer a general approach to constructing training and evaluation datasets about causes of model behavior at scale.

    Source: §6 · p. 10

    Because these behaviors have available ground-truth counterfactuals, the evaluation is only a proxy for real use cases and may face important distribution shifts.

    Source: §5 · p. 9

Abstract

from arXiv · show

Many areas of AI research, such as language model interpretability and chain of thought faithfulness, seek to explain model behaviors. But what constitutes a "good" explanation? In this work, we evaluate explanations through the lens of counterfactual simulatability-whether the explanation is useful for predicting model behaviors on related counterfactual inputs. To this end, we introduce CHIVE (Counterfactual Hypothesis Investigation Via Edits), a novel agentic pipeline that identifies unexpected model behaviors in the wild and investigates them with counterfactual prompt edits. This yields thousands of high-quality explanations for naturally-occurring model behaviors along with supporting counterfactual evidence. We apply CHIVE in two ways. First, we evaluate whether common LLM interpretability techniques improve an agent's ability to predict counterfactual model behaviors. Surprisingly, we find no uplift from any of the interpretability techniques studied. Second, we use CHIVE to generate training data. We find that training models to predict outcomes of CHIVE-generated counterfactual experiments generalizes to various out-of-distribution settings. Overall, CHIVE automatically discovers explanations of naturally-occurring LLM behaviors, enabling us to evaluate and improve methods for explaining LLM behaviors.

1 INTRODUCTION

The paper evaluates explanations by counterfactual simulatability: whether they help predict model behavior after related prompt edits. It introduces CHIVE to investigate unexpected behaviors in the wild and uses its counterfactual data to evaluate interpretability tools and train behavior predictors.

1 INTRODUCTION

  • CHIVE discovers unexpected behaviors by sampling and screening model responses, then investigating them with counterfactual prompt edits, producing thousands of in-the-wild explanations and experiments.The explanations are open-ended behavioral hypotheses, not ground truth; measured counterfactual outcomes supply the evaluation labels. The paper releases the resulting evaluation and training datasets.Source: §1 · pp. 1–2
  • The paper defines explanations as behavioral claims testable by counterfactual experiments, not accounts of internal computation or training data.Counterfactual simulatability asks whether an explanation helps predict model behavior on related counterfactual inputs.Source: §1 · p. 1
  • Three activation-reading interpretability tools provide no uplift in predicting counterfactual outcomes over a transcript-only predictor across two target models and three predictor families.The tools are activation oracles, natural-language autoencoders, and sparse autoencoders.Source: §1 · p. 2
  • Training models to predict whether prompt edits change their behavior improves performance on held-out investigations, an out-of-distribution prompt source, and a held-out hint setting.Training models to produce open-ended explanations gives weaker, mixed results.Source: §1 · p. 2

2 A PIPELINE FOR COUNTERFACTUAL-GROUNDED EXPLANATIONS OF IN-THE-WILD BEHAVIORS

CHIVE discovers in-the-wild model behaviors through prompt-edit experiments, producing open-ended explanations and independently checkable counterfactual outcomes. The evaluation dataset filters for coherent, reproducible effects while reducing confounds.

2.1 PIPELINE DESCRIPTION

  • CHIVE samples 30 responses per prompt from tens of thousands of prompts, flags unexpected behavior, tests 5–15 prompt edits, and independently judges whether experiments support explanations.Source: §2.1 · p. 3

2.2 EXPLANATIONS AND COUNTERFACTUAL CLAIMS

  • Investigations yield open-ended explanations and measured counterfactual outcomes; evaluation labels derive from the experiments, not explanations treated as ground truth.Claims assert an edit changes behavior frequency by at least 30 percentage points; true claims require a change of at least 50 points, while false claims require at most 15.Source: §2.2 · p. 4, Figure 3 · p. 4

2.3 QUALITY FILTERS AND THE EVALUATION DATASET

  • The evaluation filters target coherent mechanisms, confounds, and reproducibility; removing them does not qualitatively change the takeaways.Confounded edits and unstable effects, such as a story protagonist’s name changing after minor prompt edits, may not be predictable from articulable model properties.Source: §2.3 · pp. 4–5

3 APPLICATION 1: EVALUATING INTERPRETABILITY TOOLS

Across two target models, activation-based interpretability tools did not improve counterfactual predictions over a transcript-only baseline, despite tool outputs sometimes informing the predictor. Tool outputs rarely stated the causal relationships needed to predict whether an edit would change behavior.

3.1 BACKGROUND: EVALUATING INTERPRETABILITY TOOLS

  • Auditing games measure whether interpretability tools help an agent uncover hidden behavior, but narrow fine-tuning can make results training-dependent and their transfer to natural behaviors unclear.This study likewise measured tool uplift over a baseline predictor without tool access.Source: §3.1 · p. 5

3.2 SETUP AND TOOLS

  • Predictors estimated whether a counterfactual claim was true from the transcript, scored by accuracy and AUROC, with up to five tool calls.The tested tools were activation oracles, natural-language autoencoders, and sparse autoencoders.Source: §3.2 · pp. 5–6
  • Main-body evaluations used Opus 4.8 with read-only access to target-model internals, excluding steering and activation patching that could approximate the intervention’s ground truth.Targets were Qwen3-8B and Gemma-3-27B-IT.Source: §3.2 · p. 6

3.3 RESULTS

  • Across both target models, none of the three activation-based tool predictors improves on the transcript-only baseline in this evaluation (Figure 4), unlike the clear uplift these tools provide in narrow-fine-tune auditing games.Source: §3.3 · p. 6
  • The null result persisted across predictor families, hyperparameter sweeps, prompt optimization, and a reasoning-model target.GPT-5.5 and Gemini-3.1-Pro predictors replicated the qualitative result; Claude Fable 5 prompt and hyperparameter optimization also produced no uplift.Source: §3.3 · p. 6
  • Tool outputs sometimes informed predictions, but cases where they failed to surface relevant information or misled the predictor offset those wins.Examples of both outcomes are presented in Appendix C.5.Source: §3.3 · p. 6
  • Only 0.6% of claims whose edit changed behavior had NLA outputs explicitly state that feature X causes behavior Y.Outputs usually described both the targeted concept and behavior, leaving predictors to infer causality from correlational cues.Source: §3.3 · p. 6

4 APPLICATION 2: TRAINING MODELS TO PREDICT THEIR OWN BEHAVIOR

The section tests whether models can learn to predict the effects of counterfactual prompt edits from CHIVE investigations. It reports generalization to hint tasks and held-out investigations, mixed results for open-ended explanations, and no evidence of privileged access.

4.1 BACKGROUND: CAUSAL SELF-REPORT AND SELF-EXPLANATION

  • Prior causal self-report training centers on hint settings with one known cue and predefined outcome, and has not been shown to generalize to an entirely held-out setting.The section uses “self-explanation” for explanations of a model’s own behavior without implying introspection or privileged access; Hase & Potts (2026) report a +35pp hint-setting result but only +2pp on benchmark questions.Source: §4.1 · p. 7

4.2 SETUP

  • Training uses the target model’s own transcript to predict whether a counterfactual claim holds; the study tests Qwen3-8B and a larger model investigating itself.Investigations also provide open-ended explanations as a training target; the 8B model’s investigations are run by Opus, while Qwen3.5-397B-A17B investigates itself.Source: §4 · p. 6, §4.2 · pp. 7–8

4.3 GENERALIZATION TO A HELD-OUT SETTING: HINT INFLUENCE

  • Without training targeted to hint tasks, both trained models improve accuracy and AUROC over their base models and approach or match Opus on hint-influence prediction.Evaluation uses Reddit AITA posts with appended opinions and MMLU questions with injected cues; labels reflect whether removing the cue flips the answer.Source: Figure 5 · p. 7, §4.3 · p. 8

4.4 EVALUATIONS ON HELD-OUT INVESTIGATIONS

  • Open-ended explanation training generalizes unevenly: it improves hint attribution for the 397B target but not the 8B, and can mislead strong simulators on held-out claims.For the 397B target, cue-attribution accuracy rises from 0.59 to 0.68, reaching the Opus reference at 0.64; on held-out investigations, explanations help weak simulators but give strong ones confident, often-wrong counterfactuals.Source: §4.4 · p. 8
  • Both trained models improve substantially over the base model on held-out investigations and land within ±0.03 AUROC of Opus on every set, without clearly exceeding it.The held-out claims come from WildChat and out-of-distribution PETRI transcripts.Source: §4.4 · p. 8, Figure 6 · p. 9

4.5 TESTING FOR PRIVILEGED ACCESS

  • Cross-trained models match self-trained models on claim prediction, providing no evidence that models use privileged access to their internal state.The test compares Qwen3-8B and Llama-3.1-8B-Instruct trained on claims about themselves or the other model.Source: §4.5 · pp. 8–9

5 DISCUSSION AND LIMITATIONS

CHIVE’s discussion explains why activation-based interpretability tools may miss naturally occurring behaviors and identifies limits on what the evaluation can establish. It also argues that counterfactual investigation is adaptable beyond the studied data and behavior classes.

5 DISCUSSION AND LIMITATIONS

  • Activation tools miss natural-behavior causes because context-feature links are absent from their outputs, unlike anomalous quirks that enable uplift in narrow fine-tuning audits (Marks et al., 2025; Sheshadri et al., 2026; Cywi´nski et al., 2025).In natural behaviors, the cause is a feature of the present context rather than an anomalous concept absent from the prompt.Source: §5 · p. 9
  • The pipeline runs unchanged on other transcript sources and can retarget screening toward behavior classes such as evaluation awareness, sycophancy, or unfaithful reasoning.The PETRI claims required no pipeline modification.Source: §5 · p. 9
  • Because clean ground-truth counterfactuals are available, the evaluation is only a proxy for real use and may face distribution shifts; its null result chiefly informs harder cases without clean counterfactuals, such as unverbalized evaluation awareness.The authors note that interpretability tools and self-explanations are most valuable when no other method can check their outputs.Source: §5 · pp. 9–10
  • Because all evaluated tools are activation-based, their failure may reflect activations’ limits for causal claims, tools that do not surface available relationships, or inadequate elicitation and scaffolding.The authors suggest weight- or circuit-level access, training existing tools on their datasets, or better scaffolding as possible routes forward.Source: §5 · p. 10
  • Read-only access leaves the achievable prediction ceiling unknown, so the evaluation compares each tool’s uplift over a transcript-only baseline rather than against perfect performance.Prompt edits would run the ground-truth experiment, while internal interventions such as steering or activation patching would approximate it.Source: §5 · p. 10

6 CONCLUSION

Counterfactual investigations can produce diverse, realistic data about the causes of model behavior at scale, offering a general approach to building training and evaluation datasets.

6 CONCLUSION

  • Counterfactual investigations produce diverse, realistic data about model behavior at scale, supporting the construction of training and evaluation datasets.The paper presents this as a promising general approach to addressing limited data about the causes of model behavior.Source: §6 · p. 10

AUTHOR CONTRIBUTION STATEMENT · APPENDIX CONTENTS · A EXTENDED DISCUSSION

Adam Karvonen proposed the project, designed and ran its experiments, and wrote the paper; Euan Ong and Subhash Kantamneni contributed discussion and feedback, while Samuel Marks supervised the project.

AUTHOR CONTRIBUTION STATEMENT

A.1 THE EVALUATION PROXY ARGUMENT … B.4 EVALUATION CLAIM FORMAT

The appendix argues that counterfactual simulatability is a useful proxy for evaluating behavioral explanations despite limits on how well prompt edits overlap with interpretability use cases. It describes how CHIVE investigates behaviors, builds and filters evaluation claims, and tests whether models can predict counterfactual outcomes.

A.1 THE EVALUATION PROXY ARGUMENT

  • The authors propose a proxy with clear ground truth and real-use-case-shaped questions, while cautioning that its relevance depends on overlap with factors interpretability tools address.Many counterfactuals concern how a model internally represents its input, such as interpreting an ambiguous sentence, providing some connection to internal representations.Source: §A.1 · p. 15

A.2 HYPOTHESIS GENERATION VS. HYPOTHESIS DISCRIMINATION

  • Because unexpected causes are often guessable within 10–100 hypotheses, the authors argue that hypothesis discrimination matters for explaining specific behaviors, unlike hypothesis generation for transcript scanning or model-organism analysis.The studied tool-generated causes could be guessed from behavior descriptions alone, while other workflows may still benefit from hypothesis generation.Source: §A.2 · pp. 15–16
  • In every model–behavior pair, the tool-discovered cause appeared among the first three guesses in at least 60% of samples, most often as the first guess.The experiment asked Opus 4.1 and Opus 4.5 to generate ten sets of five hypotheses from behavior descriptions; their training cutoffs preceded the NLA paper.Source: §A.2 · p. 15, Table 1 · p. 16

B.1 DETAILED INVESTIGATION PIPELINE DESCRIPTION

  • CHIVE samples target-model responses, screens for unexpected behaviors, investigates them with edited prompts, and verifies explanations; a fixed classifier question measures behavior rates across experiments.Investigators run roughly 10–15 experiments for evaluation and 5–8 for training, and retain investigations scoring at least 8 for evaluation or 7 for training.Source: §B.1 · pp. 16–17

B.2 PROMPT SOURCES

  • The pipeline uses a WildChat-based mixture as its default source for all training data and in-distribution evaluations, and PETRI auditing transcripts as its out-of-distribution source.Source: §B.2 · p. 17

B.3 EVALUATION DATASET CONSTRUCTION AND FILTERS

  • The evaluation set begins with 4,714 prompts, and the final dataset size varies with how many claims pass its quality filters.Screening requires behaviors to occur at least 30% of the time, while positive evaluation claims require at least a 50-percentage-point change, limiting coverage of less frequent behaviors.Source: §B.3 · pp. 17–18
  • Evaluation claims are selected for concrete mechanisms, counterfactual reproducibility, and single-factor interventions, with the scored set balanced 50/50 and capped at 2,000 claims.Reproducibility and unconfounded-edit checks each retain about 90% of claims; positive claims also require a behavior change of at least 50 percentage points.Source: §B.3 · p. 18, Table 2 · p. 19

B.4 EVALUATION CLAIM FORMAT

  • Each claim asks whether a described prompt intervention changes a behavior’s baseline rate by at least 30 percentage points in either direction.In the example, renaming function parameters changed the behavior from 26/30 to 1/30 responses, an 83-point change, so the claim was labeled Yes.Source: §B.4 · pp. 18–19

B.5 DATA GENERATION COSTS … C.3 CROSS-MODEL PREDICTOR AGENT REPLICATION

CHIVE’s data-generation costs depend on the data regime and deployment assumptions, while interpretability tools provide no uplift across target models and predictor agents.

B.5 DATA GENERATION COSTS

  • Costs exclude target-model serving; screening and verification are priced at discounted Batch API rates, while investigation uses standard rates and prompt caching.The estimates use Opus 4.6 at the stated Anthropic token prices, with screening and verification discounted by 50%.Source: §B.5 · p. 20
  • $3.3k: an evaluation run of 4,714 prompts yields 1,700–2,000 verified investigations, or about $1.9 per investigation.Investigation costs $0.96 per investigation with prompt caching, versus about $2.60 uncached; verification costs nearly as much as investigation, and using one judge sample could reduce verification costs by 80%.Source: §B.5 · p. 20
  • $0.9 per investigation: the Opus-investigated Qwen3-8B training run cost about $26k for 28,330 investigations.This cheaper regime uses fewer counterfactual experiments, n=10 rather than n=30 responses for behavior rates, and one rather than five judge samples; the training set uses the roughly 22,000-investigation train split.Source: §B.5 · p. 20
  • The Qwen3.5-397B-A17B self-investigation run has no reported cost because deployment and caching choices change the cost substantially.Investigation used 180k input and 5.1k output tokens per investigation, compared with 261k billed input and 8.2k output for the Opus training run; both used a 4,096-token thinking budget.Source: §B.5 · p. 20

C.1 TOOL CALL DETAILS

  • Each tool agent can make up to five target-model calls; NLA and SAE calls return outputs for at most 32 selected token positions.Activation-oracle calls instead ask a natural-language question about a selected transcript activation segment.Source: §C.1 · pp. 20–21
  • −0.012 to +0.004: paired AUROC uplift across tool settings, with every 95% bootstrap confidence interval straddling zero.Giving agents all per-token NLA generations at once on short transcripts also did not help.Source: Table 4 · p. 21, §C.1 · p. 21

C.2 QWEN3-8B INTERPRETABILITY TOOL RESULTS

  • No activation-based tool improves on the transcript-only baseline for Qwen3-8B, matching the result for Gemma-3-27B-IT.This holds across the activation oracle, natural-language autoencoder, and sparse autoencoder.Source: §C.2 · p. 21, Figure 7 · p. 22

C.3 CROSS-MODEL PREDICTOR AGENT REPLICATION

  • No predictor agent gains uplift from interpretability tools; Gemini is significantly hurt by the SAE and NLA tools.The replication used GPT-5.5 and Gemini-3.1-Pro in place of Opus 4.8 on the same Gemma-3-27B-IT claims.Source: §C.3 · p. 21, Table 5 · p. 22

C.4 EXPLORATORY ANALYSIS OF NLA OUTPUTS … D.3 HINT-SETTING COUNTERFACTUAL PREDICTION: FALSE AND TRUE POSITIVE RATES

The section diagnoses why interpretability outputs often fail to improve counterfactual predictions, illustrates cases where they help or mislead, and details training and hint-setting evaluations whose target models show opposite response biases.

C.4 EXPLORATORY ANALYSIS OF NLA OUTPUTS

  • NLA outputs rarely state causal links and their clues largely restate transcript information, while tool-induced flips often shift predictions toward no effect.Explicit causal statements appear in 0.6% of positive claims (3/479) and 0.02% of selected generations; judging every transcript token yields 1.2% of claims. Supporting clues occur in 47% of positive versus 30% of negative claims, but achieve 0.60 AUROC versus 0.77 for transcript-only prediction, with only +0.001 AUROC when added. For NLA flips, 82% on positive and 84% on negative claims move toward no effect, roughly cancelling accuracy changes: 73%→63% on positives and 74%→81% on negatives.Source: §C.4 · pp. 22–23, Table 6 · p. 23

C.5 QUALITATIVE INTERPRETABILITY TOOL USE EXAMPLES

  • Across four examples, NLA outputs sometimes help identify a missing model distinction or memorized association, but can also mislead or fail to reveal the cue causing behavior.The encoder-versus-decoder example moves the prediction toward the correct answer (0.50→0.25), and the memorized-association example does too (0.25→0.72); misleading streamer-name generations and absent cue signals move predictions to the wrong answer (0.56→0.32 and 0.34→0.20).Source: §C.5 · pp. 23–24

D.1 TRAINING DETAILS AND TARGET FORMATS

  • Training examples append a follow-up turn to the model’s own transcript, applying language-modeling loss only to the final assistant turn.The follow-up asks about a counterfactual claim rendered using the interpretability-evaluation template; the paper also specifies an open-ended explanation format.Source: §D.1 · pp. 24–25

D.1.1 VERBATIM TARGET FORMAT

  • The verbatim counterfactual-prediction target is a single Yes or No token answering one counterfactual claim.Source: §D.1.1 · p. 25

D.1.2 DATASETS

  • Counterfactual-prediction datasets are balanced between true and false claims, whereas each open-ended explanation dataset has one example per investigation.Table 7 gives training-example counts by target model and task.Source: §D.1.2 · p. 25, Table 7 · p. 25

D.1.3 TRAINING HYPERPARAMETERS

  • Both target models are LoRA fine-tuned for one epoch on counterfactual-prediction data and three epochs on open-ended explanations, with other hyperparameters shared.Qwen3-8B is trained locally and Qwen3.5-397B-A17B through the Tinker API.Source: §D.1.3 · p. 25, Table 8 · p. 25

D.2 HINT-SETTING DATASET CONSTRUCTION

  • Hint-setting evaluations use balanced positive and negative cases, with sycophancy prompts from AITA posts and MMLU questions paired with varied answer cues.Cue-present and cue-removed responses are sampled ten times each; only questions with at least 60% cue-consistent cued responses are included, and positive/negative labels depend on cue effects. Sycophancy cues state the user’s opinion, while MMLU cues include five forms such as professor opinions and check marks; cues target uniformly random options, including correct answers, because consistently wrong answers were scarce for the 397B model.Source: §D.2 · pp. 25–26

D.3 HINT-SETTING COUNTERFACTUAL PREDICTION: FALSE AND TRUE POSITIVE RATES

  • The hint-setting targets fail in opposite directions: the 8B model almost always answers Yes, while the 397B model rarely does, leaving its accuracy below what its AUROC supports.Table 9 reports false and true positive rates by setting.Source: §D.3 · p. 26, Table 9 · p. 26

D.4 PRIVILEGED ACCESS EXPERIMENT · D.5 PRESERVING THE TARGET MODEL’S BEHAVIOR WITH KL REGULARIZATION · E OPEN-ENDED EXPLANATION TRAINING

The privileged-access experiment finds no self-predictor advantage, while KL regularization preserves the target model’s behavior without improving counterfactual evaluation. Open-ended explanation training teaches models to explain behaviors alongside the counterfactual experiments that support those explanations.

D.4 PRIVILEGED ACCESS EXPERIMENT

  • For each target, the experiment compares its self predictor with the other model’s cross predictor, trained on identical claim data and evaluated on held-out claims.Datasets are matched at 23,944 training examples; held-out sets contain 1,497 Qwen-target claims and 1,392 Llama-target claims.Source: §D.4 · pp. 26–27
  • The authors are uncertain why Li et al. (2026) found privileged access in the hint setting; they suggest effects may be fragile or require more data for broader tasks.Binder et al. (2024) found no privileged access in the hint setting with a different model pair; the authors suggest broader tasks may require learning a behavioral prior before a privileged signal.Source: §D.4 · p. 27
  • Neither target’s self predictor beats its cross predictor, and all four trained models approach or match the Opus reference.Figure 8 compares self and cross predictors trained on identical, size-matched data for each target.Source: Figure 8 · p. 27, §D.4 · p. 27

D.5 PRESERVING THE TARGET MODEL’S BEHAVIOR WITH KL REGULARIZATION

  • Self-explanation training risks behavioral drift: fine-tuning can make the model explain a behavior no longer its own and render labels stale.This concern is raised by Binder et al. (2024).Source: §D.5 · p. 27
  • With KL regularization, held-out transcript KL is 0.03; the run preserves thinking ability and ordinary-prompt behavior but does not improve counterfactual evaluation.The KL is measured against the base model over the transcript before the follow-up question.Source: §D.5 · p. 27
  • The KL recipe could enable post-hoc self-explanation training without changing production behavior or refreshing labels during training.The penalty holds behavior fixed outside follow-up questions.Source: §D.5 · pp. 27–28

E OPEN-ENDED EXPLANATION TRAINING

  • Open-ended explanation training asks models to explain why they produced a behavior and provide its causes alongside the counterfactual experiments supporting them.Pairing each cause with an experiment trains the model to propose counterfactuals that test its self-explanation, similar to Mayne et al. (2025) and Dehghanighobadi et al. (2025).Source: E OPEN-ENDED EXPLANATION TRAINING · p. 28
  • Open-ended explanation models use the counterfactual-prediction runs’ follow-up-turn structure, datasets, and hyperparameters, and are evaluated on held-out hint settings and investigations.The evaluations correspond to §4.3 and §4.4, respectively.Source: E OPEN-ENDED EXPLANATION TRAINING · p. 28

E.1 TRAINING TARGET … E.4 DISCUSSION: INTERPRETING THE MIXED RESULTS

The paper trains open-ended self-explanations against structured counterfactual investigations, then finds mixed generalization: gains depend on model and evaluation, and available explanation metrics have important limitations.

E.1 TRAINING TARGET

  • The training target is a structured report of observed behavior and rate, supported causes paired with counterfactual evidence, refuted hypotheses with null experiments, and a final narrative.The report’s template separates supported causes from negative evidence.Source: §E.1 · p. 28
  • The range is described incorrectly about 57% of the time with “upper”; errors are about 93% in isolation, 43% with bug-finding instructions, and 73% with an explicit inline explanation.Source: §E.1 · pp. 29–30
  • With “max” and “min,” the model misstates the range about 87% of the time, versus about 3% with neutral “a” and “b”; either bound-like name alone is insufficient.The supported explanation is that the co-occurring parameter names create a semantic prior that overrides the model’s correctly traced mathematics.Source: §E.1 · p. 30
  • The full structured-report target showed no consistent uplift over a short first-person paragraph, so the reported results use the structured target.The authors had hypothesized that the longer report might function as chain-of-thought by giving the model more tokens to reason before answering.Source: §E.1 · p. 30

E.2 GENERALIZATION TO THE HINT SETTING

  • On hint attribution, trained 397B generalizes to the Opus reference, while trained 8B shows no improvement; only trained 397B performs above chance on sycophancy.Models tend to underperform their counterfactual-prediction results in this setting.Source: §E.2 · p. 30, Figure 9 · p. 31

E.4 DISCUSSION: INTERPRETING THE MIXED RESULTS

  • Hint attribution and counterfactual simulatability may understate learning because hint transcripts differ from training data and simulatability depends on simulator capability.Long, messy transcripts can make explanations hard to match to a specific counterfactual; strong simulators already predict well from transcripts alone.Source: §E.4 · pp. 31–32

E.3 COUNTERFACTUAL SIMULATABILITY ON HELD-OUT INVESTIGATIONS

  • The base models’ explanations provide no counterfactual-simulatability uplift; trained explanations help weak simulators but hurt strong ones on both targets.The evaluation tests whether explanations help predict held-out counterfactual outcomes; the authors suggest the contrast with Mayne et al. (2026) may reflect their different task distributions.Source: §E.3 · p. 31, Figure 10 · p. 32, Figure 11 · p. 32

E.4 DISCUSSION: INTERPRETING THE MIXED RESULTS

  • Direct judging of open-ended explanations was unreliable; trained models improved on a single-trigger subset, but the same-pipeline evaluation is circular and only preliminary.The authors identify clean metrics for explanations of messy transcripts with several candidate causes as future work.Source: §E.4 · pp. 32–33

F UNFILTERED DATASET RESULTS

On the fully unfiltered set of counterfactual claims, activation-based interpretability tools still provide no uplift over transcript reading, while counterfactual-prediction training generalizes and slightly exceeds the Opus reference.

F UNFILTERED DATASET RESULTS

  • Removing the concreteness, reproducibility, and single-factor-edit filters expands evaluation claims to 4,433 for Gemma-3-27B-IT, 4,041 for Qwen3-8B, and 3,972 for Qwen3.5-397B.Because the unfiltered data are close to balanced (47–53% true), the evaluation reports threshold-free AUROC.Source: F UNFILTERED DATASET RESULTS · p. 33
  • On the full unfiltered Gemma dataset, activation-based tools still provide no uplift over the transcript-only baseline, with lower absolute performance amid vaguer, less predictable mechanisms.This preserves the no-uplift finding from the filtered-set evaluation.Source: F UNFILTERED DATASET RESULTS · p. 33
  • On unfiltered claims, trained Qwen3-8B and Qwen3.5-397B exceed Opus by paired ∆AUROC +0.015 [+0.002, +0.027] and +0.033 [+0.021, +0.044], respectively.The advantage is concentrated among low-concreteness claims: AUROC is 0.848 versus 0.822 for Qwen3-8B and 0.852 versus 0.811 for Qwen3.5-397B, trained models versus Opus.Source: F UNFILTERED DATASET RESULTS · p. 33, Figure 13 · p. 34

G UNFAITHFUL CHAIN OF THOUGHT IN THE WILD

On reasoning-mode data, interpretability tools again fail to improve counterfactual behavior prediction, while CHIVE surfaces decisions that are misattributed or left unexplained in reasoning traces.

Wild chain-of-thought faithfulness

  • Unlike planted-cue studies, CHIVE finds diverse wild reasoning failures and also surfaces decisions whose important causes are absent from the chain of thought.The section presents qualitative examples and leaves systematic study of these behaviors for future work.Source: G UNFAITHFUL CHAIN OF THOUGHT IN THE… · p. 34
  • The study reruns the pipeline on Qwen3-8B with reasoning enabled and releases Qwen3.5-397B-A17B reasoning investigations as an unanalyzed resource.The rerun uses a 1024-token thinking budget and the same WildChat prompt dataset used elsewhere.Source: G UNFAITHFUL CHAIN OF THOUGHT IN THE… · p. 34

Qualitative examples

  • Moving Phineas and Ferb from first to last in a 14-show list reduced its selection from 30/30 to 4/30 responses, implicating list position rather than fame.The chain of thought attributed the selection to the show’s fame.Source: G UNFAITHFUL CHAIN OF THOUGHT IN THE… · p. 34

Interpretability evaluation

  • Transcript-only AUROC is 0.87 (vs. 0.81 without thinking), yet no interpretability-tool predictor beats that baseline; it remains 0.78 on the unsure subset (65%, n=933).The unsure subset restricts evaluation to claims with transcript-only predicted probabilities in [0.25, 0.75].Source: G UNFAITHFUL CHAIN OF THOUGHT IN THE… · p. 34, Figure 14 · p. 35

Qualitative examples

  • The model described Woodrow Wilson as peace-seeking in 29/30 responses despite the prompt’s contrary framing; 27/30 reasoning traces omitted how it would portray him.Replacing Wilson’s name with a fictional one reduced peace-seeking descriptions to 3/30, suggesting real-world knowledge overrode the request without appearing in the reasoning.Source: G UNFAITHFUL CHAIN OF THOUGHT IN THE… · p. 35

Related papers

Loading 2608.16747v1…