PaperlayerView exact revision on arXiv

Source-linked AI summary

Would this change your answer? Evaluating Explanations of LLM Behavior In The Wild with Counterfactual Experiments

Adam Karvonen, Euan Ong, Subhash Kantamneni, Samuel Marks

arXiv:2608.16747v1cs.LGcs.AI

TL;DR

Evaluating explanations requires knowing whether they predict model behavior under related counterfactual changes. CHIVE investigates naturally occurring behaviors with prompt edits, finding no uplift from tested interpretability tools while producing training data that generalizes broadly.

  • Problem

    Existing explanation evaluations often use narrow settings, leaving limited evidence about explanations for diverse, naturally occurring model behaviors.

  • Method

    CHIVE samples model responses from transcript prompts, identifies unexpected behaviors, and investigates them through counterfactual prompt edits.

  • Results

    Training on CHIVE counterfactual data improves prediction over untrained baselines across held-out investigations, while tested interpretability tools provide no uplift.

  • Takeaways & Limitations

    Counterfactual investigations can produce diverse training and evaluation data at scale for studying model behavior.

  • Takeaways & Limitations

    Because clean counterfactual ground truth is available, CHIVE evaluates explanations through a proxy that may differ from more realistic use cases.

Abstract

from arXiv · show

Many areas of AI research, such as language model interpretability and chain of thought faithfulness, seek to explain model behaviors. But what constitutes a "good" explanation? In this work, we evaluate explanations through the lens of counterfactual simulatability-whether the explanation is useful for predicting model behaviors on related counterfactual inputs. To this end, we introduce CHIVE (Counterfactual Hypothesis Investigation Via Edits), a novel agentic pipeline that identifies unexpected model behaviors in the wild and investigates them with counterfactual prompt edits. This yields thousands of high-quality explanations for naturally-occurring model behaviors along with supporting counterfactual evidence. We apply CHIVE in two ways. First, we evaluate whether common LLM interpretability techniques improve an agent's ability to predict counterfactual model behaviors. Surprisingly, we find no uplift from any of the interpretability techniques studied. Second, we use CHIVE to generate training data. We find that training models to predict outcomes of CHIVE-generated counterfactual experiments generalizes to various out-of-distribution settings. Overall, CHIVE automatically discovers explanations of naturally-occurring LLM behaviors, enabling us to evaluate and improve methods for explaining LLM behaviors.

1 INTRODUCTION

The paper evaluates explanations by whether they predict model behavior on related counterfactual inputs and introduces CHIVE, an agentic pipeline for discovering and explaining unexpected behaviors at scale. It applies CHIVE to test interpretability tools and generate training data, finding no uplift from the studied tools and broader generalization from CHIVE-generated data.

  • Motivation: The paper evaluates explanations through counterfactual simulatability: whether they help predict model behavior on related counterfactual inputs.The explanations are behavioral claims testable by counterfactual experiments, rather than accounts of internal computation or training data.
  • CHIVE: CHIVE discovers unexpected model behaviors at scale by sampling prompts, screening responses, and investigating anomalies with counterfactual prompt edits.The pipeline can operate on prompts from an arbitrary distribution and produces thousands of explanations when run on real user conversations.
  • Evaluation: Three activation-reading interpretability tools provide no uplift in predicting counterfactual outcomes, despite providing uplift in prior auditing games on fine-tuned models.The evaluation compares a predictor given tool outputs with a baseline predictor that sees only the transcript.
  • Training: Training models on CHIVE-generated data generalizes to held-out settings and prompt sources, while training models to explain their behavior gives mixed results.This extends beyond the narrow generalization typically reported in hint-based settings.
  • Resources: The authors release CHIVE’s evaluation and training datasets, along with code, models, and investigation runs on five target models.The listed target models include Qwen3-8B, Qwen3-32B, Qwen3.5-397B-A17B, Gemma-3-27B-IT, and Llama-3.1-8B.

2 A PIPELINE FOR COUNTERFACTUAL-GROUNDED EXPLANATIONS OF IN-THE-WILD BEHAVIORS

CHIVE discovers unexpected in-the-wild behaviors, investigates them through prompt edits and resampling, and verifies the resulting explanations with an independent judge. Its main evaluation data are measured counterfactual claims, filtered for coherent mechanisms, limited confounds, and reproducibility.

  • CHIVE pipeline: CHIVE samples 30 responses per prompt, screens for unexpected behavior, runs 5–15 counterfactual experiments, and has an independent judge score evidential support.Prompts primarily come from WildChat, lightly augmented with approximately 9% agentic transcripts.
  • Implementation: A complete investigation costs roughly $1–2 in API calls, while Qwen3.5-397B-A17B can produce successful investigations at less than 10% of Opus’s cost.The pipeline runs target models in non-thinking instruct mode for consistency across models.
  • Investigation outputs: Each investigation produces an open-ended causal explanation and concrete counterfactual experiments with measured outcomes, but explanations are not treated as ground truth.Counterfactual experiments provide the evaluation signal, whereas explanations may be partially incorrect or omit important drivers.
  • Counterfactual labels: Up to two true and two false claims per investigation assert a prompt edit changes behavior by at least 30 percentage points; true and false labels require shifts of at least 50 and at most 15 points, respectively.Labels are assigned from observed counterfactual resampling outcomes rather than investigator judgment.
  • Dataset construction: Three LLM-judge-based filters target mechanism coherence, confounds, and reproducibility because concrete behavioral changes can still reflect unstable or confounded effects.The reported main-body results use the surviving evaluation dataset, while removing the filters does not qualitatively change the takeaways.

3 APPLICATION 1: EVALUATING INTERPRETABILITY TOOLS

The evaluation tests whether interpretability tools improve an agent’s ability to predict target-model behavior under counterfactual edits. Across both target models, activation-based tools provide no uplift over a transcript-only baseline, despite sometimes changing predictions and rarely stating explicit causal relationships.

  • Evaluation setup: The predictor estimates whether counterfactual claims are true from the transcript, optionally using interpretability tools, and is scored by accuracy and AUROC.This evaluates counterfactual simulatability: whether an explanation helps predict behavior under counterfactual edits.
  • Main result: Across both target models, none of the three activation-based tool predictors improves on the transcript-only baseline.The targets are Qwen3-8B and Gemma-3-27B-IT; main-body predictions use Opus 4.8 with read-only access to target-model internals.
  • Robustness: The no-uplift result persists across predictor families, evaluation hyperparameters, and attempts to elicit stronger tool use.Using GPT-5.5 or Gemini-3.1-Pro instead of Opus 4.8 produces the same qualitative results.
  • Mechanism: Tool outputs sometimes inform predictions, but gains are offset when outputs omit relevant information or actively mislead the predictor.Examples include tool outputs revealing a memorized association, alongside cases where the relevant information is not surfaced.
  • Causal content: Only 0.6% of claims whose edits change behavior receive NLA outputs explicitly stating that feature X causes behavior Y.Tool outputs usually describe both the edited feature and behavior, but rarely state the relationship between them.

4 APPLICATION 2: TRAINING MODELS TO PREDICT THEIR OWN BEHAVIOR

CHIVE investigations provide training data for models to predict how prompt edits affect their own behavior. Counterfactual-prediction training generalizes from narrow hint tasks to held-out in-distribution and out-of-distribution investigations, while open-ended explanation training is mixed and shows no privileged-access benefit.

  • Training setup: CHIVE investigations yield verified counterfactual experiments that can train models to predict the effects of prompt edits on their own behavior.The training task poses a counterfactual claim as a Yes-or-No follow-up on the model’s own transcript.
  • Generalization: Both trained models significantly improve accuracy and AUROC over the untrained base model on held-out hint settings, approaching or matching the Opus reference.Evaluation uses Reddit AITA and MMLU prompts with planted cues, despite no training targeted at the hint setting.
  • Generalization: Both trained models improve substantially over the base model on held-out investigations and land within ±0.03 AUROC of the Opus reference across every evaluated set.The held-out claims come from WildChat and the out-of-distribution PETRI prompt source.
  • Open-ended explanations: Open-ended explanation training generalizes to the hint setting for the 397B target but not the 8B target, while held-out explanation quality improves weak simulators but misleads stronger ones.For the 397B target, cue attribution accuracy rises from 0.59 to 0.68, reaching the Opus reference at 0.64.
  • Privileged access: Cross-trained models match self-trained models, providing no evidence that counterfactual-prediction training uses privileged access to internal state.This comparison is reported for Qwen3-8B and Llama-3.1-8B-Instruct trained on claim data about themselves or the other model.

5 DISCUSSION AND LIMITATIONS

The discussion finds that counterfactual investigation is broadly reusable, while interpretability tools provide no uplift for naturally occurring behaviors. It also frames this result as limited by clean ground-truth counterfactuals, activation-only methods, and read-only evaluation.

  • Interpretability results: Interpretability tools show no uplift on natural behaviors, contrasting with auditing settings where narrow fine-tuning creates anomalous content that simpler detection strategies can exploit.The authors suggest natural behaviors do not produce anomalous activation-tool outputs.
  • Generality: Counterfactual investigation runs unchanged on other transcript sources, and its screening can target behavior classes such as evaluation awareness, sycophancy, or unfaithful reasoning.The PETRI claims required no pipeline modification.
  • Limitations: Because the studied behaviors have clean ground-truth counterfactuals, anyone with sampling access can run the counterfactual directly, making the evaluation a proxy for real use cases.The authors say this limitation is shared with other behavioral-explanation evaluations and that the null result mainly bears on harder cases without clean counterfactuals.
  • Limitations: All three evaluated tools are activation-based, but activations may be inherently incomplete for causal claims or current tools may fail to recover relationships present in the activations.The authors note that weight- or circuit-level access might be required for some causal relationships.
  • Limitations: Read-only access leaves the achievable prediction ceiling unknown because some counterfactuals may not be predictable without being run.Prompt editing runs the ground-truth experiment, while internal interventions such as steering or activation patching approximate it.

6 CONCLUSION

Counterfactual investigations can produce diverse, realistic data about the causes of model behavior at scale. This offers a general approach for constructing training and evaluation datasets.

  • Counterfactual investigations produce diverse, realistic data about the causes of model behavior at scale.
  • The approach addresses a lack of diverse, realistic data limiting many areas of AI safety.
  • The resulting data can support construction of both training and evaluation datasets.

AUTHOR CONTRIBUTION STATEMENT … A EXTENDED DISCUSSION

The supplied material consists mainly of an author-contribution statement and an appendix contents outline covering extended discussion, pipeline and dataset details, interpretability-tool evaluation, counterfactual-prediction training, and open-ended explanation training. The author statement credits Adam Karvonen with the project’s conception, experiments, and paper, with contributions from Euan Ong, Subhash Kantamneni, and Samuel Marks.

  • AUTHOR CONTRIBUTION STATEMENT: Adam Karvonen proposed the project, ran all experiments, and wrote the paper, while Euan Ong and Subhash Kantamneni provided discussion and feedback and Samuel Marks supervised.
  • A Extended discussion: The extended discussion covers the evaluation proxy argument and the distinction between hypothesis generation and hypothesis discrimination.
  • A EXTENDED DISCUSSION: The listed appendix materials span extended discussion, pipeline and dataset details, interpretability-tool evaluation, counterfactual-prediction training, and open-ended explanation training.
  • B Pipeline and dataset details: Pipeline and dataset details include the investigation pipeline, prompt sources, evaluation-dataset construction and filters, evaluation-claim format, and data-generation costs.
  • APPENDIX CONTENTS: The appendix contents also list sections on unfiltered dataset results and unfaithful chain of thought in the wild.
  • C Interpretability-tool evaluation: details and additional results: Interpretability-tool evaluation details cover tool calls, Qwen3-8B results, cross-model predictor replication, exploratory NLA-output analysis, and qualitative examples.
  • D Counterfactual prediction training: details and additional results: Counterfactual-prediction training details address training targets and formats, hint-setting dataset construction, false- and true-positive rates, privileged access, and KL regularization.
  • E Open-ended explanation training: Open-ended explanation training covers the training target, generalization to the hint setting, counterfactual simulatability on held-out investigations, and mixed-results interpretation.

A.1 THE EVALUATION PROXY ARGUMENT … B.4 EVALUATION CLAIM FORMAT

The paper frames counterfactual simulatability as a proxy for evaluating explanations, while acknowledging that direct target-model access makes evaluation performance intrinsically limited. It operationalizes this proxy through CHIVE’s investigation pipeline, filtered datasets, and standardized counterfactual claims.

  • A.1 THE EVALUATION PROXY ARGUMENT: A proxy with clear ground truth can test explanation reliability on questions shaped like real model-behavior investigations, despite not being directly useful for answering them.Its validity depends on overlap between prompt-counterfactual factors and the factors interpretability tools address.
  • A.2 HYPOTHESIS GENERATION VS. HYPOTHESIS DISCRIMINATION: Interpretability tools may generate hypotheses in some debugging workflows, but explaining a specific behavior requires discriminating among hypotheses rather than merely proposing them.The paper distinguishes this target from scanning transcripts or generating hypotheses about fine-tuned model organisms.
  • A.2 HYPOTHESIS GENERATION VS. HYPOTHESIS DISCRIMINATION: In all model–behavior pairs, the discovered hypothesis appeared within the first three guesses in at least 60% of samples, usually as the first guess.The paper argues that hypothesis discrimination matters because frontier models can often generate approximately correct hypotheses within 10-100 guesses.
  • B.1 DETAILED INVESTIGATION PIPELINE DESCRIPTION: CHIVE separates sampling, screening, investigation, and verification across distinct models, with the investigator using edited-prompt resampling to test behavioral explanations.The classifier reports each counterfactual rate relative to a baseline graded with the same frozen question.
  • B.2 PROMPT SOURCES: The pipeline uses a WildChat-based mixture for training and in-distribution evaluation, while PETRI transcripts provide the out-of-distribution source.The WildChat mixture is 91.3% WildChat, 5.2% ToolACE, 2.5% SystemChat-2.0, and 1.0% Hermes.
  • B.3 EVALUATION DATASET CONSTRUCTION AND FILTERS: Evaluation construction produces counterfactual investigations, extracts four claims per investigation, and applies filters for mechanism concreteness, reproducibility, and unconfounded interventions.The single-factor filter retains ∼90% of claims, while positive claims require at least a 50pp behavioral change.
  • B.4 EVALUATION CLAIM FORMAT: Each claim compares a baseline behavior rate with an intervention and asks whether the intervention changes occurrence by at least 30 percentage points in either direction.The same natural-language format is used for interpretability evaluation and the training application.
  • B.4 EVALUATION CLAIM FORMAT: In the randomNum example, renaming function parameters while leaving the code unchanged reduced the behavior from 26/30 to 1/30 responses, an 83-point change.Because this exceeded the 30-point threshold, the claim was labeled Yes.

B.5 DATA GENERATION COSTS … C.3 CROSS-MODEL PREDICTOR AGENT REPLICATION

CHIVE data generation costs about $1.9 per verified evaluation investigation and $0.9 per training investigation, with prompt caching substantially reducing investigation-stage costs. Across target models, tool configurations, and predictor agents, interpretability tools provide no counterfactual-prediction uplift, with Gemini harmed by SAE and NLA tools.

  • B.5 DATA GENERATION COSTS: $1.9 per investigation is the evaluation-data cost, with each 4,714-prompt run costing roughly $3.3k and yielding 1,700–2,000 verified investigations.The investigation stage consumes ∼470k input tokens per investigation, but 80% of billed input tokens are cache reads, reducing its cost to $0.96 instead of ≈$2.6 uncached.
  • B.5 DATA GENERATION COSTS: $0.9 per investigation is the training-data cost for the Opus-investigated Qwen3-8B run, totaling ≈$26k for 28,330 investigations.The cheaper regime uses fewer counterfactual experiments, n=10 rather than n=30 responses for behavior rates, and one judge sample rather than five for verification.
  • B.5 DATA GENERATION COSTS: No cost is reported for Qwen3.5-397B-A17B self-investigation because deployment, caching support, and cache-hit rates determine the cost.Different investigators use similar token counts because each stage uses a consistent 4,096-token thinking budget.
  • C.1 TOOL CALL DETAILS: Each tool agent can make up to 5 tool calls on the target model before answering, with tool outputs differing by interpretability method.The activation oracle asks separate natural-language questions about chosen activation segments, while NLA and SAE calls return outputs for selected token positions.
  • C.1 TOOL CALL DETAILS: No tool hyperparameter setting produces uplift: varying the call budget from 2 to 10 or positions per call from 16 to 64 leaves paired ∆AUROC confidence intervals straddling zero.The sweep uses Gemma-3-27B-IT with Opus 4.8 on the final evaluation dataset, with n = 1,294.
  • C.2 QWEN3-8B INTERPRETABILITY TOOL RESULTS: No tool improves on the transcript-only baseline for Qwen3-8B, extending the no-uplift result across both target models and all three activation-based tool families.The replication uses the same evaluation-dataset construction and hyperparameters as the Gemma-3-27B-IT evaluation.
  • C.3 CROSS-MODEL PREDICTOR AGENT REPLICATION: No predictor model obtains tool uplift in the cross-model replication using GPT-5.5 and Gemini-3.1-Pro on the Gemma-3-27B-IT claims.The evaluation uses the same claims, an identical tool interface, and high reasoning effort; n = 1,294.
  • C.3 CROSS-MODEL PREDICTOR AGENT REPLICATION: Gemini receives significant downlift from the SAE and NLA tools, unlike GPT-5.5 and Opus 4.8.Table 5 reports AUROC per agent and shows that no tool beats its own baseline for any agent.

C.4 EXPLORATORY ANALYSIS OF NLA OUTPUTS … D.3 HINT-SETTING COUNTERFACTUAL PREDICTION: FALSE AND TRUE POSITIVE RATES

The paper finds that interpretability outputs rarely expose causal relationships, often shift predictions toward no effect, and can either help or mislead in individual cases. It then describes counterfactual-prediction training and hint-setting evaluations, where the two target models fail in opposite directions.

  • C.4 EXPLORATORY ANALYSIS OF NLA OUTPUTS: NLA outputs explicitly state that edited factors cause behaviors in only 0.6% of positive claims, providing near-zero direct causal evidence.This corresponds to 3 of 479 positive claims and 0.02% of selected NLA generations.
  • C.4 EXPLORATORY ANALYSIS OF NLA OUTPUTS: 47% of positive claims contain supporting NLA clues versus 30% of negative claims, but clues achieve only 0.60 AUROC versus 0.77 for transcript-only prediction.Adding clue rates changes transcript-only AUROC by only +0.001.
  • C.4 EXPLORATORY ANALYSIS OF NLA OUTPUTS: Each tool flips 8–16% of claims, with NLA flips moving toward no-effect predictions for 82% of positive and 84% of negative claims.The analysis uses the 1,294-claim Gemma evaluation dataset.
  • C.5 QUALITATIVE INTERPRETABILITY TOOL USE EXAMPLES: Qualitative examples show that NLA outputs sometimes help by revealing model representations or memorized associations, but can also mislead or omit the causal relationship.One successful intervention changes the prediction from 0.50 to 0.25 and reaches the correct False answer.
  • D.1 TRAINING DETAILS AND TARGET FORMATS: Training examples append a follow-up turn to the model’s transcript, apply loss only to the final assistant turn, and use single-token Yes/No targets for counterfactual prediction.Counterfactual-prediction datasets are balanced 50/50 between true and false claims.
  • D.1 TRAINING DETAILS AND TARGET FORMATS: The models are fine-tuned with LoRA for one epoch on counterfactual prediction and three epochs on open-ended explanation data, using shared hyperparameters otherwise.Qwen3-8B models are trained locally, while Qwen3.5-397B-A17B models are trained through the Tinker API.
  • D.2 HINT-SETTING DATASET CONSTRUCTION: Hint-setting evaluations sample each cue condition 10 times at temperature 1.0, retain questions with at least 60% cue-consistent cued responses, and balance each dataset at 150 positive and 150 negative cases per model.The evaluations cover sycophancy on Scruples AITA posts and five cue types applied to MMLU questions, using freeform responses.
  • D.3 HINT-SETTING COUNTERFACTUAL PREDICTION: FALSE AND TRUE POSITIVE RATES: The base 8B almost always answers Yes, with FPR = TPR ≈1, whereas the 397B rarely answers Yes after training and has absolute accuracy below its AUROC-supported level.Thus, the two targets fail in opposite directions on hint-setting counterfactual prediction.

D.4 PRIVILEGED ACCESS EXPERIMENT · D.5 PRESERVING THE TARGET MODEL’S BEHAVIOR WITH KL REGULARIZATION · E OPEN-ENDED EXPLANATION TRAINING

The privileged-access experiment finds no self-predictor advantage, while KL regularization largely preserves the target model’s behavior during self-explanation training. The paper also extends training to open-ended explanations that include both behavioral causes and supporting counterfactual experiments.

  • D.4 PRIVILEGED ACCESS EXPERIMENT: D.4 PRIVILEGED ACCESS EXPERIMENT: The comparison trains self and cross predictors on identical, size-matched claim data containing 23,944 examples.Each predictor receives the same evidence about the target model, with shared training choices, hyperparameters, and data subsample.
  • D.4 PRIVILEGED ACCESS EXPERIMENT: D.4 PRIVILEGED ACCESS EXPERIMENT: Self predictors do not outperform cross predictors for either Qwen3-8B or Llama-3.1-8B-Instruct.All four trained models approach or match the Opus reference.
  • D.4 PRIVILEGED ACCESS EXPERIMENT: D.4 PRIVILEGED ACCESS EXPERIMENT: The authors suggest privileged-access effects may be fragile or dependent on the broader task distribution.They contrast their result with Li et al. (2026), who found privileged access in a narrower hint setting using the same model pair.
  • D.5 PRESERVING THE TARGET MODEL’S BEHAVIOR WITH KL REGULARIZATION: D.5 PRESERVING THE TARGET MODEL’S BEHAVIOR WITH KL REGULARIZATION: Self-explanation training can cause behavioral drift, making the fine-tuned model increasingly unlike the model that generated its labels.Drift also makes labels stale in iterated training procedures.
  • D.5 PRESERVING THE TARGET MODEL’S BEHAVIOR WITH KL REGULARIZATION: D.5 PRESERVING THE TARGET MODEL’S BEHAVIOR WITH KL REGULARIZATION: A KL penalty to the base model is applied on 1% of training samples alongside follow-up-answer SFT loss.The penalty uses full-vocabulary next-token distributions on WildChat general-chat data.
  • D.5 PRESERVING THE TARGET MODEL’S BEHAVIOR WITH KL REGULARIZATION: D.5 PRESERVING THE TARGET MODEL’S BEHAVIOR WITH KL REGULARIZATION: 0.03 exact full-vocabulary KL is measured before the follow-up question, while the run preserves thinking ability and essentially unchanged ordinary-prompt behavior.This keeps self-explanations targeted at very nearly the original model.
  • E OPEN-ENDED EXPLANATION TRAINING: E OPEN-ENDED EXPLANATION TRAINING: Models are trained to explain why a behavior occurred by producing its causes together with counterfactual experiments that support them.Because each cause is paired with an experiment, the model learns to propose counterfactuals that test those causes.
  • E OPEN-ENDED EXPLANATION TRAINING: E OPEN-ENDED EXPLANATION TRAINING: Evaluation uses the same held-out hint and investigation settings as counterfactual prediction, with matching follow-up structure, datasets, and hyperparameters.The settings are the held-out hint settings of §4.3 and held-out investigations of §4.4.

E.1 TRAINING TARGET … E.4 DISCUSSION: INTERPRETING THE MIXED RESULTS

The paper trains models to produce structured causal explanations supported by counterfactual experiments, but these explanations show mixed generalization: the 397B model transfers to hint attribution, while simulatability gains depend on simulator strength. Evaluation is complicated by distribution shift, simulator capability, unreliable judging, and circular verification.

  • E.1 TRAINING TARGET: The training target is a structured report containing the observed behavior and rate, supported causes with counterfactual evidence, refuted hypotheses with null experiments, and a narrative answer.The authors also tested a short first-person paragraph, but found no consistent uplift from the full structured report.
  • E.1 TRAINING TARGET: About 87% of the time, the model incorrectly summarizes randomNum(max, min) as [min, max), despite correctly tracing the mathematics and deriving [min, max+min).This example illustrates a behavior where the model’s final summary contradicts its own step-by-step reasoning.
  • E.1 TRAINING TARGET: Renaming both parameters to neutral names raises correct range identification to about 97%, versus only about 13% with max and min, implicating their co-occurrence.Replacing only max or min yields about 83% and about 93% correctness, respectively; descriptive names yield about 90%.
  • E.1 TRAINING TARGET: Names suggesting upper and lower bounds produce intermediate error rates, with upper/lower causing incorrect claims about 57% of the time and range/offset about 20%.The results support a broader naming prior tied to boundary semantics rather than only the literal names max and min.
  • E.2 GENERALIZATION TO THE HINT SETTING: In the hint setting, trained models generally underperform their counterfactual-prediction results: the trained 397B reaches the Opus reference, whereas the trained 8B shows no improvement and remains below it.Opus is strongest on MMLU, but on sycophancy it always attributes the verdict to the user’s stated opinion.
  • E.3 COUNTERFACTUAL SIMULATABILITY ON HELD-OUT INVESTIGATIONS: Across five simulators ranging from Gemma-3-12B to Opus 4.8, training does not consistently improve counterfactual simulatability for either Qwen3-8B or Qwen3.5-397B-A17B.Trained explanations help weak simulators but hurt strong ones because confident, specific counterfactuals are often wrong and mislead the simulator.
  • E.4 DISCUSSION: INTERPRETING THE MIXED RESULTS: The mixed results may understate learning because hint transcripts differ sharply from training transcripts, while simulatability depends on simulator capability and messy transcripts may not align explanations with specific counterfactuals.Strong simulators already predict well from transcripts alone, requiring substantial training improvement before explanations can provide uplift.
  • E.4 DISCUSSION: INTERPRETING THE MIXED RESULTS: LLM judging of open-ended explanations is unreliable, so the authors evaluated single-trigger investigations; trained models improved there, but the measure is circular and preliminary.The training data and evaluation are generated by the same LLM pipeline, motivating cleaner metrics for messy transcripts with multiple candidate causes.

F UNFILTERED DATASET RESULTS

Evaluations on fully unfiltered counterfactual claims confirm the main findings: activation-based interpretability tools provide no uplift, while trained counterfactual-prediction models generalize and slightly exceed the Opus reference. Removing filters substantially expands the evaluation sets, though performance is somewhat lower because the added claims contain vaguer mechanisms.

  • Overall robustness: Together, the unfiltered evaluations show that the two main results are not artifacts of the dataset filters.The appendix reruns both evaluations on the fully unfiltered set of counterfactual claims.
  • Unfiltered dataset: Removing all filters expanded the evaluation from 1,294 to 4,433 claims for Gemma-3-27B-IT, 1,497 to 4,041 for Qwen3-8B, and 1,076 to 3,972 for Qwen3.5-397B.The main-body evaluation filtered for mechanism concreteness, counterfactual reproducibility, and single-factor edits; concreteness removed one-third to two-thirds of claims.
  • Interpretability evaluation: Activation-based interpretability tools still provide no uplift over the transcript-only baseline on the fully unfiltered Gemma dataset.Absolute performance is a few points lower than on the filtered set because unfiltered claims contain more vague, less predictable mechanisms.
  • Counterfactual-prediction evaluation: On the unfiltered bank, trained counterfactual-prediction models exceed the Opus reference by +0.015 [+0.002, +0.027] for Qwen3-8B and +0.033 [+0.021, +0.044] for Qwen3.5-397B.The models reached the Opus reference on the filtered dataset, but were slightly above it when evaluated without filters.

G UNFAITHFUL CHAIN OF THOUGHT IN THE WILD

On reasoning-mode data, interpretability tools still provide no uplift over transcript-only prediction, while counterfactual investigations reveal causes and actions that chain-of-thought does not faithfully capture.

  • 0.87 vs. 0.81: transcript-only AUROC is higher with visible reasoning than in the non-thinking evaluation.The higher baseline score does not imply saturation because the result also holds on a harder subset.
  • List position, not stated fame, causes the model’s repeated selection of Phineas and Ferb.It is selected in 30/30 responses when first and becomes less selected when moved to the end.
  • No tool predictor beats the transcript-only baseline on reasoning-mode data, reproducing the interpretability null result.The evaluation was rerun on Qwen3-8B with reasoning enabled.
  • The model writes the real, peace-seeking Woodrow Wilson in 29/30 alternate-history responses despite the prompt describing him as a fascist dictator.In 27/30 traces, the reasoning says nothing about Wilson’s portrayal; the remaining three assert his historical neutrality without addressing the prompt’s contradiction.
Loading 2608.16747v1…