PaperlayerView exact revision on arXiv

Source-linked AI summary

Train the Model, Not the Reader: Decodability Supervision for Verifiable Activation Explanations

Hiskias Dingeto

arXiv:2607.20379v1cs.AIcs.CL

TL;DR

Reconstruction scores can reward gist or private codes rather than faithful individual claims. The paper audits these failures and introduces RECAP, which makes designated content probe-decodable and improves claim-faithful verbalization.

  • Problem

    Reconstruction-based explanation tests reward sufficiency but do not penalize false additions when alternative claim values produce the same reconstruction.

  • Method

    The paper combines grounded-vs-true and evaluator-swap audits with RECAP, co-training linear predictors to preserve designated content for independent probing.

  • Results

    AUC 0.965 versus 0.823 shows probes rank verbalizers’ true claims above false ones, while RECAP eliminates private codes in 5/5 sandbox runs.

  • Takeaways & Limitations

    High reconstruction does not certify individual claims; independent probes provide a more reliable check of designated internal content.

  • Takeaways & Limitations

    Evidence is limited to synthetic audits, one released Qwen-2.5-7B verbalizer, and continued pretraining of one small Pythia-160M model; frontier scale is untested.

Abstract

from arXiv · show

Natural-language autoencoders score explanations of hidden activations by reconstruction: an explanation is deemed faithful if the activation can be regenerated from it. The test is structurally insensitive to individual false claims: if flipping a claim does not change the reconstruction, the claim is never penalized. We show the test is passed in two ways, neither faithful. On a released Qwen-2.5-7B verbalizer, explanations reconstruct well above chance while ~2% of specific claims are reconstruction-dependent, so the score tracks gist, not specific facts. Under exact synthetic ground truth, the standard recipe develops co-adapted private codes (false wording the reconstruction depends on) in 5/5 runs, and fixes that leave the target model unchanged do not help. We contribute two audit protocols, the grounded-vs-true cross and the evaluator swap, and RECAP (Readable Encodings via Co-trained Auxiliary Predictors): linear heads trained alongside the target model to keep designated content decodable. On RECAP-trained sandbox models, fresh verbalizers state the designated content truly and the codes vanish, at a +0.001-nat cost. This replicates on a pretrained Pythia-160M: the content becomes reliably probe-decodable, though a fresh verbalizer conveys it only in part (truth 0.44-0.46 vs a near-zero control). For interpretability, high reconstruction does not certify individual claims. For AI safety, RECAP makes designated internal content independently checkable against probes rather than asserted by prose a model can game: an independent probe scores the verbalizer's true claims above its false ones (AUC 0.96, vs 0.82 without RECAP). Against an adversary that edits an explanation to maximize the reconstruction score while lying (suppressing ~87% of its lie penalty), the RECAP probe still flags the lies (AUC 0.95) while the control probe collapses to chance (0.51).

1 Introduction

Reconstruction-based natural-language autoencoder scores test sufficiency, not whether individual claims are true. The paper shows this gap through gist-level scoring and co-adapted private codes, then introduces audits and RECAP to make designated content independently decodable.

  • The structural gap: Reconstruction rewards explanations that suffice to regenerate an activation but does not penalize claims that do not affect reconstruction.This creates a structural gap between high reconstruction and per-claim faithfulness.
  • Two ways to pass: ˜r = 0.84 reconstruction on a released Qwen-2.5-7B-based NLA coincides with only ∼2% of salient specific claims detected as grounding-dependent.The flip audit provides a sensitivity-limited lower bound, indicating that the score can track gist rather than specific facts.
  • Two ways to pass: 5/5 synthetic runs develop co-adapted private codes: false wording that the reconstruction depends on despite exact ground truth.The grounded-vs-true cross and evaluator swap are introduced to detect and quantify these codes.
  • RECAP: RECAP co-trains linear heads on external targets with the target model, so fresh NLAs state designated sandbox content faithfully and the code signature disappears.The intervention that survives the paper’s controls trains the target model rather than the activation reader.
  • Audits: The paper contributes a counterfactual, claim-level audit with per-edit validity controls for exposing ungrounded specific claims in reconstruction-scored explanations.The audit demonstrates that a released verbalizer can pass the reconstruction test even when its specific claims are largely ungrounded.

2 Preliminaries and Experimental Setup

The paper defines claim-level grounding by measuring reconstruction sensitivity to minimal claim flips, then tests whether this criterion tracks computational sensitivity and truth across three experimental settings. It distinguishes gist from specific facts and uses probes to assess decodability independently of reconstruction.

  • NLA framework: A natural-language autoencoder pairs an activation verbalizer that generates explanation z from hidden activation h with a reconstructor that estimates h from z.The activation verbalizer is trained against reconstruction reward after cold start while target-model weights remain frozen.
  • Audit definitions: Grounding is the reconstruction drop caused by minimally flipping claim c alone, with c grounded when Δr̃(c) ≥ τ.Edit validity is checked per edit because changing additional content can inflate apparent grounding severalfold.
  • Audit definitions: The paper separates grounding from truth, classifying each claim along both axes while distinguishing gist—the input’s overall meaning—from specific facts.Faithfulness is defined at the claim level: whether stated content is true of the activation’s source.
  • Evaluation terminology: A probe is a freshly fitted linear decoder, whereas the reconstructor is AR; content is decodable when a fresh probe can read it and designated when an auxiliary predictor is co-trained.These terms support comparing independent probe decodability with reconstruction-based evaluation.
  • Experimental settings: Experiments cover a released Qwen-2.5-7B layer-20 NLA, an exact-ground-truth synthetic sandbox, and continued pretraining of Pythia-160M with self-supervised heads.The Qwen audit contains n=1,517 claims; the sandbox makes audit quantities computable without a judge, and Pythia uses held-out language-modeling loss.

3 Auditing Reconstruction-Scored Explanations

The audit finds two distinct failures: released-system scores track gist because specifics are missing from the tap, while sandbox scores can rely on false, pair-private wording conventions. Grounded-vs-true crosses and evaluator swaps expose these failures, including a separate gap where present content is not truthfully verbalized.

  • Released system: ˜r = 0.84 reconstruction coexists with grounding in only about 2% of LLM-salient specific claims under minimal-pair flips.Rates are 4.2% / 2.1% / 1.6% at τ = 0.02 / 0.05 / 0.10; paraphrase preserves reconstruction, while masking content words collapses it.
  • Sandbox ground truth: 5/5 sandbox runs pass at ˜r 0.71–0.86 while truthful reporting varies widely across decodability conditions.Fully decodable slots are stated truly at 0.44–0.96, partially decodable slots at 0.15–0.26, and unread slots at chance, showing that the test does not distinguish truthful claims.
  • Sandbox ground truth: 0.30–0.80 grounded-minus-true gaps reveal co-adapted private codes in 5/5 standard-recipe runs across both domains.The grounded-vs-true cross detects wording that reconstruction depends on even though it is false; codes can occupy several slots simultaneously.
  • Evaluator swap: 0.53–0.89 code-slot grounding collapses to 0.08–0.23 under an independent evaluator, while grounding of true claims survives.The cross-grader ˜r gap is 0.17–0.38 on standard runs versus ∼0.003 on RECAP runs, locating the passing score’s evaluator-specific convention.
  • Readout gap: 55–72% target-model decodability yields only 15–26% truthful statements in domain 2, exposing a readout gap beyond storage failure.The standard-recipe verbalizer under-reads content that is present in the activation, not merely content absent upstream.

4 Representation Collapse and Probe Drift

The target-side reconstruction objective removes decodable content without changing perplexity, while continued training can make matched probes fail despite preserving exactly recoverable information. Probe monitoring during training therefore requires recalibration, which restores access but does not ensure the stored content is correct.

  • Target-side collapse: At every tested strength and across 3/3 seeds, moving activations toward their reconstructions removes decodable slot content while leaving perplexity unchanged.Freshly retrained probes also lose accuracy, indicating that the information itself—not merely its probe-readable coordinates—has disappeared.
  • Probe drift: Within 1,500 continued-training steps, probes matched to the training model decay to near-chance in 3/3 seeds, although the information remains intact and exactly recoverable.Across seeds, probes are at chance in 6/6 pairs.
  • Probe drift: A label-free linear map fitted on paired activations restores native probe performance in 9/9 cases.The finding shows that continued training can change probe-accessible coordinates without destroying the underlying information.
  • Deployment consequence: Probe-based monitoring of a model that is still training must include explicit recalibration, because realignment restores access to stored content but does not establish that the content is correct.Realignment adjusts coordinates, whereas RECAP changes what those coordinates represent.

5 The RECAP Objective

RECAP trains linear auxiliary predictors with the target model so designated content remains independently decodable from hidden states, rather than optimizing only a verbalizer against frozen activations. This supervision yields truthful fresh verbalizations in sandbox models and transfers to Pythia-160M, though generation remains incomplete and decodability must be maintained during fine-tuning.

  • Objective: RECAP adds auxiliary heads predicting external targets from selected hidden states, and adds their loss to the language-modeling objective.Unlike Jread, which leaves the target model frozen, Jmodel trains the target model on ground truth needed by the verbalizer and reconstructor.
  • Sandbox results: 100% decodability was reached for designated content at +0.001 nats in 5/5 sandbox runs, while unread targets stayed at chance.Independently fitted probes evaluated tuple-disjoint or held-out splits, and the control confirms that causally excluded content cannot be encoded.
  • Sandbox results: ∼1.00 truth and grounding were achieved by fresh NLAs on RECAP targets, with remaining false content at grounding ∼0 and no code signature.The comparison used identically trained NLAs, isolating target-model decodability as the difference.
  • Transfer to Pythia-160M: 0.95–1.00 fresh-probe AUC transferred to Pythia-160M at K=8 and K=64, versus a 0.77–0.80 control baseline.At K=512, a naive presence head minimized training loss while making nothing decodable to a fresh probe; robustness held across head capacity and tap choice, including 0.95–0.98 AUC at +0.01 nats.
  • Propagation and limitations: 0.44–0.46 stated-word truth was stable across three RECAP seeds, versus 0.00–0.07 for controls, but the verbalizer’s precision remained 0.44–0.63 versus ∼0.80 for a same-tap probe.A stronger reconstructor still favored RECAP on truth, 0.57 versus 0.15, and recall, 0.47 versus 0.04.
  • Propagation and limitations: +0.010 nats of continuous maintenance, or intermittent maintenance at about one-tenth the compute, is needed because decodability erodes within a few hundred head-free fine-tuning steps.The result favors training decodability in from the start rather than restoring it later.

6 Probe-Based Monitoring

RECAP enables independent probe-based monitoring of verbalizer claims, eliminating false pre-tap claims in the sandbox and substantially improving detection on Pythia-160M. The monitor remains effective against report-space adversaries that manipulate reconstruction scores, while control monitoring can collapse below chance.

  • Sandbox monitoring: Fresh per-slot probes flag claims contradicting confident internal readouts, with exact ground truth scoring detector performance.Claims are flagged when a probe reads the tap activation with p>0.9 and contradicts the verbalizer.
  • Sandbox monitoring: 27 of 733 false pre-tap claims are caught by the standard target, while RECAP leaves no false pre-tap claims and eliminates abstention.The standard detector has zero false alarms but low recall because many slots are insufficiently decodable; post-tap claims are abstained on both models.
  • Real-model monitoring: 0.965 AUC ranks true above false claims on Pythia-160M, versus 0.823 for the control; false-claim detection reaches 0.93 recall at 0.87 precision.The AUC estimates use 672 held-out claims, and the flag threshold was set on a disjoint calibration split.
  • Adversarial monitoring: 0.022 is the lie penalty for a score-optimal false edit to a stated true word, compared with 0.173 for the unmanipulated comparison condition.The passage reports the RECAP probe’s AUC declining from 0.993 to 0.952 across adversary rungs as semantically nearer absent words are selected.
  • Adversarial monitoring: 0.909 AUC remains under a probe-aware selection adversary for RECAP, while the control falls below chance to 0.242.This stronger adversary is outside the stated report-space threat model.

7 Related Work

Prior work improves activation readers, critiques verbalization benchmarks, and studies hidden-channel steganography, but this paper audits explanation faithfulness per claim and repairs it by training the target model. RECAP relates to deep supervision while targeting external content rather than task labels.

  • Activation verbalization and its critiques: Existing activation-verbalization work improves readers through supervised decoders, explainers, assistants, training-free readouts, and cross-model adapters.These approaches include Pan, Chen, and Steinhardt (2026), Karvonen et al. (2025), Bauer et al. (2026), Huang et al. (2025), Ghandeharioun et al. (2024), and Zhao et al. (2026).
  • Activation verbalization and its critiques: This paper differs from benchmark-level critiques by auditing verbalization itself per claim and counterfactually on a released system, then training the target model rather than a better reader.Prior work also includes causal tests finding high error and little efficacy and counterfactual faithfulness tests that prefigure the flip protocol.
  • Steganography in learned channels: Reconstruction objectives and optimized reasoning channels have established precedents for hiding information, while misspecified incentives can produce steganographic collusion.Related work also examines tool use as a wider channel and probing internals to detect collusion.
  • Representation drift, stitching, and monitor staleness: Representation-equivalence, affine feature transfer, stable pretraining dynamics, and tuned lenses motivate the paper’s analysis of representation drift, stitching, and monitor staleness.The passage also notes concurrent work reporting the same staleness and a label-free repair.
  • Deep supervision and intrinsic interpretability: RECAP is related to deep supervision and intrinsic-interpretability methods, but its auxiliary heads target external content to keep decodable rather than task labels.Adjacent methods include concept bottlenecks, concept whitening, and discretized codebook features.

8 Discussion and Limitations

The paper’s evidence is limited to a synthetic sandbox, one released NLA, and continued pretraining of one small model, leaving frontier-scale and open-ended faithfulness untested. It defines success as verifying specific claims, calibrated abstention, and recalibrated monitoring—not merely increasing reconstruction.

  • Scope and limitations: The evidence covers a synthetic sandbox, one released Qwen-2.5-7B NLA, and continued pretraining of one small Pythia-160M model, mostly at single seeds.Frontier scale remains untested.
  • Scope and limitations: The audits use closed-vocabulary, template-parsed claims, while the monitor covers only designated content, leaving open-ended faithfulness unmeasured.These constraints limit how broadly the conclusions can be applied.
  • Success criteria: Success means verifying explanations’ specific claims against the target model, using calibrated abstention when content is absent, and applying monitors with recalibration protocols.A higher reconstruction score alone is not the success criterion.

9 Conclusion · A Edit Validity for Minimal-Pair Flips

The audit finds that reconstruction can be satisfied by gist or private codes rather than faithful claims, while RECAP preserves claim faithfulness and probe-decodability. Valid minimal-pair edits require changing exactly one claim and nothing else, and the open judge produced valid edits 99.3% of the time versus 68.4% for the closed-model judge.

  • 9 Conclusion: Reconstruction is passed through gist and private codes, so optimizing it can produce collapse or proxy gaming instead of faithfulness.The conclusion identifies RECAP as the repair that survives the authors’ controls.
  • 9 Conclusion: RECAP trains the target model rather than the reader to produce claim-faithful verbalizers in the sandbox.This result held in 5/5 runs.
  • 9 Conclusion: RECAP makes designated content probe-decodable on Pythia-160M, where a fresh verbalizer states it truly at 0.44–0.46.The supplied conclusion reports this range for the fresh verbalizer.
  • A Edit Validity for Minimal-Pair Flips: An edit is valid only when it flips exactly one claim and changes nothing else.Grounding measurements depend on the validity of these edits.
  • A Edit Validity for Minimal-Pair Flips: 68.4% of edits from a closed-model judge were valid on the released system.The passage contrasts this with the open judge used throughout the paper.
  • A Edit Validity for Minimal-Pair Flips: 99.3% of edits from the open judge were valid on the released system.The open judge was Gemma-4-12B at a pinned revision and temperature 0.

B Safety Experiment 1: The Discrepancy Detector

The discrepancy detector is sound when probes can decode the target activation, but verification coverage is limited by target-model decodability. RECAP eliminates pre-tap lies and raises pre-tap verification coverage from 37% to 100%.

  • Detector performance: All 27 detector flags were genuinely false claims, with zero false alarms.The detector flags disagreements only when a probe exceeds 0.9 confidence.
  • Detector performance: 63% of control-model pre-tap claims fell in slots too faded for reliable adjudication.This makes target-model decodability, rather than detector quality, the principal bottleneck.
  • RECAP target: 0 false claims appeared among 1,200 RECAP pre-tap claims, while abstention fell to 0%.RECAP removes the pre-tap lie problem at the source and enables complete auditing of that content.
  • Coverage limits: 37% to 100% was the shift in pre-tap verification coverage under RECAP.Post-tap claims were 100%-abstained on both models because absent activation content cannot be adjudicated from the activation; the grounding cross addresses those confabulations.

C Safety Experiment 2: Legibility Persistence Under Head-Free Fine-Tuning · D The Synthetic Domains

RECAP’s designated-content decodability largely disappears during head-free fine-tuning, including on same-distribution data, but intermittent head supervision preserves most of it. The synthetic domains use templated, exactly auditable stories and marketplace scenarios to test these effects across controlled shifts in slot structure and tap position.

  • C Safety Experiment 2: Legibility Persistence Under Head-Free Fine-Tuning: C Safety Experiment 2: Legibility Persistence Under Head-Free Fine-Tuning — The protocol starts from RECAP Pythia-160M and control checkpoints, removes the heads, and fine-tunes on 131M shifted-corpus tokens with snapshots at 250, 1,000, and 2,000 steps.Fresh ridge probes are fit at every snapshot under the specified language-modeling-only training setup.
  • C Safety Experiment 2: Legibility Persistence Under Head-Free Fine-Tuning: C Safety Experiment 2: Legibility Persistence Under Head-Free Fine-Tuning — RECAP’s decodability surplus largely vanishes within 250 steps and approaches the control baseline by 2,000 steps.The control run and never-designated bucket remain flat, while probes are refit at every snapshot, indicating loss of linear decodability rather than stale probes.
  • C Safety Experiment 2: Legibility Persistence Under Head-Free Fine-Tuning: C Safety Experiment 2: Legibility Persistence Under Head-Free Fine-Tuning — At step 2,000, same-distribution head-free fine-tuning yields b1/b2/ident .790/.811/.151 versus .789/.812/.169 under shifted data.This follow-up shows that erosion is intrinsic to removing the heads rather than caused by corpus shift.
  • C Safety Experiment 2: Legibility Persistence Under Head-Free Fine-Tuning: C Safety Experiment 2: Legibility Persistence Under Head-Free Fine-Tuning — Applying the head loss every tenth step during shifted fine-tuning retains most decodability, with b1/b2/ident .849/.900/.891.The result concerns supervision frequency; the intermittent run’s separate language-modeling cost was not measured.
  • D The Synthetic Domains: D The Synthetic Domains — Both sandbox domains are templated grammars over closed vocabularies, giving every story an exact ground-truth tuple and audit quantities computable without a judge.Each story samples one value per slot, formats a fixed template, and adds random filler sentences.
  • D The Synthetic Domains: D The Synthetic Domains — The story domain has five slots, with name, place, and object before the tap, while activity and ending occur only afterward.The fixed tap sentence aligns the tap token offset; the name is reused after the tap, whereas place and object finish before it.
  • D The Synthetic Domains: D The Synthetic Domains — The marketplace domain shifts structure using vocabularies of sizes 20, 10, 14, 6, and 10, two template variants, and variable tap positions.Customer, shop, and item precede the tap; payment and outcome follow it, while the rest of the pipeline is unchanged.
  • D The Synthetic Domains: D The Synthetic Domains — Eight-layer, d=256 transformers trained from scratch on 120,000 stories transfer the emergence phenomena across domains, with domain-2 perplexity 1.304 matching domain 1.The tap is read at layer 4 of 8.

E Released-System Audit Details … H Reader Drift and Realignment

The audits show that reconstruction-based grounding is weak and gameable, whereas independent probes remain effective when designated content is decodable. Additional tests provide judge-free monitoring and show that reader failure reflects drift rather than information loss.

  • E Released-System Audit Details: 3.2–5.2% / 1.5–2.9% / 1.0–2.2% are the base-system bootstrap 95% intervals for grounded fractions at τ=0.02/0.05/0.10.The flip-noise floor remains below the smallest threshold, with 95th-percentile |∆˜r| of 0.017.
  • E Released-System Audit Details: 0.6% to 10.1% is the model-readout increase in missed true values entering the vocabulary top-50 from the read layer to layer 27.The generic trained probe stays flat, indicating that late-layer recovery is partial and specific to the model’s own output basis.
  • E Released-System Audit Details: 0.163 was the initially inflated negative-control score caused by adjacent rows sharing document specifics; shifting by half the dataset restored exact chance.This control-design result shows that near-duplicate controls can make audit instruments appear more favorable than they are.
  • F Adversarial Monitor: the Lie-Ladder: ∼87% of a random lie’s reconstruction penalty is suppressed by the optimal L3 lie, while the RECAP probe reaches AUC 0.952 and the control probe 0.508.The result supports monitoring through independently decodable designated content rather than through the editable report alone.
  • F Adversarial Monitor: the Lie-Ladder: 0.878 (L3) / 0.732 (L4) is RECAP recall using a frozen threshold calibrated on a disjoint split, versus 0.86 / 0.70 when eval-tuned.The operating point is not eval-inflated, but protection applies only to designated vocabulary claims and report-space attacks.
  • G An API-Free Grounding Signal, and a Selection Null: AUC 0.997 is the agreement between mask-and-rescore and the flip instrument under the validated open judge.This enables inference-time per-claim grounding with one extra frozen-reader pass and no judge in the loop.
  • G An API-Free Grounding Signal, and a Selection Null: 0.747 to 0.688 is the false-claim-rate change from random sampling to best-of-N selection with N=5 and an external truth judge.The small improvement indicates that selection remains near the same neighborhood when faithful explanations are rare.
  • H Reader Drift and Realignment: 1,500 continued-training steps drive matched readers to near-chance across 3/3 seeds, while label-free linear realignment restores native performance in 9/9 cases.Rotation alone recovers about half, whereas rank 16–64 non-orthogonal maps restore nearly all accuracy.

I Ablation Index · J Scale Verbalizer Audit Protocol · K Hyperparameters and Reproducibility Notes

The ablations identify conditions that preserve grounding and decodability, while the scale audit shows that probe-decodable content still exceeds what free-form verbalizers communicate. Reproducibility notes specify fixed training procedures, held-out evaluation, and controlled synthetic and scale experiments.

  • I Ablation Index: Independent evaluators collapse code grounding, while nonlinear heads match linear designated-probe performance at layer 6.The MLP head reaches designated-probe AUC 0.98–0.99 at layer 6 with a +0.09 tax; layer 9 reaches 0.95–0.98 from 0.77–0.79 at +0.01 nats.
  • J Scale Verbalizer Audit Protocol: Scale audits parse exact context-window claims and evaluate grounding by swapping stated words under a co-trained reconstructor.Truth is defined by exact occurrence in the trailing 64-token window, with tap-clustered bootstrap 95% intervals.
  • J Scale Verbalizer Audit Protocol: 0.574 versus 0.152: stronger retraining preserves RECAP’s truth advantage, while recall on shared present words is 0.470 versus 0.037.RECAP states 312 words versus 92 for control and leads on both precision and recall.
  • J Scale Verbalizer Audit Protocol: 0.79–0.81: a redesigned reader makes designated content linearly readable, but generated precision remains 0.56–0.63 versus 0.03 for control.The redesign uses six injected tokens, an MLP adapter, MLP-LoRA, and a decode-point presence head trained on exact window-occurrence labels.
  • K Hyperparameters and Reproducibility Notes: Scale runs use 500M continued-pretraining tokens, fixed optimizer settings, layer-6 linear heads, and a full rerun with fixed λ reproducing bucket AUCs.Held-out language-modeling loss uses 20M tokens, and method-specific hyperparameters are varied through the Ablation Index.
  • K Hyperparameters and Reproducibility Notes: 0.44 versus 0.06: the RECAP scale model states words truly far more often than control, while offset-8 truth is a base-rate artifact.The designated top-64 offset-8 target occurs in only 2.2% of taps, making its near-zero truth rate uninformative.
  • K Hyperparameters and Reproducibility Notes: Sandbox experiments use closed-vocabulary story and marketplace domains with 8-layer, d=256 transformers and a released-recipe-style toy verbalizer.Audits parse stated specifics by exact closed-vocabulary scans; scale probes use tuple-disjoint and held-out domain splits.
Loading 2607.20379v1…