Source-linked AI summary
Legible Failures: Detecting and Repairing In-Context Binding Errors
Manas Venkata Sai Ravulapalli, Samrath Singh Chadha, Abhinav M. Hari
TL;DR
Models can fail to use bindings that remain recoverable from their hidden states, creating failures that are potentially detectable and repairable. The paper evaluates this with leak-free linear probes on a controlled in-context binding task. Probe–output disagreement improves failure detection, and steering toward the probe-decoded binding improves accuracy, within the study’s bounded protocol.
Problem
External behavior cannot distinguish whether a wrong answer reflects missing information or a retained binding that the model failed to use.
Method
The paper fits linear probes to frozen hidden states using separate training, validation, and test folds, then evaluates disagreement-based detection and residual-stream steering.
Results
+0.168 mean accuracy gain across all eight intervention models, while signed probe–output disagreement improves failure detection over model confidence by +0.079 AUROC.
Takeaways & Limitations
In-context binding is a setting where probe-decoded information can be actionable: disagreement helps detect failures, and steering toward the probe’s decode raises accuracy.
Takeaways & Limitations
The evidence is bounded to a synthetic task, single-token answers, white-box hidden-state access, and frozen checkpoints, with intervention tested at one site and direction family.
Abstract
from arXiv · showhide
A wrong answer does not show whether the model lacked the needed information or held it and failed to use it. On an entity-obligation binding task, a language model can emit an incorrect prompt-supplied binding while a linear probe can recover the correct one from its frozen hidden state. We measure how often this occurs across 16 public checkpoints, each evaluated with three seeds. We fit a probe on a training fold, select its layer on a validation fold, and report results on a disjoint test fold. On the trials each model gets wrong, probe accuracy exceeds the strict present-obligation baseline, 1/K = 0.125, by +0.196 (95% CI [+0.101, +0.296], bootstrapped over models). A query-entity counterfactual rules out token presence and recency. A score built from the sign of probe-output disagreement improves failure detection over the model's own confidence by +0.079 AUROC (95% CI [+0.036, +0.126]). Raw probe confidence gives no measurable improvement over model confidence. Steering the residual stream toward the probe-decoded binding, with no gold label, raises accuracy on all eight models tested by a mean of +0.168 (95% CI [+0.066, +0.280]). Where recent studies report that probe-detected errors are resistant to interventions, we find that in-context binding is a setting in which probes are actionable.
1 Introduction
The paper distinguishes failures caused by missing information from legible failures, where the correct binding remains recoverable but does not determine the output. Across controlled evaluations, probes detect and repair these failures under bounded synthetic-task conditions.
- Legible failures: +0.196 probe-accuracy margin above the strict present-obligation baseline on wrong trials across 14 of 16 models.The probe is trained and selected on separate folds before evaluation on a disjoint test fold.
- Legible failures: A query-entity counterfactual shows that the probe reads a query-conditional signal rather than lexical presence or recency.Declarations and their order remain fixed while the queried entity changes.
- Failure detection: +0.079 AUROC improvement over model confidence from signed probe–output disagreement.Raw probe confidence adds no measurable improvement, while predictive entropy remains competitive.
- Intervention: +0.168 mean accuracy gain across all eight tested models from steering toward the probe-decoded binding without a gold label.A matched-norm random direction repairs fewer trials, while a gold-target arm bounds maximum improvement.
- Scope: The study is bounded to a synthetic task, single-token answers, white-box hidden-state access, and frozen checkpoints.The intervention is treated as a causal test of the decoded state under this protocol.
2 Experimental setup
The experiments use a controlled entity–obligation binding task and leak-free probe evaluation across frozen public checkpoints. Failure-conditioned accuracy is reported only when enough wrong trials support estimation, while detection uses all test trials.
- Task: Each trial pairs K = 8 entities with obligations, inserts D = 64 random token IDs, and queries one entity whose identity carries no obligation information.Entities and obligations are resampled independently on every trial.
- Task: The strict reference baseline is 1/K = 0.125, corresponding to naming one obligation present in the context rather than any pool-wide obligation.The model and probe face the same pool-wide decision, making this a conservative reference point.
- Probe and failure-conditioned accuracy: 600 trials are split into training, validation, and 200-test folds, with probes fitted, layers selected, and results reported without test-fold reuse.The probe is refitted on the first two thirds at the selected layer before test reporting.
- Probe and failure-conditioned accuracy: pAcc | wrong measures probe accuracy only on test trials where the model answers incorrectly, so above-baseline accuracy indicates retained but unused bindings.Models below the prespecified minimum of 30 expected wrong trials do not receive this estimate.
- Models and statistical analysis: The sweep covers 16 public checkpoints from 410M to 14B parameters, all evaluated as frozen models without fine-tuning.Checkpoint revisions were not pinned, so reruns against changed default branches need not be bit-identical.
- Models and statistical analysis: Cross-model intervals are 95% bootstrap intervals computed by resampling models after averaging their three seeds.Each model contributes once to cross-model estimates, while seeds and trials are nested observations.
3 Incorrect outputs retain the correct binding
Across evaluated checkpoints, wrong outputs often retain a query-specific linear signal for the correct binding, though this measurement is aggregate and does not identify individual legible failures.
- 3.1 Failure-conditioned probe accuracy across sixteen checkpoints: +0.196 probe accuracy above the present-set baseline of 1/K = 0.125 on wrong trials across 14 reported models.The 95% CI is [+0.101, +0.296], and 12 of 14 seed means exceed the baseline.
- 3.1 Failure-conditioned probe accuracy across sixteen checkpoints: Probe performance varies widely and rises with task accuracy, with Spearman ρ = +0.63 across the 14 reported models.The passage characterizes this as a descriptive correlation rather than a causal relationship.
- 3.1 Failure-conditioned probe accuracy across sixteen checkpoints: Two near-ceiling models fall below the failure-count floor, so their unreported values are excluded rather than treated as failures to exceed baseline.The passage notes that the positive correlation may mean these models have especially legible failures, but this remains unmeasured.
- 3.2 Counterfactual control: A query-entity counterfactual tests whether probe accuracy reflects the queried binding rather than lexical presence or recency.The declarations and their order remain fixed while the queried entity changes.
- 3.2 Counterfactual control: The measurement averages over a model’s failures and shows linear recoverability, but does not identify individual legible trials or prove the model uses the information.Intervention is required to test the stronger claim that the decoded state can affect the output.
4 Internal–external disagreement predicts error
Signed probe–output disagreement detects failures better than model confidence, whereas raw probe confidence does not; the advantage persists under several distribution shifts.
- 4.1 Signed disagreement score: Signed disagreement combines probe confidence with whether the probe and model output agree, and is evaluated on all test trials.The score is not conditioned on failure trials.
- 4.2 Baselines: +0.079 AUROC: signed disagreement improves failure detection over the model’s own confidence.The 95% CI is [+0.036, +0.126], with positive gains on 13 of 16 models.
- 4.2 Baselines: +0.003 AUROC: raw probe confidence provides no measurable improvement over model confidence.Its 95% CI is [-0.017, +0.024], which crosses zero.
- 4.2 Baselines: +0.065 AUROC: signed disagreement also beats sampled self-consistency, while its +0.025 gain over predictive entropy is uncertain.The predictive-entropy interval crosses zero.
- 4.3 Transfer under distribution shift: +0.100 mean advantage: a frozen probe remains better than model confidence across three single-axis distribution shifts.The 95% CI is [+0.065, +0.136], and all six models are positive under each shift.
5 The model’s own decoded state repairs behaviour
Steering the residual stream toward the probe-decoded binding repairs model outputs without a gold label, but its effectiveness depends on the intervention site and target quality.
- 5.1 Self-gated activation repair: +0.168 mean accuracy: self-gated steering raises accuracy on all eight tested models without using a gold label.The 95% interval is [+0.066, +0.280].
- 5.1 Self-gated activation repair: The intervention changes accuracy only on trials flagged by the detector, while preserving 0.64–0.97 of already-correct trials.When the probe misreads a correct binding, steering can move the output away from the correct answer.
- 5.2 Control arms: Direction drives the repair effect: matched-norm random directions repair fewer failures than decoded directions.Random directions repair 0.02–0.18 of failures, whereas the gold-target arm repairs 0.18–1.00.
- 5.3 Boundary conditions: The fixed-site intervention transfers poorly to smaller non-Pythia models, where gold-target repair reaches only 0.130–0.378.The authors distinguish this boundary from the separately tuned eight-model repair result.
- 5.3 Boundary conditions: Failure-conditioned probe accuracy returns near the present-set baseline on mutable and scoped binding variants.At Pythia-1.4B, accuracies are 0.307 and 0.284 at matched difficulty.
6 Related work
The paper builds on work showing that hidden states can contain answers not emitted by models, probe-based error detection, and inference-time intervention, while targeting prompt-supplied in-context bindings.
- Related work: Prior work establishes that models’ internal states can carry information about answers they do not produce.The cited studies concern parametric knowledge and reasoning traces.
- Related work: This paper differs by studying in-context bindings whose ground truth is supplied by the prompt and resampled each trial.That design removes parametric recall as an explanation of the decoded binding.
- Related work: Probe-based error detection commonly reads internal properties, whereas this work uses signed disagreement between probe and model outputs.Raw probe confidence supplies no advantage over model confidence on the reported sweep.
- Related work: The intervention follows residual-stream steering work but adds a trial-specific decoded target, validation-selected read layer, and matched controls.Earlier reports found related interventions failed on hallucination, knowledge–action, and clinical tasks.
7 Discussion and conclusion
The paper shows that hidden-state probes can detect and repair some in-context binding failures, while emphasizing boundaries imposed by the synthetic task, white-box access, and probe accuracy.
- The evidence is bounded to a synthetic single-token task, frozen checkpoints, and white-box hidden-state access.The authors identify validation on naturalistic multi-token agent trajectories as a next test.
- Probe-based methods cannot detect or repair failures when the probe cannot read the binding, and their deployment requires inference-time hidden-state access.The intervention and detector are both bounded by probe accuracy.
- The reported legibility estimate covers 14 of 16 models because near-ceiling models lack reported conditional accuracy and have rare failures.Extending the analysis to multi-token settings would require free-form comparison against semantic entropy.
- The intervention establishes a causal effect on this task, but the mechanism by which displacement changes the output remains incomplete.A Jacobian measurement rules out the explanation that the readout cannot see the binding without identifying an alternative account.
- A subset of failed binding trials retains a query-specific signal for the correct binding, enabling probe-based detection and intervention.The signed disagreement detector improves on model confidence, and residual-stream steering toward the probe decode raises accuracy.
B Per-model legibility, counterfactual and detection
Table 3 organizes per-model legibility, query-entity counterfactuals, and detector comparisons, while the surrounding protocol defines how these quantities are reported.
- Table 3 reports one row per model, with conditional probe accuracy and query-entity counterfactual results alongside detector comparisons.The table averages the first columns over seeds and reports ranges in parentheses.
- The query-entity counterfactual is evaluated against the 1/K baseline that present-token or recency heuristics would achieve.The counterfactual changes the queried entity while keeping declarations and order fixed.
- Models below 30 expected wrong trials have pAcc|wrong marked not reported and are excluded from the conditional-accuracy summaries.Detection AUROC is still reported for every model over all 200 test trials.
- Table 4 extends detector evaluation to frozen probes under single and stacked distribution shifts.The shifts include unseen vocabularies, distractor prose, and interference load.
D Activation repair by model, control arm and seed
The activation-repair results compare decoded steering with matched controls across models and seeds, using detector-gated and strength-sweep analyses.
- Activation repair by model, control arm and seed: Table 3 summarizes per-model legibility, counterfactual performance, and signed-disagreement detector comparisons over three seeds.The detector columns compare signed disagreement with reference detectors, while raw probe compares probe confidence with model self-confidence.
- Activation repair by model, control arm and seed: Table 4 reports changes in AUROC over model self-confidence when an in-domain probe is frozen under single and stacked shifts.This visual concerns transfer of detection rather than activation repair.
- Activation repair by model, control arm and seed: The intervention-strength analysis reports accuracy change and preservation of correct trials above α = 0.5 for the self-gated decoded arm.Values at α = 0.5 are taken from Table 5.
- Activation repair by model, control arm and seed: Per-seed activation-repair gains remain positive across the reported Pythia and OLMo models, with smaller but positive gains for Qwen2.5-7B.The three-seed changes are reported separately for each model.
E Prompt repair and its provenance
Prompt-level repair re-presents the hidden-state-decoded binding before the query without a gold label, and matched controls distinguish content effects from perturbation format.
- Prompt repair and its provenance: Mean prompt-level recovery is +0.205 (95% CI [+0.076, +0.330]) across six models and three seeds, reaching +0.407 on DeepSeek-Coder-6.7B.A format-matched random injection lowers accuracy by -0.096 on average across 18 runs.
- Prompt repair and its provenance: The random-injection control differs from re-presented binding only in injected content, supporting the content-specific interpretation of the effect.The gated arm intervenes only on trials flagged by the detector.
- Prompt repair and its provenance: The three-seed battery is reproducible at run granularity but not trial granularity because per-trial records were not retained.The activation-repair result, rather than this arm, carries the causal claim.
- Prompt repair and its provenance: Table 5 provides the activation-repair arm context, including the validation-selected read layer used for intervention.Its scope is α = 0.5, K = 6, D = 256, and 1200 trials per model.
- Prompt repair and its provenance: Table 6 reports how intervention strength above α = 0.5 changes accuracy and the fraction of correct trials preserved.The α = 0.5 values come from Table 5.
F Hyperparameters and compute
The study uses standardized logistic-regression probes and controlled fold splits for frozen-checkpoint evaluations, with separate configurations for legibility and activation repair.
- K = 8 and D = 64 are used for legibility and detection, across 600 trials split into three folds of 200.
- K = 6 and D = 256 are used for activation repair, across 1200 trials with the same three-way split.
- All measurements are forward passes on frozen public checkpoints, with no model training.