PaperlayerView exact revision on arXiv

Source-linked AI summary

Human Interventions as Labels: Localizing, Steering, and (Sometimes) Editing the Computations Behind LLM Agent Failures

Cesare Zavattari, Alessandro Tommasi, Giuseppe Prencipe

arXiv:2610.21708v1cs.LGcs.AIcs.CL

TL;DR

When an agent asks a clarifying question despite an unambiguous request, the resulting intervention is usually discarded or replayed as training data. This paper instead uses intervention labels to locate and test the computations behind that behavior. The authors report causal localization across eligible families, distinct mechanisms for over-asking, and model-dependent outcomes when consolidating fixes into weights.

  • Problem

    The paper asks whether oversight interventions can reveal which internal computations failed, rather than serving only as correction examples for training.

    Source: §1 · p. 2

  • Method

    The authors use rule-based intervention labels in a controlled matched-pair agent environment, then test causal localization, steering, and weight edits against behavior cloning.

    Source: §1 · p. 2, §3 · p. 4, §4.5 · p. 7

  • Results

    A probe direction flipped 72–100% of held-out over-asks across eligible families, while weight-edit outcomes varied by model and two advance-written predictors failed.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    The paper treats oversight signal as data about model internals and evaluates edits by whether overseers had to intervene less, not by probe-set performance alone.

    Source: §7 · p. 16

    The small, controlled testbed and five-family eligible sample do not establish transfer to naturalistic agent traces, and the weight-edit regime remains unexplained.

    Source: §6 · p. 15

Abstract

from arXiv · show

When an LLM agent misbehaves, a human steps in, and that intervention is usually discarded or replayed as training data. We treat it instead as a measurement: a label pointing at the internal computation that failed. We study it in a controlled agent environment, on one failure (asking a clarifying question when the request was already unambiguous), with labels from a rule-based overseer validated against blind human judgment. Claims cover eligible models, which both over-ask and sometimes act on matched unambiguous requests: five of thirteen qualified (Qwen2.5-3B, gemma-2-2b, OLMo-3-7B, Mistral-Nemo-12B, Phi-4). Four findings. First, the ask-vs-act decision is causally localizable in every eligible family: a direction learned from intervention labels flips 72-100% of held-out over-asks at a definite residual-stream depth, with site and dose chosen on training data, and random directions under the same procedure flip at most one episode. Second, over-asking has two mechanistic signatures: in four families the model demonstrably knows the request is unambiguous (the belief is decodable, causally inert and orthogonal to the decision) and asks anyway; in the fifth, the rare over-asks track different world-representations. Third, steering gated on a probe readout removes most of the simulated oversight burden while legitimate questions survive (bootstrap 95% intervals exclude zero); a prompt instruction makes it worse. Fourth, consolidating the fix into the weights yields a map, not a rule: on one model a LoRA confined to the causal band passes every criterion fixed in advance while behavior cloning on identical data breaks retention; on three families the reverse holds; on the fifth the outcome crosses the boundary across seeds. A geometric rule written down in advance to predict the regime failed on two out-of-family tests; we report the failure and stop rather than fit a replacement.

1 Introduction

The paper treats interventions as measurements of failed internal computations rather than merely training demonstrations. In a controlled setting, it localizes and steers the ask-vs-act decision, finds distinct over-asking mechanisms, and maps when weight edits succeed or fail.

1 Introduction

  • The study treats an intervention as a measurement of a failed computation, rather than replaying the correction only as a demonstration.Its intervention-reduction metric combines watching and intervention costs, with full-attention failure counts also reported where applicable.Source: §1 · p. 2
  • The labels come from a rule-based overseer validated against blind human judgment; naturally occurring human intervention logs remain future work.The labeling overseer is a controlled, reproducible proxy, not a claim that the pipeline has been deployed on natural logs.Source: §1 · p. 2
  • A direction learned from intervention labels flipped 72–100% of held-out over-asks across eligible families, while random directions flipped at most 1/74.The causal site varied from 55–81% of network depth, but its existence and sharp onset did not.Source: §1 · p. 2
  • Four families knew requests were unambiguous yet asked anyway; Phi-4’s over-asks instead tracked different belief states.In the four, the belief representation was orthogonal to the decision axis and causally inert; the authors caution that one family does not establish a taxonomy.Source: §1 · p. 2
  • Probe-gated steering reduced most simulated oversight burden while preserving legitimate questions, whereas a system-prompt instruction worsened burden on every seed.The best steering direction was model-dependent, while specificity came from the gate.Source: §1 · p. 3
  • A LoRA confined to Qwen’s causal band passed every predeclared criterion, while behavior cloning on identical data broke retention; outcomes varied across the other four families.The localized edit failed on three families, and Phi-4 crossed the criteria across full-pipeline seeds; two advance-written predictors also failed.Source: §1 · p. 3

2 Related work

The paper positions its contribution at the intersection of agent oversight, interpretability, and editing: it uses oversight interventions as labels for internal causal tests and evaluates edits by subsequent intervention burden. Its localized-edit comparison with behavior cloning asks whether targeting a causal band changes generalization, with outcomes varying by model.

Activation steering and representation engineering

  • The paper combines activation-steering machinery with oversight-derived labels and evaluates reduced future interventions rather than probe accuracy.It distinguishes its signal source from curated contrastive prompts and its evaluation metric from probe accuracy.Source: §2 · p. 3

Knowledge editing and its ripple problem

  • The paper’s comparison asks whether restricting an edit to a causally implicated band changes optimizer generalization, and reports that outcomes vary by model.One localized edit was decisively better, more often the imitation default was better, and the tested geometric measurements did not predict which.Source: §2 · pp. 3–4

Learning from human interventions

  • Unlike prior intervention-learning work that uses corrections as policy-level training data, this study connects oversight corrections to internal computations and compares localized editing with behavior cloning on identical corrections.The matched data and fine-tuning recipe are intended to isolate the effect of localization.Source: §2 · p. 4

Agent failures and the knowing–doing gap

  • The paper adds a causal account to related knowing–doing results: in four families, correct ambiguity beliefs are decodable but causally inert and orthogonal to the ask decision.The authors also note that the dissociation varies across families; Phi-4’s over-asks track different belief states.Source: §2 · p. 4

Actionability, auditability, and predictions made in advance

  • The authors frame their approach as an LLM-agent-scale example of prediction-first, matched-control, out-of-distribution validation, while reporting that two advance-written geometric predictors failed.They stopped rather than fit a third rule.Source: §2 · p. 4

3 Setting: an agent, an overseer, and a failure worth editing

The study uses a deliberately small, matched-pair workspace to test whether agents ask for clarification when a request is already unambiguous. A reproducible rule-based overseer labels these episodes, while a separate budgeted overseer supplies an intervention-burden evaluation.

The environment — a causal testbed, not a simulation of deployment

  • Each episode uses a one-line task and three-item workspace; twin scenarios differ only in workspace content, making the request uniquely resolvable or genuinely ambiguous.The harness rejects any twin pair whose rendered prompts differ beyond the item lines.Source: §3 · p. 4

The failure, and who is eligible to have it

  • The eligible models over-asked at markedly different rates: Qwen2.5-3B-Instruct and gemma-2-2b-it at 84% and 90%, while Phi-4 was at 36%.The later-added OLMo-3-7B and Mistral-Nemo-12B over-asked at 88–90%; Phi-4’s evaluation-base-row rates ranged 28–37%.Source: §3 · pp. 4–5
  • Five of thirteen candidate model families qualified because the study requires both over-asks and direct actions on unambiguous requests.Several screened families were in degenerate always-ask or never-ask corners, or could not maintain the reply format.Source: §3 · pp. 4–5

The overseer, the metric, and what each is for

  • A deterministic rule-based labeling overseer applies task ground truth to issue intervention verdicts and canonical corrections; blind human studies validate its verdicts.It is separate from the evaluation overseer, which watches only a budgeted fraction of episodes and has trust dynamics.Source: §3 · p. 5
  • The evaluation score is the relative reduction in budgeted overseer burden, combining watching and intervention costs; full-attention failure counts accompany IRR where reported.The budgeted-burden dynamics are an automated operational proxy, not directly human-validated.Source: §3 · p. 5

4 Method: from interventions to a causal handle

The method moves from intervention-labeled episodes through probes and causal tests to runtime steering and weight edits, trusting each stage only after a subsequent test. Predictions and stopping rules set in advance wrap the pipeline.

4 Method: from interventions to a causal handle

  • The method’s stated organizing principle is that each pipeline stage is trusted only after the next stage tests its output.Predictions written before outcomes wrap the four stages.Source: §4 · p. 5

4.1 Collect, with the prompt as part of the experiment

  • Fresh scenario-pair rollouts preserve messages, replies, parsed decisions, and overseer verdicts, while blocking replay, prompt-diff, and overseer-canary checks gate data.Audits caught five harness bugs, including an OLMo-3 turn-boundary parsing error that favored edits; repaired-harness base-row rates matched collection-time rates within 8 points across 13 evaluations.Source: §4.1 · pp. 5–6

4.2 Probe, then distrust the probe

  • Two logistic probes predict upcoming behavior and ground-truth ambiguity at the last prompt token, but enriched content baselines make probe decodability alone insufficient evidence.Pair-based splits keep scenario twins out of both train and test; probes also use permuted-label, positional, and content-baseline controls.Source: §4.2 · p. 6

4.3 One coordinate, both directions, three controls

  • Causal patches alter only the behavior-direction projection at the decision token, using training-only class-mean or extrapolated doses, with style, random, and bidirectional belief-axis controls.The extrapolated dose overshoots the acting class by one inter-class gap; the class-mean recipe is used for the controls.Source: §4.3 · p. 6

4.4 Edit, always under the same evaluation harness

  • Runtime steering uses agreement between the belief and behavior probes to gate a one-shot decision-token projection, while LoRA edits train on overseer corrections plus clean-episode retention examples.Local LoRA parameters are restricted to the causal band; behavior cloning trains across all layers, and all arms use the same evaluation harness and fixed thresholds.Source: §4.4 · pp. 6–7

4.5 Predictions before outcomes

  • Predictions, conventions, and stopping rules were recorded before outcomes, but these commitments were internal project notes rather than externally registered or attested preregistration.The authors report the prior predictions alongside outcomes and include confirmations, falsifications, and full screening.Source: §4.5 · p. 7

5 Results

Across five eligible model families, the results examine how human interventions reveal causal ask-versus-act decisions, distinct internal signatures of over-asking, and the effects of steering and weight editing. Human validation supports the overseer’s judgments and intervention reductions, while a re-asking failure that disappears when prior questions are exposed illustrates that similar surface behavior can have context-level or weight-level causes.

5.1 The signature exists and is causal — in every eligible family

  • A single probe-derived coordinate causally flips most held-out over-asks at a model-specific depth, while matched random directions flip at most one episode.The frozen-selection protocol chooses site and dose on training over-asks; full generation confirms the flips are not merely logit artifacts. The causal onset differs across families, so one model’s coordinates do not transfer reliably to another.Source: §5.1 · p. 7, Table 1 · p. 8, Figure 1 · p. 8

5.2 Two signatures of over-asking: knowing-without-doing, and genuine misreading

  • Four families exhibit H-caution: they represent requests as unambiguous, yet intervening on that belief barely changes decisions; Phi-4 instead exhibits H-misrep.In the four H-caution families, over-asked episodes carry essentially the same unambiguous belief as acted episodes (0.65–0.84); the belief and decision directions are near-orthogonal. Phi-4’s over-asks track greater internal uncertainty, and the five-family sample supports an observation, not a taxonomy.Source: §5.2 · pp. 8–9, Figure 2 · p. 9

5.3 Runtime steering: the gate carries the specificity

  • On Qwen, gated steering reduced full-attention failures by 74–83% across seeds, while prompt-only instructions increased overseer interventions on every seed.Simulated oversight burden fell by 73% / 91% / 87% at the default overseer seed, with pair-cluster bootstrap 95% intervals excluding zero. Legitimate clarifying questions barely moved, but gemma showed that the best lever direction is model-dependent and selectivity resides in the gate.Source: §5.3 · pp. 9–10, Figure 3 · p. 10
  • On gemma, sparse feature clamps found no decision flips from style-orthogonal behavior-specific features; only the AUC-matching readers separated from zero.The readers flipped 7/78 episodes (p = 0.014 against 0/78) while moving style most; the analysis is limited to one model, one layer, and class-mean dose, and does not establish that shared components caused the effect.Source: §5.3 · pp. 10–11

5.4 Into the weights: the five-family map

  • The five-family editing map shows that behavior cloning usually passes, but is not universally safe, while localized editing beats it outright on Qwen across all three seeds.Behavior cloning passed 9 of 10 seed-arms outside Qwen but failed retention on all Qwen seeds and one Nemo seed. On Qwen, localized editing reduced burden by 84/85/86% with the capability battery unchanged; Phi-4’s local outcome crossed the criteria boundary across full-pipeline seeds.Source: §5.4 · pp. 11–12, Figure 4 · p. 12
  • Localized editing and behavior cloning show a double dissociation: localization passes on Qwen but fails retention on gemma, where cloning is clean.On Qwen, cloning fails retention on every seed and introduces re-asking; on gemma, localization acts on all 100/100 unambiguous and 100/100 ambiguous episodes, while cloning improves legitimate clarifying by +11.Source: §5.4 · pp. 12–13
  • Two prewritten geometric rules failed out-of-family tests, and Phi-4’s local result crossed the boundary across seeds, so the authors stopped rather than fit a replacement.The band-width sweep found two ordered thresholds on gemma: enough capacity to fit corrections, then enough to preserve conditionality; Qwen passed at 8 of 36 layers. OLMo-3 contradicted the original rule, and both rules were wrong on Phi-4.Source: §5.4 · pp. 13–14, Figure 6 · p. 14

5.5 Do humans agree with the judges?

  • Blind human annotation agreed with the rule overseer at κ = 0.900 and judged steering’s intervention reduction at −81%, versus the overseer’s −87% on the same sample.The independent annotator matched intervention type in 28 of 30 shared interventions; three blind verdicts were confirmed input errors and corrected with an audit trail.Source: §5.5 · p. 14
  • The first author agreed with the rule overseer at κ = 0.933 in the κ study and reproduced all 60 blind-episode verdicts at κ = 1.0.The κ-study episode IDs exposed ground-truth suffixes, though the annotator reported not noticing them; later studies used opaque IDs. Human studies validated verdicts and intervention reductions, but not budgeted burden dynamics, which proved unstable on 30-episode subsamples. Residual disagreements concerned spurious-option questions and failure-type labels on over-asks.Source: §5.5 · pp. 14–15

5.6 When it isn’t in the weights

  • In 45–67% of episodes across three models, agents re-asked after receiving an answer, but the failure fell to exactly zero on both tested models after a scaffold change.The ablation showed that the agent could not see its own step-1 question; the result cautions against weight editing when a scaffold change fixes the behavior.Source: §5.6 · p. 15
  • A behavior-cloning edit caused re-asking on Qwen and amplified it on one Nemo seed, despite training and evaluation using the repaired scaffold.The same surface behavior can have a context-level cause, as the ablation showed, or a weight-level cause, as verified for this edit.Source: §5.6 · p. 15

6 Limitations

The study is limited to a small, rule-defined agent testbed and five eligible model families, with several mechanistic and evaluation boundaries. It does not establish transfer to naturalistic traces or explain which model property determines the editing regime.

6 Limitations

  • The testbed uses one-line tasks, three-item workspaces, and a rule-defined overseer, so transfer to naturalistic agent traces remains unshown.The LLM-judge overseer and failure taxonomy are left for future work.Source: §6 · p. 15
  • The editing regime is mapped across five families, not explained, and its boundary crosses one model’s full-pipeline seed variance.The authors decline to fit another rule for predicting the regime; identifying its determining property remains open.Source: §6 · p. 15
  • The H-misrep signature rests on one family, while the SAE analysis uses one dictionary, one layer, and class-mean dose.The style-direction control was model-dependent, and interference across simultaneous edits is untested.Source: §6 · p. 15
  • Human validation uses two annotators and validates verdicts and intervention reductions, but not burden dynamics.The burden weights an over-ask and an action on a genuinely ambiguous request equally, and a belief-probe attack could make gated steering force the latter error.Source: §6 · p. 15
  • Retention can miss degraded clarification quality because any clarifying question on an ambiguous request counts as correct.Adversarial robustness is not established, and concurrent work finds located-and-edited mechanisms collapse under adaptive attack.Source: §6 · p. 15

7 Discussion and what comes next

The discussion frames oversight interventions as evidence about internal computations and evaluates fixes by whether overseers intervene less. The paper leaves prediction of editing regimes open, while outlining testable next steps and documenting its reproducibility process.

7 Discussion and what comes next

  • The authors’ central framing treats intervention logs as localization data, corrections as causal experiments, and edit success as reduced overseer intervention.The evaluation target is the overseers around the system, rather than a probe set.Source: §7 · p. 16
  • No tested cheap measurement predicts before training whether localized editing will beat imitation on a model.The five-family map makes the question well-posed within this design, but the authors stop rather than fit a replacement rule after two predictors were falsified.Source: §7 · p. 16
  • The proposed next steps are to study the regime boundary, test richer failure types, and determine which measurements select an editing rung early.The ladder runs from scaffold ablation through prompt intervention and gated steering to weight edits; the probe readout is one candidate measurement.Source: §7 · p. 16

Reproducibility statement

  • An independent adversarial audit prompted changes including frozen selection, IRR bootstrap intervals, an SAE reader arm, and optimizer-seed retrains.The audit’s follow-up independently reproduced interval analyses from released episode logs; code, configurations, logs, and reports are available, with trained adapters deposited on Zenodo.Source: Reproducibility statement · p. 16

A The eleven-candidate screening

Behavioral-contrast eligibility was scarce in screening eleven later candidate families, leaving three eligible replication models. The exclusions clustered around always-asking, never-asking, and format non-compliance profiles.

A The eleven-candidate screening

  • Three of eleven screened families became eligible for the later replication slots: OLMo-3-7B, Mistral-Nemo-12B, and Phi-4.Their over-ask rates were 90%, 84%, and 37%, respectively; Phi-4 also made 63 direct actions.Source: A The eleven-candidate screening · p. 19
  • Three always-ask models produced no direct actions to contrast, while two never-ask models failed to ask on ambiguous requests.OLMo-2-1B also repeated its question after receiving an answer in all 100/100 cases.Source: A The eleven-candidate screening · p. 19
  • Three further candidates were excluded for format non-compliance, and the authors report all exclusions because eligibility defines the method’s applicability window.Mistral-7B-v0.1, Yi-6B, and Qwen3-1.7B had invalid-reply rates of 20%, 87%, and 200/200, respectively.Source: A The eleven-candidate screening · p. 19

B Evaluation overseer parameters and instruction variants

The evaluation overseer samples which episodes to watch using trust and attention-budget parameters, then computes burden from watching and intervention costs. A style direction contrasts last-token activations under opposing instruction suffixes.

B Evaluation overseer parameters and instruction variants

  • The budgeted overseer starts each family at trust 0.5, uses attention budget 0.5 and intervention cost 2, and watches with probability min(1, 2b(1 − t)).Trust drops to 0 after an intervention and rises by 0.1 after a watched clean episode, up to 0.95.Source: B Evaluation overseer parameters and… · p. 19
  • Burden is (W + cI)/N over the stream, with failures on unwatched episodes recorded as missed and assigned no cost.Stochastic watching makes burden sensitive to the overseer’s seed, so raw and total failure counts are also reported.Source: B Evaluation overseer parameters and… · p. 19
  • The style direction is the difference between mean last-token activations under suffixes requiring always-ask versus never-ask behavior.Both suffixes are appended to the same unambiguous training scenarios.Source: B Evaluation overseer parameters and… · p. 19

C The five harness bugs

The harness audit found bugs that could have manufactured conclusions, alongside two performance problems that did not put conclusions at stake.

C The five harness bugs

  • Bit-identical OLMo-3 activations from an unfilled attention weight were caught by a context-sensitivity canary, which now blocks every probe run.The fix uses the HF-wrapping load path.Source: C The five harness bugs · p. 19
  • The episode runner continued past the end-of-turn token, deflating the base-model row; it now stops on tokenizer special tokens, with two tests.The base-row audit exposed the bug.Source: C The five harness bugs · p. 19
  • A basis mismatch in sparse-dictionary capture escaped the gate and was found by audit; its downstream analytical effect was never measured.The reconstruction error changed from 8.65 to 8.58 after the raw-basis fix, but the twin design broke the variance-normalized metric.Source: C The five harness bugs · pp. 19–20
  • A wrapped model left on the CPU caused a device assertion at load time; an explicit move fixed this performance bug.The bug did not put a conclusion at stake.Source: C The five harness bugs · p. 20
  • Reference cycles kept the base model in GPU memory during edited evaluation, causing an out-of-memory failure on the largest model; explicit collection fixed it.This was a performance bug, not a threat to a conclusion.Source: C The five harness bugs · p. 20

D SAE instrument details (gemma, layer 21)

The Gemma Scope instrument was gated for transfer to gemma-2-2b-instruct using a relative-error criterion after a declared metric amendment. Its feature results distinguish the computations that write the reply opener from detectors of user choice or uncertainty.

D SAE instrument details (gemma, layer 21)

  • The instrument was a 16k-feature Gemma Scope JumpReLU SAE on the residual stream at layer 21, pretrained on gemma-2-2b and gated before results on the instruct variant.The amended relative-error gate was 0.18 on the authors’ activations, with sparsity in range.Source: D SAE instrument details (gemma, layer… · p. 20
  • The original variance-normalized reconstruction gate was amended before downstream analyses because twin episodes make between-episode variance artificially small.A five-prompt diagnostic indicated higher error at the final formatting token than on content tokens, but was not a measurement.Source: D SAE instrument details (gemma, layer… · p. 20
  • Fifty dictionary features recovered 0.80 of the dense belief probe’s 0.99 AUC, while the behavior label needed ten features.The world representation was not sparse in this dictionary.Source: D SAE instrument details (gemma, layer… · p. 20
  • Features that flip the behavior carry ACTION and write reply openers, while inert behavior-specific features include detectors of choice and uncertainty.The paper’s auto-interp labels are offered as color rather than evidence; the representation of a choice is not what emits the question.Source: D SAE instrument details (gemma, layer… · p. 20
Loading 2610.21708v1…