PaperlayerView exact revision on arXiv

Source-linked AI summary

Heads of Security: Learned Attention Gates Improve Prompt Injection Robustness

Igor Shilov, Xiaoxue Yang, Alexander Panfilov, Evgenii Kortukov, Matthieu Meeus, Peter Romov, Wojciech Samek, Jonas Geiping, Yves-Alexandre de Montjoye

arXiv:2610.37657v1cs.LGcs.CLcs.CR

TL;DR

Prompt injection remains difficult to defend against, particularly for open-weight models facing adaptive attacks. HoS uses learned per-head attention gates to separate trusted instructions from untrusted content. In the tested open-weight models, it improves adaptive robustness and utility while matching or improving SecAlign’s security–utility trade-off.

  • Problem

    Prompt injection remains largely unsolved, with adaptive attacks succeeding against evaluated systems and leading open-weight models less robust than frontier models under comparable red-teaming.

    Source: §1 · p. 1

  • Method

    HoS enforces instruction–data separation with learned per-head attention gates that redistribute attention between trusted and untrusted tokens.

    Source: §1 · p. 2

  • Results

    97% ASR on GPT-OSS-20B and 85% on SecAlign versus 10% on HoS; across tested open-weight models, HoS is more robust to adaptive attacks and better preserves utility outside training, while matching or improving SecAlign’s security–utility trade-off.

    Source: §1 · p. 3, §6 · p. 10

  • Takeaways & Limitations

    For the open-weight models tested, instruction–data separation can be enforced by constraining attention rather than broadly fine-tuning model behavior.

    Source: §6 · p. 10

    Instruction–data separation does not determine trusted sources or authorized instructions, and applies only when external content is intended as data rather than instructions to follow.

    Source: §6 · p. 10

Abstract

from arXiv · show

We introduce Heads of Security (HoS), a prompt injection defense that enforces instruction-data separation via a small set of learned per-head attention gates. HoS learns how each head should redistribute attention between trusted and untrusted tokens, enabling models to reason over untrusted content without following injected instructions. We evaluate HoS on Llama-3.3-70B-Instruct and GPT-OSS-20B. We compare it against SecAlign++, a state-of-the-art finetuning-based defense, and ASIDE, a defense that, like ours, builds instruction-data separation into the model architecture. We show that HoS offers greater robustness to adaptive attacks, better performance on standard utility benchmarks, and matches top performance on standard agentic prompt injection benchmarks. More specifically, the black-box RL-Hammer attack reaches 97% ASR against GPT-OSS-20B and 85% against SecAlign, but only 10% against HoS. Claudini, a state-of-the-art white-box attack, strengthened against HoS through autoresearch, requires 3 times more FLOPs to reach the same ASR on HoS as it does on GPT-OSS-20B or SecAlign. HoS preserves base-model utility across standard multiple-choice benchmarks and prompting setups, whereas SecAlign and ASIDE degrade in few-shot settings. On agentic benchmarks with static attacks, HoS and SecAlign perform similarly and substantially improve the security-utility trade-off over the base model. ASIDE struggles with tool use, achieving only 10% utility. These results show that prompt injection robustness can be achieved and improved through targeted attention interventions.

1 INTRODUCTION

HoS addresses persistent prompt-injection vulnerability by enforcing instruction–data separation with learned per-head attention gates. Across the reported evaluations, it improves adaptive-attack robustness and standard utility while performing strongly on agentic benchmarks.

1 INTRODUCTION

  • Prompt injection remains largely unsolved: adaptive attacks succeed against extensively evaluated systems, and comparable red-teaming finds frontier models more robust than leading open-weight models.The attack surface becomes increasingly consequential as language models access more external data and tools.Source: §1 · p. 1
  • HoS enforces instruction–data separation through learned per-head attention gates that redistribute attention between trusted and untrusted tokens.This lets models reason over untrusted content without following injected instructions.Source: §1 · p. 2
  • HoS treats the trust boundary as externally supplied and learns head-specific attention redistribution, rather than requiring the model to infer source trust or acquire new capabilities.The rationale is that heads specialize in different functions, so their attention can be adjusted according to their roles.Source: §1 · p. 2
  • RL-Hammer reaches 97% ASR on GPT-OSS-20B and 85% on SecAlign, but only 10% on HoS; Claudini requires roughly 3× more compute to reach 50% ASR on HoS.On Llama-3.3-70B, the strongest RL-Hammer attacker reaches 99% ASR against the undefended model and 82% against Meta-SecAlign-70B, versus 1% against HoS.Source: §1 · p. 3
  • HoS closely matches base-model utility across standard benchmarks, while SecAlign and ASIDE often struggle in few-shot and multi-turn settings.On 5-shot MMLU-Pro, HoS scores 67% versus GPT-OSS-20B’s 68%; on Llama-3.3-70B, it scores 55% versus the base model’s 56%.Source: §1 · p. 3
  • On static agentic benchmarks, HoS and SecAlign perform comparably, while ASIDE produces near-zero utility on three benchmarks and 10% on AgentDojo.Across AgentDojo, AgentDyn, LLMail-Inject, and WASP, HoS and SecAlign substantially reduce ASR with little or no utility penalty in most evaluations.Source: §1 · p. 3

2 RELATED WORK

Prior defenses encode input roles through fine-tuning or architectural changes, while related work also steers or studies attention. HoS builds on attention findings by learning per-head redistribution to ignore injected instructions.

Prompt injection defenses

  • Meta-SecAlign uses DPO to teach models to follow user instructions while ignoring data injections; architectural alternatives add role embeddings or rotate data-token embeddings.HoS is also architectural, but reweights attention according to token trust levels.Source: §2 · p. 3

Attention and prompt injection

  • PASTA and InstABoost steer attention toward selected text or instruction tokens, while Causal Head Gating suppresses attention-head outputs to study instruction following.Prior studies report that specific heads shift attention from benign to injected instructions; HoS lets each head learn its own redistribution to ignore injections.Source: §2 · p. 3

(Adaptive) attacks

  • Existing defenses remain vulnerable to strong adaptive attackers; RL-Hammer reports up to 92% ASR against GPT-5-series models, 77% against Claude Sonnet 4, and near-100% against Llama-3.3-70B-Instruct defended with SecAlign.These results motivate evaluations using black-box RL and white-box optimization-based attacks.Source: §2 · p. 3

Prompt injection defenses

  • The paper uses “SecAlign” as shorthand for the SecAlign++ training recipe.Source: §2 · p. 3

3 METHOD

HoS assigns tokens to trust channels and uses learned per-head gates to redistribute attention across them. Training changes only the gates, using preference pairs that contrast following the trusted instruction with following an injected instruction.

3.1 PRELIMINARIES

  • Tokens are assigned TRUSTED, UNTRUSTED, or TEMPLATE channels, with tool outputs typically untrusted and chat-template tokens assigned to TEMPLATE.The channel assignment can also represent more complex trust boundaries, such as untrusted spans in user prompts or trusted tools.Source: §3.1 · p. 4

3.2 CHANNEL-CONDITIONED ATTENTION GATES

  • HoS adds learned per-head, per-channel biases to attention logits, allowing a head to reduce attention to untrusted content while other heads can still process it.The sigmoid gates scale attention to channels before softmax normalization; a single shared gate across all keys cancels out.Source: §3.2 · p. 4

3.3 TRAINING

  • Training uses preference pairs in which the preferred response follows the user instruction and the rejected response follows the injected instruction.HoS trains only its gate parameters; GPT-OSS requires an additional key-only channel for its explicit attention sink.Source: §3.2 · p. 4, §3.3 · p. 5

4 RESULTS

HoS resists adaptive attacks while largely preserving standard utility and achieving strong security–utility trade-offs on agentic benchmarks. Its results include lower attack success rates than the baselines, stable utility across few-shot and multi-turn settings, and performance comparable to SecAlign on agentic tasks.

4.1 ADAPTIVE ATTACKS

  • 97% ASR against GPT-OSS-20B and 85% against SecAlign contrast with 10% against HoS under the strongest RL-Hammer attack.Against the HoS-targeted white-box attack, HoS requires roughly three times as much compute as the base model or SecAlign to reach 50% ASR.Source: Table 1 · p. 5, §4.1 · pp. 5–6, Figure 2 · p. 6

4.2 BENIGN UTILITY

  • 67.2% on 5-shot MMLU-Pro for GPT-OSS HoS nearly matches the base model’s 68.3%, while SecAlign scores 59.4% and ASIDE 52.1%.In three-turn AlpacaFarm conversations, HoS closely matches the base model, while SecAlign’s utility drops on the input turn and remains degraded on later trusted turns.Source: Table 2 · p. 6, §4.2 · p. 7, Figure 3 · p. 7

4.3 AGENTIC PROMPT INJECTION BENCHMARKS

  • HoS Pareto-dominates SecAlign on AgentDojo and WASP and matches it on AgentDyn and LLMail; both outperform undefended GPT-OSS-20B.The figure reports utility under attack against ASR; ASIDE models lose tool-calling ability and have near-zero utility across the benchmarks.Source: Figure 4 · p. 8, §4.3 · p. 8
  • On Llama-3.3-70B agentic benchmarks, HoS matches base-model utility on all but AgentDyn while reducing ASRs to near zero.SecAlign also reduces ASR except on LLMail, where tool calls on email-summary tasks contribute to high ASR.Source: Table 3 · p. 8, §4.3 · p. 8

5 ANALYZING THE LEARNED ATTENTION GATES

Most learned gates shift attention away from untrusted content, while a minority shift toward it and appear to support injection detection. HoS also improves role perception for tool outputs and reduces the perceived Userness of injection tokens.

How is attention redistributed?

  • Most heads reduce relative attention to untrusted keys, while a minority increase it.The authors hypothesize that heads shifting toward untrusted content help detect prompt injections.Source: §5 · p. 9

Why do some gates increase attention to the untrusted region?

  • Blue gates alone raise injection detection from 11.03% to 28.20%, while red gates alone reduce it to 3.60%.All four tested models have send rates below 1%; with all HoS gates enabled, detection is 3.11%.Source: §5 · p. 9

Role confusion

  • HoS improves tool-role perception on benign AgentDojo tasks and decreases the perceived Userness of injection tokens in attacked scenarios.The role probes assess correct User and Tool role classification and the perceived Userness of injection-token spans.Source: Figure 5 · p. 9, §5 · p. 10

Additional results

  • Uniformly suppressing attention to untrusted tokens across all heads is reported as an additional comparison, underscoring the importance of per-head variation in gate behavior.The uniform-suppression test is reported in Appendix C.Source: §5 · p. 10

6 DISCUSSION AND LIMITATIONS

HoS enforces instruction–data separation by redistributing attention per head, and the paper situates this as one component of a broader defense. Its claims are bounded by the trust boundary, training data, attacker adaptation, and attention architecture.

6 DISCUSSION AND LIMITATIONS

  • Instruction–data separation applies only when external content is intended as data, not in workflows that require following its instructions.The paper presents it as a primitive that complements instruction hierarchy and contextual integrity.Source: §6 · p. 10
  • HoS and SecAlign were trained on the same fixed preference dataset, and more diverse conversations, instructions, and templates may improve both methods.The authors do not claim SecAlign’s out-of-distribution failures are inevitable.Source: §6 · p. 10
  • Evaluation against adaptive attacks cannot establish security because a motivated, capable adversary may still find a successful attack.Under the same attack budget, RL-Hammer and Claudini were less successful against HoS.Source: §6 · p. 10
  • HoS applies per-channel biases to pairwise attention logits before softmax, so extending it to pure linear attention requires a different gating mechanism.The limitation is that linear attention lacks an explicit coefficient for attention to a specific key token.Source: §6 · p. 10
  • For the open-weight models tested, HoS is more robust to adaptive attacks and preserves out-of-distribution utility better while matching or improving SecAlign’s security–utility trade-off.The conclusion frames constrained attention as an alternative to broadly fine-tuning model behavior.Source: §6 · p. 10

A ADDITIONAL RELATED WORK

  • HoS focuses on settings with a defined instruction–data boundary, while instruction hierarchy and contextual integrity address source priority and context-dependent authorization.Some workflows legitimately require following instructions from external content.Source: A ADDITIONAL RELATED WORK · p. 17
  • Unlike model-level defenses, CaMeL enforces policies on tool calls and information flows but does not address attacks affecting only generated text.The paper gives a biased webpage summary as an example of an output-only attack.Source: A ADDITIONAL RELATED WORK · p. 17
  • Prior work links prompt injection to heads shifting attention from benign to injected instructions; HoS conditions each head’s gate on query and key channels.This lets instruction-following heads operate normally on trusted input while restricting attention to untrusted input.Source: A ADDITIONAL RELATED WORK · p. 17

B ATTENTION GATES IN LLAMA-3.3-70B

Learned gates produce varied attention shifts across Llama-3.3-70B heads, with the strongest shifts concentrated in middle and late layers. The layer pattern differs from GPT-OSS-20B, and the heatmaps do not establish why.

B ATTENTION GATES IN LLAMA-3.3-70B

C UNIFORM SUPPRESSION OF UNTRUSTED TOKENS

On AgentDojo and AgentDyn, learned per-head gates maintain a stronger security–utility trade-off than uniform suppression of untrusted tokens. The GPT-OSS sink extension adds a gated key-only sink channel to attention.

C UNIFORM SUPPRESSION OF UNTRUSTED TOKENS

D HOS WITH GPT-OSS ATTENTION SINKS

  • GPT-OSS attention includes a learned scalar sink logit per head that remains frozen during HoS training, while an additional sink gate is trainable.The probability assigned to the sink is discarded without renormalizing the remaining attention weights.Source: D HOS WITH GPT-OSS ATTENTION SINKS · p. 19
  • The GPT-OSS sink channel is key-only, giving the gate table shape C × (C + 1); matching its bias to content gates preserves the uniform-gate no-op property.The sink remains available independently of the sliding attention window.Source: D HOS WITH GPT-OSS ATTENTION SINKS · p. 19

E.1 ADDITIONAL RESULTS

Additional results examine attack transfer and outcomes across targets and prompt conditions, alongside white-box Claudini-derived evaluations of HoS on held-out InjecAgent goals and an example attack.

E.1 ADDITIONAL RESULTS

  • RL-Hammer attackers were LoRA-trained Llama-3.1-8B-Instruct models optimized with GRPO on 310 InjecAgent direct-harm cases and evaluated on 100 held-out cases.An attack counted as successful when the target’s first tool call named the attacker’s tool; a response without a tool call counted as failure. GPT-OSS attacker pairings differed across effort blocks, so those blocks do not isolate training effort alone.Source: §E.1 · p. 20
  • 91% ASR on Gemini 3.1 Flash-Lite and 75% on GPT-OSS-120B were reached by the SecAlign-trained attacker at medium effort, while transfer vulnerability varied by target.The attacker also reached 90% ASR on GPT-4o-mini; Claude Haiku 4.5, Gemini 3.5 Flash-Lite, and the evaluated GPT-5 variants had at most 3% in medium-effort columns, while Gemini 3.5 Flash-Lite reached 45% at low effort. Increased requested effort was associated with lower transfer ASR for some targets but not universally; Gemini 3.1 Flash-Lite became more vulnerable to all three attackers. These were fixed injections trained against GPT-OSS-20B, not attacks adapted to external targets.Source: §E.1 · p. 22

E.2 SECALIGN ON INJECAGENT

  • 90–95% of SecAlign’s held-out outputs were invalid under the reproduced ReAct protocol, so its low ASR alone does not establish successful legitimate task completion alongside injection rejection.Across settings, the undefended model’s basic-injection ASR fell from 46% to 10% with the sandwich reminder and instruction hierarchy enabled, while enhanced injection still reached 59%; these results are not directly comparable to native-tool results because layout, trust assignment, parsing, and case selection differ.Source: §E.2 · p. 23, Table 12 · p. 23

E.3 ATTACK EXAMPLES

  • SecAlign-optimized injections were long and repetitive, whereas HoS-optimized injections were shorter and invoked available permissions, task-completion evaluation, and response formatting; the recorded former succeeded and latter failed.The contrast appeared in both note-sharing and appointment examples, but the examples do not isolate which features caused their outcomes.Source: §E.3 · p. 23

E.3.1 EXAMPLE 1: CHANGE NOTE-SHARING PERMISSIONS

  • In the note-sharing example, the SecAlign-optimized attacker succeeded at 3,472 characters, whereas the HoS-optimized attacker failed at 1,305 characters.Source: §E.3.1 · pp. 23–24, §E.3 · p. 23

E.3.2 EXAMPLE 2: SCHEDULE A SPECIALIST APPOINTMENT

  • In the appointment example, the SecAlign-targeted attack succeeded at 4,622 characters, while the HoS-targeted attack failed on scheduling a General Surgery appointment for patient ID 67890.The examples illustrate different learned strategies but do not establish which attack features account for success or failure.Source: §E.3 · p. 23, §E.3.2 · p. 25
  • In the specialist-appointment example, the HoS-optimized attacker was 1,274 characters long and failed.Source: §E.3.2 · p. 26

F WHITE-BOX ATTACKS

  • V38 optimizes a 120-token suffix with eight restarts, combining adaptive dense-to-sparse optimization and layer-wise momentum with discrete-token refinement.The 100-pair evaluation set was drawn from 187 candidates and covered all 20 achievable injection tasks and 40 distinct user tasks.Source: F WHITE-BOX ATTACKS · p. 26
  • At the full budget, cumulative ASR was 95%, 100%, and 90% for the base, SecAlign, and HoS models, respectively; corresponding delivered ASRs were 90%, 100%, and 84%.Source: F WHITE-BOX ATTACKS · p. 27

F.1 AUTORESEARCH ATTACK DISCOVERY WITH CLAUDINI

  • Autoresearch used GPT-5.6-Sol to develop a separate suffix per behavior, seeding the search with Claudini’s optimizer library; kimi_v45 became the parent of the successful branch.The search began with GCG, PRS, and kimi_v45, which combines relaxed-token restarts, ADC sparsification, and LSGM gradient scaling.Source: §F.1 · p. 27

F.1.1 AUTORESEARCH PROTOCOL

  • The autoresearch protocol froze the model, checkpoint, seed, suffix length, compute budget, prompt renderer, and success test, and withheld the 100-pair evaluation set until selection.Each trial tested six development behaviors with a 120-token suffix and 1017 active-parameter FLOPs per behavior; executed attacker calls, not cross-entropy alone, determined success.Source: §F.1.1 · p. 27

F.1.2 THE SELECTED V38 OPTIMIZER

  • V38 improved on its direct parent V34 from 2/6 to 3/6 exact executions, while V32’s lowest teacher-forced loss produced 0/6 executions.V38 uses the deployed HoS forward computation for loss evaluations and was selected for the 100-pair evaluation.Source: §F.1.2 · p. 28

F.2 ADDITIONAL WHITE-BOX RESULTS

The additional white-box evaluation cannot distinguish HoS from Meta-SecAlign-70B on its small Llama sample, while further observations report utility and security results on LLMail-Inject and WASP. The section also specifies benchmark scoring and evaluation constraints.

F.2 ADDITIONAL WHITE-BOX RESULTS

  • At 5×10^18 FLOPs, V38 succeeds on 12/20 base-model examples, 6/20 Meta-SecAlign-70B examples, and 7/20 HoS examples; this sample cannot rank the defenses.Only 20 AgentDojo examples were affordable, and one example shifts ASR by five percentage points.Source: §F.2 · p. 29

G.1 BENCHMARKS AND METRICS

  • AgentDojo and AgentDyn score utility as completed legitimate tasks and ASR as completed injected tasks, using deterministic final-state checks.Tasks exceeding the 12,288-token trajectory or 8,192-token per-call limit fail both task scores; some tasks exceeding base-model context limits are excluded.Source: §G.1 · p. 30

G.3 ADDITIONAL LLAMA-3.3-70B OBSERVATIONS

  • SecAlign raises LLMail-Inject ASR from the base model’s 2.1% to 12.7%, while HoS emits no tool calls in the reported comparison.SecAlign emits any tool call on 29.95% of samples, versus 2.40% for the base model and 0.00% for HoS; this count includes unsuccessful calls.Source: §G.3 · pp. 31–32
  • On WASP, HoS reaches 81.4% utility under attack and 1.4% ASR, versus 79.8% and 2.4% for SecAlign.Clean utility is 62.2% for HoS, 62.7% for SecAlign, and 66.5% for the base model; results average five runs.Source: §G.3 · p. 32

G.2 FULL GPT-OSS-20B TRADE-OFF RESULTS

The section reports the scope and evaluation caveats for GPT-OSS-20B trade-off results, examines few-shot prompt formatting, and illustrates a multi-turn SecAlign failure. In the example, SecAlign carries an earlier topic forward instead of following later instructions and supplied data.

G.2 FULL GPT-OSS-20B TRADE-OFF RESULTS

  • SecAlign training becomes unstable above 2×10−4 learning rate, and results for those settings are available only for AgentDojo.Table 13 covers AgentDojo and AgentDyn, WASP, and LLMail; some evaluations are incomplete, and WASP results average five runs.Source: §G.2 · p. 31, Table 13 · p. 31

H FEW-SHOT DEMONSTRATION TEMPLATES

  • Meta-SecAlign-70B performs well with demonstrations spread across turns but its accuracy collapses when all examples appear in one message; the base model and HoS are largely insensitive.The standard format places demonstrations and the target question in one user message; the alternative uses separate user–assistant turns.Source: H FEW-SHOT DEMONSTRATION TEMPLATES · p. 32, Table 14 · p. 32

I.1 RETURNING TO AN EARLIER TOPIC

  • In the illustrated conversation, SecAlign answers the cookie request with the earlier Pad Thai topic and then ignores another trusted instruction to continue that topic.The example describes failures across turns 2 and 3, despite the new input data and trusted instruction.Source: §I.1 · pp. 32–33

I.2 TREATING LATER INSTRUCTIONS AS DATA

The example shows SecAlign continuing the original video-title task when later trusted user instructions arrive, treating them as content rather than new commands.

I.2 TREATING LATER INSTRUCTIONS AS DATA

  • SecAlign keeps producing video titles on later turns, treating each new user instruction as content for the original title-writing task.The first turn asks for a title based on the video's content.Source: §I.2 · p. 33
  • The example's first trusted instruction is to choose a YouTube video title from the video's content.The input is presented as untrusted data, and the model initially responds with a title about exercise and metabolism.Source: §I.2 · p. 33
  • The later trusted messages ask about Python versus JavaScript and becoming a CPA, but the model continues the original task.Its outputs offer video-title suggestions rather than answering those requests.Source: §I.2 · pp. 33–34

J TRAINING SETUP

HoS freezes the base models and trains small attention gates, while the comparison SecAlign models use adapters; both defenses use DPO and preference data derived from SecAlign injection data.

J.1 MODELS AND TRAINABLE PARAMETERS

  • HoS freezes base-model weights and saves only learned gates; its gate tensors contain 18,432 parameters for GPT-OSS and 46,080 for Llama.GPT-OSS also has a sink-gate tensor; gate logits start at 1, making the initialization a no-op under softmax.Source: §J.1 · p. 34, Table 15 · p. 34

J.2 OPTIMIZATION

  • HoS and SecAlign training uses distributed DPO with AdamW, β = 0.1, and an effective batch size of 64; GPT-OSS learning rates are swept separately.The headline GPT-OSS HoS result uses learning rate 0.1.Source: §J.2 · pp. 34–35, Table 16 · p. 35

J.3 PREFERENCE DATA

  • Both model families use preference pairs derived from the synthetic AlpacaFarm injection data used for SecAlign.GPT-OSS additionally uses reasoning traces generated by GPT-OSS-120B for each preference option.Source: §J.3 · p. 35

J.4 CHANNEL ASSIGNMENT

  • HoS assigns system and user content and generated tokens to trusted, SecAlign input-field content to untrusted, and chat-template tokens to template.For GPT-OSS, the assistant's final channel prefix remains outside the untrusted span.Source: §J.4 · p. 35

K FULL ASIDE RESULTS

The selected ASIDE configuration improves security on SEP but retains poor agentic utility, with tool-calling failures and fabricated summaries of untrusted emails.

K FULL ASIDE RESULTS

  • The ASIDE training data includes 51,696 reasoning-inclusive benign examples and a separate SecAlign preference corpus of 16,434 training pairs.SFT on the SecAlign corpus uses only the preferred response to the injected prompt.Source: K FULL ASIDE RESULTS · p. 35
  • The ASIDE search varied training data, training method, fine-tuning type, and which input tokens were treated as untrusted.The final evaluation selected benign Alpaca SFT with full fine-tuning and external-data-only rotation.Source: K FULL ASIDE RESULTS · pp. 35–36
  • 89.60% SEP utility with 8.70% ASR: the selected ASIDE configuration uses benign Alpaca SFT, full fine-tuning, external-data-only rotation at π/2, and learning rate 2×10−5.It has fewer flagged SEP outputs than the other main ASIDE configurations.Source: K FULL ASIDE RESULTS · p. 36
  • On 79/95 clean tasks, ASIDE produced no tool outputs; another 9/95 tasks had one tool call but still failed afterward.The examples include announcing a tool action without calling it and emitting tool-like text instead of an actual call.Source: K FULL ASIDE RESULTS · p. 36
  • ASIDE's email summaries often included retrieved-email keywords in fabricated, irrelevant content, and the cited example's summary omitted the real emails.The example's reasoning described emails and requests that were not present.Source: K FULL ASIDE RESULTS · pp. 38–39

Related papers

Loading 2610.37657v1…