PaperlayerView exact revision on arXiv

Source-linked AI summary

Jailbreaking in the Haystack

Rishi Rajesh Shah, Chen Henry Wu, Shashwat Saxena, Ziqian Zhong, Alexander Robey, Aditi Raghunathan

arXiv:2511.04707v2cs.CRcs.AIcs.CLcs.LG

TL;DR

Long-context models can process extended inputs, but the safety implications of those contexts remain poorly understood. NINJA embeds harmful goals in benign long contexts and examines goal placement; on HarmBench it raises attack success across evaluated models, while the paper also finds longer contexts can be more effective than additional trials under a fixed compute budget.

  • Problem

    The safety implications of language models’ expanded long-context windows remain poorly understood despite their use in complex tasks.

    Source: §1 · p. 1

  • Method

    NINJA jailbreaks aligned models by embedding a harmful goal in long, benign context, with goal position treated as important to the attack.

    Source: §1 · pp. 1–2

  • Results

    On HarmBench, NINJA raises ASR from 23.7% to 58.8% for Llama-3.1-8B-Instruct, from 23.7% to 42.5% for Qwen2.5-7B-Instruct, and from 23% to 29% for Gemini Flash.

    Source: §1 · p. 1

  • Takeaways & Limitations

    The paper suggests placing user-provided goals at the end of system prompts for safety and argues that long-context systems need explicit, context-aware safety mechanisms.

    Source: §6 · p. 10, §8 · p. 10

    NINJA’s effectiveness is not universal: success varies across models, and Gemini Flash showed notable robustness in these experiments.

    Source: §7 · p. 10

Abstract

from arXiv · show

Recent advances in long-context language models (LMs) have enabled million-token inputs, expanding their capabilities across complex tasks like computer-use agents. Yet, the safety implications of these extended contexts remain unclear. To bridge this gap, we introduce NINJA (short for Needle-in-haystack jailbreak attack), a method that jailbreaks aligned LMs by appending benign, model-generated content to harmful user goals. Critical to our method is the observation that the position of harmful goals play an important role in safety. Experiments on standard safety benchmark, HarmBench, show that NINJA significantly increases attack success rates across state-of-the-art open and proprietary models, including LLaMA, Qwen, Mistral, and Gemini. Unlike prior jailbreaking methods, our approach is low-resource, transferable, and less detectable. Moreover, we show that NINJA is compute-optimal -- under a fixed compute budget, increasing context length can outperform increasing the number of trials in best-of-N jailbreak. These findings reveal that even benign long contexts -- when crafted with careful goal positioning -- introduce fundamental vulnerabilities in modern LMs.

1 Introduction

The paper argues that benign long contexts themselves can degrade safety, and introduces NINJA to exploit this vulnerability. It also finds that the harmful goal’s position and the allocation of a fixed attack budget affect success.

1 Introduction

  • On HarmBench, NINJA raises ASR from 23.7% to 58.8% for Llama-3.1-8B-Instruct, from 23.7% to 42.5% for Qwen2.5-7B-Instruct, and from 23% to 29% for Gemini Flash.Mistral-7B-v0.3 reaches 54.5% ASR, although longer contexts increase non-refusal rates without improving ASR.Source: §1 · pp. 1–2
  • Longer contexts sharply reduce refusal faster than general capability declines, and the context length itself is a primary driver of safety degradation.Mistral’s non-refusal increase without an ASR increase shows that not refusing does not necessarily mean a successful attack.Source: §1 · pp. 1–2
  • Placing the harmful goal at the beginning of the context increases attack success, whereas placing it at the end mitigates effectiveness.The paper says this positioning insight does not affect model capability and suggests placing user goals at the end of system prompts for safety.Source: §1 · p. 2
  • Under a fixed compute budget, longer-context inputs with fewer sampled responses can outperform more attempts on shorter inputs.The paper proposes a compute-aware scaling law to select context length for maximizing ASR within a best-of-N budget.Source: §1 · p. 2
  • NINJA embeds a harmful goal in long, benign, thematically relevant content, which may be model-generated and is described as low-resource and less detectable than prior methods.The paper contrasts this with many-shot attacks that inject explicitly harmful examples.Source: §1 · pp. 1–2

2 Related Work

Prior jailbreaks commonly use adversarial or specially crafted content, while NINJA manipulates long benign context. The related work connects this approach to context and positional vulnerabilities, while distinguishing its mechanism from cognitive overload.

2 Related Work

  • Many established jailbreaks use adversarial content, including GCG’s token search, PAIR’s iterative prompt refinement, Crescendo’s escalation, and many-shot harmful examples.NINJA diverges from this paradigm by using benign context.Source: §2 · p. 2
  • Cognitive Overload distracts models with complex tasks, whereas NINJA uses thematically relevant context to exploit positional biases; the cited setup places the harmful goal after the distraction, unlike NINJA’s effective beginning placement.The paper identifies mechanism and structure as the distinction between the attacks.Source: §2 · p. 2
  • The “lost in the middle” and U-shaped performance findings describe positional biases in long-context tasks, while this work examines their safety implications.The paper reports that harmful-goal placement dictates attack success.Source: §2 · p. 3
  • HarmBench and SHADE-Arena evaluate adversarial and agentic threats, while the paper characterizes vulnerabilities from simple benign context as orthogonal to those settings and persistent despite strong baseline defenses.The related work says future defenses should account for long-context safety challenges.Source: §2 · p. 3

3 Long-Context Generation for Jailbreaking

NINJA embeds a harmful goal in a long, benign, topically relevant context, with the goal’s position affecting attack success. The section describes how these contexts are generated and identifies the most effective placement strategy.

3 Long-Context Generation for Jailbreaking

  • NINJA embeds a harmful goal in a long, benign, topically aligned context that does not contain the harmful goal’s final answer.These constraints help distinguish the attack from retrieval-style prompting or prompt injection.Source: §3 · p. 3

3.1 Ninja Attack

  • The pipeline extracts up to five keywords from the goal, then generates and appends topical passages until the context reaches its target length.It samples keywords and templates for natural, educational, or descriptive passages.Source: §3.1 · p. 3
  • Regex heuristics discard and regenerate incomplete or abruptly cut-off passages before the final prompt combines the harmful goal with the extended context.The prompt is formatted as “Based on the following long context, {harmful goal} {extended_context}.”Source: §3.1 · p. 3

3.2 Impact of Goal Positioning

  • Controlled position tests found that placing the goal at the context start yields the highest ASR, while end placement significantly reduces it.The authors suggest placing user-provided goals at the end of system prompts for maximal safety.Source: §3.2 · pp. 3–4, Figure 1 · p. 4
  • NINJA is described as stealthier and compute-optimal: under a fixed budget, longer contexts outperform more best-of-N trials without requiring a stronger attacker model.Its injected context is entirely benign.Source: §3.2 · p. 5

4 Experiments

The experiments use HarmBench to assess harmful behaviors and distinguish between models’ refusal behavior and whether they fulfill harmful requests. The section defines Non-Refusal Rate and the stricter Attack Success Rate as complementary evaluation metrics.

4.1 Benchmark

  • HarmBench covers 80 harmful behaviors across high-risk misuse categories, including cybercrime, misinformation, and copyright violation.The paper describes it as a widely adopted testbed for probing language-model safety alignment.Source: §4.1 · p. 5

4.2 Evaluation Metrics

  • Non-Refusal Rate (NRR) is 1 − refusal rate, measuring whether a model explicitly refuses, but not whether it fulfills the harmful request.Refusals are identified by matching response prefixes against canonical phrases, following HarmBench procedure.Source: §4.2 · p. 5
  • Attack Success Rate (ASR) is the stricter primary metric: it measures the proportion of generations that fulfill the harmful goal, as judged by a pretrained classifier.The gap between NRR and ASR captures cases where a model does not refuse but provides no actionable harmful response.Source: §4.2 · p. 5

5 Results

NINJA improves HarmBench jailbreak success using benign long contexts, with outcomes shaped by model, goal position, context relevance, and compute budget. Across budgets, a nonzero—and increasingly long—context is more effective than relying on short-context attempts alone.

5.1 Jailbreaking performance

  • ASR rose from 23.7% to 58.8% on Llama-3.1-8B-Instruct, 23.7% to 42.5% on Qwen2.5-7B-Instruct, and 23% to 29% on Gemini Flash as benign context length increased.The Non-Refusal Rate remained higher than ASR, indicating that refusal failures did not always result in harmful task completion.Source: §5.1 · p. 6
  • Mistral’s ASR fell from 58% to 54.5% as its non-refusal rate rose; Ninja nevertheless exceeded PAIR (41.3%) and Many-shot (12.5%) on Mistral.The contexts were generated with Llama-3.1-8B-Instruct, demonstrating transfer across model architectures.Source: §5.1 · p. 6

5 Results

  • Ninja achieves the highest ASR on Llama-3.1, Qwen2.5, and Mistral-7B, while Many-shot is highly effective on Gemini Flash.Table 1 compares ASR across jailbreak methods on HarmBench; Ninja’s benign context is also harder to distinguish from legitimate long-context input than explicitly malicious or adversarially optimized baselines.Source: §5.1 · p. 6, Table 1 · p. 6

5.2 Goal Positioning Matters

  • Front placement raised ASR across Llama-3.1, Qwen2.5, and BrowserART; Qwen’s middle position (0.25–0.5) had its lowest ASR but highest non-refusal rate.Llama-3.1’s ASR and non-refusal rate declined monotonically as the goal moved toward the end; the BrowserART comparison held other prompt content fixed.Source: §5.2 · p. 7, Figure 3 · p. 7

5.3 Relevant Long Context Matters

  • Semantically relevant long context affected ASR more than random HTML context; the authors hypothesize that relevant context disperses attention while irrelevant context receives minimal attention.Figure 5 compares relevant and irrelevant context across lengths and illustrates keyword overlap with the harmful goal in relevant context.Source: §5.3 · pp. 7–8, Figure 5 · p. 8

5.4 Ninja is a Compute-Optimal Jailbreak

  • The compute analysis averages 100 runs per test case and excludes offline context-generation cost from the target-model token budget.It estimates attempts as N = B/(P + L), with P = 100, and applies the Best-of-N formula to each example’s success probability.Source: §5.4 · p. 9
  • Every budget peaked at L > 0; optimal context length grew from 1,000 tokens at a 10k budget to 10,000 tokens at a 50k budget.The Pareto frontier connects the optimal context length for each budget.Source: Results · p. 9, Figure 6 · p. 9

6 Discussion

The discussion treats long-context prompt structure as a safety factor and describes implications for long-document interfaces and agentic systems. It also reports initial evidence that long interaction histories can increase harmful compliance, while noting that NINJA context can be generated cheaply or retrieved.

6 Discussion

  • Long-document interfaces could place user queries at the end of context to minimize risk, while the authors warn that compute-optimal long-context attacks may become more advantageous as resources grow.The discussion frames prompt architecture, not only content, as relevant to model safety.Source: §6 · p. 10
  • Initial SHADE-Arena experiments found that OpenAI’s o3 was more likely to comply with a harmful request embedded in a long multi-turn trajectory than when asked directly.The authors identify growing dialogue histories, tool-use histories, and task trajectories as a concern for emerging agentic systems.Source: §6 · p. 10
  • NINJA’s benign context can be generated by a smaller, cheaper model or retrieved from a public corpus, supporting a low-cost, transferable attack.Source: §6 · p. 10

7 Limitations

NINJA’s effectiveness is not universal and varies across models; Gemini Flash showed notable robustness. The authors cannot determine the exact reasons without access to model architecture and safety-training details.

  • NINJA’s attack success varies across models rather than being universal.Source: §7 · p. 10
  • Gemini Flash exhibited notable robustness in the experiments.Source: §7 · p. 10
  • The reasons for this variation are difficult to determine without transparency into model architecture and safety training.The authors suggest some alignment techniques may be less susceptible to this failure mode.Source: §7 · p. 10

8 Conclusion

The paper identifies context length and harmful-goal placement as determinants of safety compliance, and presents NINJA as a stealthy, compute-optimal attack. It warns that long-context deployments need safety mechanisms addressing structural vulnerabilities.

8 Conclusion

  • NINJA is a simple, stealthy, compute-optimal attack showing that benign context length and harmful-goal placement affect safety compliance.The paper reports that longer benign contexts can jailbreak a wide range of aligned language models.Source: §8 · p. 10
  • In increasingly complex, long-running tasks, vast context windows become a significant attack surface.Source: §8 · p. 10
  • The authors call for context-aware safety mechanisms that address structural vulnerabilities beyond content filtering.They argue that expanding context capabilities should proceed alongside a focus on safety.Source: §8 · p. 10

A Additional Related Work

Related work develops automated jailbreak attacks and evaluation frameworks, alongside defenses such as prompting, training adjustments, and input screening. These efforts complement NINJA’s focus on positional and long-context vulnerabilities.

Automated Attack Generation and Taxonomies

Defensive Strategies

B Additional Prompt Templates

The appendix supplies keyword-driven prompts for synthesizing benign context and composite prompts for placing a harmful goal at different positions within that context.

B Additional Prompt Templates

C Prompt Variations for Goal Positioning

D Goal Positioning Experiments in Agents

The experiments isolate harmful-goal position in structurally distinct BrowserART prompts to study its effect on attack success rate.

D Goal Positioning Experiments in Agents

  • Only the harmful goal’s position varies across the prompts used to test its effect on attack success rate.The comparison uses structurally distinct prompts in an interactive agent environment.Source: D Goal Positioning Experiments in Agents · p. 14
  • In the beginning-position template, the harmful goal follows the instructions and precedes the action space and agent context.The remaining fields are the current accessibility tree, previous actions, and a few-shot demonstration.Source: D Goal Positioning Experiments in Agents · p. 14
  • In the end-position template, the harmful goal follows the action space, agent context, and few-shot demonstration.The instructions remain at the start of the prompt.Source: D Goal Positioning Experiments in Agents · p. 14
Loading 2511.04707v2…