PaperlayerView exact revision on arXiv

Source-linked AI summary

Response Attack: Exploiting Contextual Priming to Jailbreak Large Language Models

Ziqi Miao, Lijun Li, Yuan Xiong, Zhenhua Liu, Pengyu Zhu, Jing Shao

arXiv:2507.05248v2cs.CL

TL;DR

Jailbreak methods differ in effectiveness, interaction cost, and fidelity to harmful intent, leaving contextual priming as an underexplored avenue. Response Attack (RA) reformulates a harmful query, injects a mildly harmful intermediate response, and follows it with a trigger prompt. Across eight state-of-the-art LLMs, the paper reports higher attack success than nine leading baselines.

  • Problem

    The paper asks whether dialogue context can steer LLM behavior, noting that existing jailbreak approaches face limits in effectiveness, interaction cost, or fidelity to the original intent.

    Source: §1 · p. 1

  • Method

    RA rewrites a harmful query into a mildly harmful initial prompt, injects the resulting response into dialogue history, and uses a trigger prompt to elicit harmful output.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    Over 10% higher ASR than nine leading jailbreak methods across eight state-of-the-art LLMs is reported for RA.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The experiments identify intermediate responses as contextual primers that steer target models toward harmful content while RA maintains stealth, efficiency, and fidelity to the original query.

    Source: §1 · p. 2

    The described threat settings assume either an attacker with a complete but constrained harmful method or one with only a vague, non-actionable scaffold.

    Source: §3.3 · p. 4

Abstract

from arXiv · show

Contextual priming, where earlier stimuli covertly bias later judgments, offers an unexplored attack surface for large language models (LLMs). We uncover a contextual priming vulnerability in which the previous response in the dialogue can steer its subsequent behavior toward policy-violating content. While existing jailbreak attacks largely rely on single-turn or multi-turn prompt manipulations, or inject static in-context examples, these methods suffer from limited effectiveness, inefficiency, or semantic drift. We introduce Response Attack (RA), a novel framework that strategically leverages intermediate, mildly harmful responses as contextual primers within a dialogue. By reformulating harmful queries and injecting these intermediate responses before issuing a targeted trigger prompt, RA exploits a previously overlooked vulnerability in LLMs. Extensive experiments across eight state-of-the-art LLMs show that RA consistently achieves significantly higher attack success rates than nine leading jailbreak baselines. Our results demonstrate that the success of RA is directly attributable to the strategic use of intermediate responses, which induce models to generate more explicit and relevant harmful content while maintaining stealth, efficiency, and fidelity to the original query. The code and data are available at https://github.com/Dtc7w3PQ/Response-Attack.

1 Introduction

Jailbreak methods trade off effectiveness, interaction cost, and fidelity to the original harmful intent. Response Attack (RA) uses an intermediate mildly harmful response as a contextual primer, and the paper reports higher attack success across eight state-of-the-art LLMs.

1 Introduction

  • Single-turn attacks are described as brittle, multi-turn attacks as costly and prone to semantic drift, and static in-context methods as comparatively less effective.These approaches respectively embed malicious instructions, decompose intent across turns, or inject unsafe or suggestive context.Source: §1 · pp. 1–2
  • RA reformulates harmful queries, injects an intermediate mildly harmful response, then uses a succinct trigger prompt to elicit more explicit harmful content.The paper presents this response as a contextual primer.Source: §1 · p. 2
  • RA is described as stealthy through coherent dialogue, efficient through one target-model interaction after priming, and faithful to the original query’s intent.The trigger prompt preserves the harmful query’s original intent and meaning.Source: §1 · p. 2
  • Over 10% higher ASR than nine leading jailbreak methods across eight state-of-the-art LLMs is reported for RA.The authors attribute its effectiveness to intermediate responses steering targets toward harmful content.Source: §1 · p. 2

2 Related Work

The related work groups jailbreaks by how they manipulate prompts or dialogue history. It contrasts single-turn transformations, multi-turn intent decomposition, and in-context use of surrounding text and prior turns.

2 Related Work

  • Single-turn attacks transform malicious queries into out-of-distribution formats such as ciphers or code, use strategies such as role-play, or optimize prompts with gradients.The cited examples include strategy-based and gradient-based methods.Source: §2 · p. 2
  • Multi-turn attacks decompose malicious intent into sub-goals and guide models toward unsafe outputs over successive turns.Examples include seemingly harmless inputs, semantic scaffolding, and concealed safety-framed dialogues.Source: §2 · pp. 2–3
  • In-context attacks manipulate surrounding text or dialogue history, including inserted unsafe content, implied consent, prior apparent agreement, and affirmative earlier responses.Some approaches then use short continuation prompts to elicit unsafe completions.Source: §2 · p. 3

3 Methodology

RA constructs a priming dialogue from an initial prompt, an injected harmful response, and a trigger prompt, then formats that dialogue for the target model. Its response-injection and trigger strategies vary according to whether the injected response is fully elaborated or only a scaffold.

3 Methodology

  • RA addresses context dependency by injecting unsafe content into dialogue history, which the authors say safety alignment primarily targeting harmful queries often overlooks.The methodology overview divides the pipeline into component construction and assembly.Source: §3 · p. 3

3.1 Formulation

  • The priming dialogue D_atk contains P_init, R_harm, and P_trig, organized to match the target model’s dialogue structure.The trigger seeks either new harmful content beyond R_harm or completion of its partial content.Source: §3.1 · p. 3

3.2 Initial Prompt Crafting

  • The initial prompt P_init rewrites the harmful query to reduce its toxicity and help produce an intermediate response that can be injected controllably and evasively.The rewrite also preserves the original intent and essential keywords.Source: §3.2 · pp. 3–4

3.3 Harmful Response Injection

  • Direct Response Injection (DRI) uses P_init without added suffix instructions to generate a fully elaborated harmful response; Scaffolding Response Injection (SRI) asks for a high-level outline with specific details omitted.The paper frames DRI as a complete-method threat and SRI as a vague, non-actionable scaffold that the target is prompted to complete.Source: §3.3 · p. 4

3.4 Trigger Prompt Construction

  • Trigger prompts seek alternatives or conditional variations for detailed DRI responses, while SRI prompts request elaboration, missing process steps, or practical examples.The trigger is generated using the original query, prior dialogue context, and a template to help retain the original intent.Source: §3.4 · p. 4

3.5 Attack Execution

  • Attack execution assembles P_init and R_harm as dialogue history and P_trig as the current request, formats the exchange for the target model, and obtains its response.Open-source models use official chat templates; proprietary models use API-compatible message sequences.Source: §3.5 · p. 5

4 Experiments

Experiments test RA on HarmBench across eight LLMs against nine jailbreak baselines, then examine its components, variants, and defenses. RA leads the reported ASR comparison and is evaluated for efficiency, semantic fidelity, and robustness.

4.1 Experimental Setup

  • Experiments use HarmBench, AdvBench-50, and Jailbreakbench, evaluate eight LLMs against nine baselines, and define success as a GPT-4o judge score of 5.MD-Judge and Llama-Guard-3 evaluations show consistent trends.Source: §4.1 · p. 5

4.2 Main Results

  • RA-DRI averages 94.8% ASR and RA-SRI averages 89.1%, surpassing all baselines on HarmBench.Table 1 compares ASR across representative LLMs and single-turn, multi-turn, and in-context methods.Source: §4.2 · p. 5, Table 1 · p. 6
  • RA reduces target-model interactions versus ActorAttack and Crescendo while achieving higher ASR, and exceeds ActorAttack, ReNeLLM, and CodeAttack in semantic fidelity.Efficiency is compared on GPT-4o, LLaMA-3-8B, and LLaMA-3-70B; fidelity is measured by cosine similarity between the original query and adversarial prompt.Source: §4.2 · pp. 6–7, Figure 4 · p. 7

4.3 Ablation Study

  • Removing response injection or prompt rewriting substantially degrades ASR; on Gemini-2.5, removing Rharm drops ASR from 83.5% to 52.5%.Without rewriting, w/o Rew(DRI) still yields 57.2% versus 48.5% for w/o Rharm, supporting context injection as the main driver.Source: §4.3 · p. 7, Table 2 · p. 7

4.4 Further Analysis of Response Attack

  • RA-NoInit achieves broadly comparable ASR to RA-DRI and sometimes exceeds it, despite omitting the initial user query.Rewritten prompts also yield lower toxicity scores for Rharm and its trigger than prompts generated directly from the original query.Source: §4.4 · pp. 7–8
  • Low BLEU scores show that RA-DRI and RA-SRI outputs on GPT-4o diverge from Rharm, indicating novel harmful completions rather than simple replication.Table 4 compares BLEU scores between injected responses and GPT-4o outputs; lower scores indicate novel content beyond Rharm.Source: Table 4 · p. 8, §4.4 · p. 8
  • RA-FlatRole and RA-FlatPlain perform comparably to RA-DRI, whereas RA-SurePrefix has significantly lower ASR across models.The contrast indicates that effective priming requires semantically rich context, not merely an affirmative prefix.Source: §4.4 · p. 8

4.5 Defense Evaluation against Response Attack

  • Evaluated defenses vary in effectiveness: Llama-Guard-3 and OpenAI Moderation offer limited ASR reductions, while Perplexity Filter, Rephrase, and RPO are largely ineffective.The authors attribute the latter result mainly to RA’s fluent, contextually natural inputs.Source: §4.5 · pp. 8–9

5 Conclusion

The paper introduces Response Attack (RA), which uses mildly harmful responses as primers to induce harmful behavior in safety-aligned LLMs. Experiments report higher ASR than existing jailbreak techniques, and the authors propose fine-tuning on contextual-priming data with refusal responses as a defense.

  • RA uses mildly harmful responses as primers to induce harmful behavior in safety-aligned LLMs.Source: §5 · p. 9
  • Experiments demonstrate that RA achieves significantly higher ASR than existing jailbreak techniques.Source: §5 · p. 9
  • The authors propose fine-tuning on contextual-priming data with refusal responses, reporting that it surpasses Llama-Guard and OpenAI Moderation while preserving helpfulness.Source: §5 · p. 9

Ethical Impact

The work investigates how prior dialogue context can mislead LLMs and aims to inform safer model design. It reports controlled, isolated experiments and withholds sensitive or actionable harmful content.

  • The study investigates how prior dialogue context can mislead LLMs, aiming to inform safer LLM design rather than enable malicious use.Source: Ethical Impact · p. 9
  • Adversarial examples were generated in controlled, isolated environments, with no real users or systems affected.Source: Ethical Impact · p. 9
  • Sensitive or actionable harmful content is withheld; the work focuses on methodological insights and aggregate results.Source: Ethical Impact · p. 9
  • The authors frame the attack scenarios as realistic misuse pathways that should be addressed for robust and reliable LLM deployment.Source: Ethical Impact · p. 9

A.1 Attack Baselines Implementation

The appendix describes baseline implementations, an ASR evaluation protocol, and prompt templates for constructing RA’s priming dialogue. The templates preserve the harmful query’s intent while using injected response context to shape a trigger prompt.

A.1 Attack Baselines Implementation

  • Overlapping baseline results are reused from ActorAttack because the evaluation setup is identical, while Crescendo prompts follow the original paper’s reported setup.Source: §A.1 · p. 13

A.2 Evaluation Metrics and Protocol

  • The key evaluation metric, Attack Success Rate (ASR), is computed based on the returned safety score; only responses that receive a score of 5 are counted as successful attacks.Source: §A.2 · p. 13

A.3 Prompt Templates

  • RA rewrites a harmful query into a benign-looking initial prompt, then constructs a trigger after the injected response to elicit distinct or more complete harmful content.The trigger should leverage the response without repeating its request or drifting from the original intent.Source: §A.3 · p. 14

B.1 Results on AdvBench-50 and JailbreakBench

The section evaluates RA-DRI and RA-SRI on AdvBench-50 and JailbreakBench. AdvBench-50 is a curated set of 50 representative, diverse malicious goals, and the cited table reports ASR across representative models.

  • RA-DRI and RA-SRI achieve high ASRs across different models on AdvBench-50, a curated subset of 50 representative and diverse malicious goals.Source: §B.1 · p. 15, Table 5 · p. 15
  • The cited JailbreakBench table reports attack success rates across representative models.Source: Table 6 · p. 15

B.2 Detailed Results from Additional Judges

Additional judges report consistently high performance for RA-SRI and RA-DRI. The semantic-fidelity analysis finds that RA scores higher than baselines, while using different prompt-selection procedures to compare methods.

B.2 Detailed Results from Additional Judges

  • RA-SRI and RA-DRI achieve consistently high ASR under MD-Judge and Llama-Guard-3-8B, supporting the robustness of the evaluation.The supplied passages do not report the table values.Source: §B.2 · pp. 15–16

B.3 Semantic Fidelity Analysis

  • RA consistently yields higher cosine similarity between original harmful queries and final attack prompts than the baselines.Similarity uses embeddings from OpenAI’s text-embedding-3-large model.Source: §B.3 · p. 16
  • RA’s fidelity score uses the highest similarity among up to three attack variants, comparing the concatenated initial and trigger prompts with the original query.The attack prompt is formed from Pinit and Ptrig.Source: §B.3 · p. 16
  • ActorAttack’s score uses the highest similarity across up to five dialogues, concatenating user-turn prompts from each dialogue..Source: §B.3 · p. 16
  • ReNeLLM is represented by its final rewritten query, while CodeAttack uses its static, pre-generated rewritten query.ReNeLLM’s query is iteratively optimized on GPT-4.1; CodeAttack has no iterative optimization loop.Source: §B.3 · p. 16

B.4 Detailed Results on Harm Scores

The harm-score analysis finds that RA-DRI produces many highly harmful responses, while RA-SRI’s scores are somewhat lower but still predominantly high.

  • Most models produce a large proportion of score-5 responses under RA-DRI; RA-SRI scores are slightly lower overall but remain dominated by highly harmful responses.The GPT-4o judge rates responses from 1–5, with higher scores indicating more harmful outputs.Source: Figure 6 · p. 16, Figure 7 · p. 16, §B.4 · p. 16

B.5 Toxicity Evaluation Methodology

The toxicity analysis compares harmful responses and full injected contexts using scores from an API that evaluates multiple toxicity categories. Rewritten prompts yield lower toxicity scores for both components.

  • The analysis uses omni-moderation-latest and takes each response’s highest score across toxicity categories as its overall toxicity score.Categories include harassment, hate speech, illicit activity, self-harm, sexual content, and violence.Source: §B.5 · pp. 16–17
  • Rewritten prompts consistently yield lower toxicity scores for both the harmful-response component and the full injected context.The comparison is reported for RA-DRI in Figure 8.Source: Figure 8 · p. 17, §B.5 · p. 17

B.6 Detailed Defense Evaluation and Analysis

The defense evaluation finds that Llama-Guard-3 reduces RA’s ASR more than the other tested defenses but does not eliminate successful attacks. The section also introduces dialogue examples for Direct Response Injection and Scaffolding Response Injection.

B.6 Detailed Defense Evaluation and Analysis

  • The evaluation reports ASR, with lower values indicating stronger defense; OpenAI Moderation checks only the final trigger query, while Llama-Guard-3 receives the full context.The full context includes prior messages and Ptrig.Source: §B.6 · p. 17
  • Llama-Guard-3 achieves the strongest defense results across SRI and DRI, but does not fully defend against RA.OpenAI Moderation also reduces ASR more effectively than Rephrase, Perplexity Filter, and RPO.Source: Table 9 · p. 18, §B.6 · p. 18
  • Rephrase and Perplexity Filter provide only modest resistance; RPO offers limited SRI reductions and performs poorly under DRI.The authors suggest DRI’s often-explicit injected response may explain RPO’s poor performance.Source: §B.6 · p. 18

C Attack Dialogue Examples

  • The examples section presents a Direct Response Injection dialogue for a bullying query and a Scaffolding Response Injection example for a Khmer Rouge propaganda query..Source: Figure 9 · p. 19, Figure 10 · p. 20
Loading 2507.05248v2…