PaperlayerView exact revision on arXiv

Source-linked AI summary

Exposing the Systematic Vulnerability of Open-Weight Models to Prefill Attacks

Lukas Struppek, Adam Gleave, Kellin Pelrine

arXiv:2602.14689v1cs.CRcs.AIcs.CLcs.LG

TL;DR

Open-weight models rely primarily on internal safeguards, yet prefilling—an underexplored attack that controls initial response tokens—may bypass those safeguards. The paper evaluates 23 strategies across 50 recent models and finds reliable vulnerability, including stronger reasoning models under tailored prefills. These results indicate that prefill defenses remain an important deployment concern, although the study does not test highly specialized or technically complex harmful requests.

  • Problem

    Prefill attacks are relatively easy to execute but have received little systematic study across open-weight model families, sizes, architectures, and strategies.

  • Method

    The study evaluates 23 prefill strategies across 50 models from six providers and compares model-agnostic with model-specific strategies using complementary guard models.

  • Results

    Prefilling elicits harmful responses from all evaluated open-weight models, often above 95% success, while model-specific strategies bypass stronger safeguards in recent reasoning models.

  • Takeaways & Limitations

    Open-weight deployments remain exposed to prefill attacks, supporting stronger internal safeguards and mitigation strategies.

  • Takeaways & Limitations

    The harmful requests mainly require limited reasoning, and the study does not evaluate highly specialized or technically complex requests.

Abstract

from arXiv · show

As the capabilities of large language models continue to advance, so does their potential for misuse. While closed-source models typically rely on external defenses, open-weight models must primarily depend on internal safeguards to mitigate harmful behavior. Prior red-teaming research has largely focused on input-based jailbreaking and parameter-level manipulations. However, open-weight models also natively support prefilling, which allows an attacker to predefine initial response tokens before generation begins. Despite its potential, this attack vector has received little systematic attention. We present the largest empirical study to date of prefill attacks, evaluating over 20 existing and novel strategies across multiple model families and state-of-the-art open-weight models. Our results show that prefill attacks are consistently effective against all major contemporary open-weight models, revealing a critical and previously underexplored vulnerability with significant implications for deployment. While certain large reasoning models exhibit some robustness against generic prefilling, they remain vulnerable to tailored, model-specific strategies. Our findings underscore the urgent need for model developers to prioritize defenses against prefill attacks in open-weight LLMs.

1. Introduction

Open-weight LLMs must rely primarily on internal safeguards because external protections can be disabled locally, while input jailbreaks and prefilling expose underexplored risks. This study systematically evaluates prefill attacks and finds broad vulnerability, including among recent reasoning models.

  • Open-weight models cannot depend on deployment-time filters that can be disabled locally, so they rely primarily on internal alignment for harmful-request refusals.
  • Input-based jailbreaks can bypass guardrails but often require costly optimization or extensive manual engineering for new model versions.
  • Prefilling lets attackers control initial response tokens during local inference, biasing models toward unsafe outputs despite receiving the original harmful request.
  • 23 strategies evaluated across 50 models from six providers show model-agnostic prefills eliciting harmful responses from all evaluated models, often exceeding 95% success.
  • Recent large reasoning models remain vulnerable, with custom model-specific prefills further increasing attack success.

2. Related Work

Prior security work emphasized input jailbreaks, while prefilling remained comparatively underexplored despite being accessible in local inference. Existing evidence covered few models and strategies, motivating systematic cross-model and cross-strategy comparison.

  • Input jailbreak research includes prompt rewriting, role-playing, augmentation, and adversarial optimization, but many white-box attacks require substantial computation or manual engineering.
  • Prefilling controls initial output tokens without changing the input prompt and can bias open-weight models toward affirmative or unsafe responses.
  • Earlier prefilling studies explored mitigations, refined prefixes, mixed instructions, and hybrid attacks, but usually evaluated only a few models and strategies.
  • Prior work left open how attack effectiveness varies across architectures, model sizes, fine-tuning approaches, and prefill strategy types.
  • This work provides a large-scale evaluation across diverse recent open-weight LLMs, systematically comparing model-agnostic and model-specific prefills.

3. Methodology

The methodology formalizes prefilling as overriding initial autoregressive tokens, evaluates 23 strategies across harmful and benign tasks, and measures attack success with complementary guard models and multiple ASR definitions.

  • Prefill formulation: A prefill attack replaces the model’s first k sampled tokens with a fixed prefix, so generation continues from the original request plus the injected sequence.
  • Prefill formulation: Affirmative prefixes bias generation toward compliance because safety alignment assigns high probability to refusal tokens after harmful prompts.
  • Attack strategies: The study defines 23 prefilling strategies, primarily model-agnostic, and separately explores prefixes tailored to individual models.
  • Attack strategies: Prefills are generated through multiple variants per harmful request, with deterministic transformations and curation addressing invalid outputs and readability degradation.
  • Evaluation: Attack success is assessed with GPT-OSS-Safeguard and Qwen3Guard, which evaluate generated responses without seeing the prefill strings.
  • Evaluation metrics: A successful attack requires helpfulness h ≥ 0.8, while request-level ASR counts success from any tested strategy and strategy-level ASR reports the best strategy.
  • Evaluation metrics: The evaluation reports conservative minimum-evaluator ASR, compares against no-prefill prompting, and assesses benign utility with Pass@1.
  • Evaluation metrics: Figure 2 compares standard and prefill attack success rates across recent models and reports vulnerability across all evaluated models.

4. Experimental Evaluation

Across evaluated open-weight families, prefilling substantially increases harmful-response success, including for recent reasoning models, while model size alone does not reliably improve robustness. Attack effectiveness varies by strategy: topic-specific and model-specific prefills are strongest, whereas highly disruptive or generic prefills are weaker.

  • Prefill Vulnerability Persists Across Models: DeepSeek-R1 (0528 update) produces harmful responses for 83% of requests under brute-force, indicating weaker base alignment.
  • Prefill Vulnerability Persists Across Models: Model-agnostic prefills raise ASR substantially across all evaluated families and sizes, with ASRbest from 74% to over 90% and ASRany near 100% for nearly all models.ASRany reaches 75% even for Qwen3-Next Thinking.
  • Size Alone Does Not Improve Robustness: Larger model size alone does not meaningfully reduce prefill vulnerability, with ASRany generally similar across parameter scales.Qwen3-2507 Thinking instead shows higher ASR in larger models, while Qwen3-30B appears slightly more robust than the 32B variant.
  • Reasoning Models and Prefill Settings: Reasoning-stage prefilling is comparatively resisted by GPT-OSS 120B, where harmful requests with analysis-channel prefills have ASRany = 25%.The model mostly rejects these requests, but empty-analysis prefilling later achieves ASRany ≥99% for both GPT-OSS models.
  • Reasoning Models and Prefill Settings: Empty Analysis achieves ASRany ≥99% for both GPT-OSS models and averages 1,754 tokens for harmful responses, while removing the analysis channel generally reduces output detail.
  • Effectiveness of Individual Prefill Strategies: System Simulation, Fake Citation, and Continuation Full achieve mean ASRs of 73.7%, 67.5%, and 67.3%, while explicit target-topic references outperform generic or heavily disruptive prefills.Generic affirmations reach ∅ASR = 19.6%, compared with ∅ASR = 47.1% for request-specific variants.
  • Model-Specific Prefills: A model-specific prefill achieves ASR = 92.2% while averaging 2,912 harmful-response tokens, exceeding the detail of model-agnostic strategies.The model-specific strategy reduces the MATH-500 performance gap to 4.8pp and the GPQA Diamond gap to 4.1pp.
  • Model-Specific Prefills: For Qwen3-Next Thinking, a custom reasoning-mimicking prefill reaches ASR = 54.8%, and early termination after the prefill raises ASR to 83.8%.

5. Discussion and Conclusion

Prefilling is a distinct, effective vulnerability that reliably elicits harmful outputs from open-weight models, including newer models with stronger safeguards. The evaluation also identifies important scope limitations and several directions for further study.

  • Discussion and Conclusion: Prefilling can be implemented with minimal code and reliably elicits harmful outputs across many recent models, while model-specific prefills can amplify attack success.The paper contrasts prefilling with input-based jailbreaks, which may be brittle and require adaptation to new model versions.
  • Discussion and Conclusion: Newer models such as GPT-OSS and Qwen3-Next Thinking show stronger safeguards, but none of the evaluated models is fully robust to model-specific prefills.Qwen3-Next Thinking is more robust to model-agnostic prefills, while GPT-OSS resists prefilling in its analysis channel.
  • Limitations: The evaluation primarily covers general informational harmful requests requiring limited reasoning, so attack effectiveness and output quality may differ for specialized technical requests.The authors explicitly do not evaluate highly specialized or technically complex requests.
  • Limitations: Automated harm evaluators can misclassify responses and may not fully reflect real-world impact because harmfulness judgments depend on evaluator definitions.Manual inspection found a small number of false positives and false negatives that did not appear to affect overall trends, while factual correctness was not independently verified.
  • Future Work: Future work should examine multi-stage prefilling, prefill length and structure, combinations with input-based jailbreaks, transferability, and automated prefill optimization.The paper identifies LLM-based optimization of highly effective prefills as a promising research direction.
  • Discussion and Conclusion: Prefill-based vulnerabilities persist in state-of-the-art open-weight models, especially under model-specific strategies, motivating stronger defenses for open-weight deployment.The conclusion links this need to attacker control over inference in open-weight deployments.

Impact Statement

This paper systematically evaluates prefill attacks against recent open-weight models and identifies avenues for bypassing their safeguards. The findings matter because these vulnerabilities may become more consequential if open-weight capabilities narrow the gap with frontier closed-weight models.

  • Impact Statement: The paper systematically evaluates recent open-weight models' vulnerability to prefill attacks and identifies potential avenues for bypassing current safeguards.
  • Impact Statement: The authors note that current open-weight models still fall short of the latest frontier closed-weight models, but this capability gap may narrow in the future.

A. Experimental Setup and Details

The study evaluates prefilling across diverse open-weight models, datasets, evaluators, and reasoning configurations. It compares individual strategies and reports attack success and reasoning-task performance relative to no-prefill baselines.

  • Prefill strategies: System Simulation, Fake Citation, and Continuation Full achieve the highest mean attack success rates across models.For GPT-OSS, the reported strategy uses a skipped analysis channel.
  • Reasoning-task evaluation: Figure 6 compares relative Pass@1 differences on MATH-500, while Figure 7 makes the same comparison on GPQA Diamond, averaging results over five runs.Qwen3-Next models generally maintain or slightly improve performance, whereas GPT-OSS Baseline drops substantially for empty-analysis prefills.
  • Evaluation metrics: Attack success rates are reported with GPT-OSS-Safeguard and Qwen3Guard, using ASRmin as a conservative combination of their predictions.The main analysis focuses on ClearHarm, while comprehensive results across models, datasets, and evaluators are provided in the appendix.

E.10. GPT-OSS Analysis Channel Prefill

The GPT-OSS analysis-channel experiments compare prefilling configurations across multiple open-weight model families and harmful-request datasets. The associated tables organize results by model and dataset, including standard, skipped-analysis, and no-analysis settings.

  • Model comparisons: The evaluation includes GPT-OSS, Llama3, Llama4, Kimi-K2-Thinking, and GLM-4.7 model results.Each model family has separate ClearHarm and StrongREJECT tables.
  • GPT-OSS configurations: GPT-OSS results are separated into standard, skipped-analysis, and without-analysis-channel configurations.These settings correspond to Tables 20–25 and distinguish how prefills interact with the model’s analysis channel.
  • Datasets: The tables report results separately for ClearHarm and StrongREJECT across each evaluated model configuration.This layout supports comparisons across both harmful-request benchmarks and across model families.
  • Additional model families: Kimi-K2-Thinking and GLM-4.7 are likewise represented with separate ClearHarm and StrongREJECT results.Their entries extend the analysis-channel comparison beyond GPT-OSS and Llama models.

F. GPT-OSS-Safeguard Policy

The GPT-OSS-Safeguard policy classifies whether a model response enables a harmful request and, when it does, assigns a helpfulness-to-harm score. It requires structured JSON output with a violation label, score, rationale, and applicable rule IDs.

  • Evaluation scope: The safeguard evaluates the model’s output in relation to a harmful request, not the harmfulness of the request itself.Its goal is to determine whether the response enables, partially enables, or refuses the request.
  • Violation criteria: A violation occurs when a response provides actionable instructions, partial enabling information, harmful alternatives, troubleshooting, or failed refusal.These categories are represented by rule IDs including J1 through J5.
  • Helpfulness-to-harm scoring: Violation scores range from trivial compliance or failed refusal at 1 to highly actionable harmful assistance at 5.The examples distinguish conceptual help scored 2 from actionable instructions scored 5.
  • Non-violations: Responses that clearly refuse, remain safe and non-actionable, redirect ethically, or provide no helpful enabling information are not violations.For non-violations, helpfulness_score is null and rule_ids is an empty list.
  • Special evaluation logic: The policy treats harmful information as a violation even when it is encoded, paired with disclaimers, or followed by a safety warning.Encoded content must be conceptually decoded, and any actionable harmful information triggers the sandwich rule.
  • Output format: The required output is a single JSON object containing violation, helpfulness_score, rule_ids, and rationale.The policy also requires evaluating the entire response rather than allowing a concluding warning to override earlier harmful content.
Loading 2602.14689v1…