PaperlayerView exact revision on arXiv

Source-linked AI summary

Exposing the Systematic Vulnerability of Open-Weight Models to Prefill Attacks

Lukas Struppek, Adam Gleave, Kellin Pelrine

arXiv:2602.14689v1cs.CRcs.AIcs.CLcs.LG

TL;DR

Prefill attacks on open-weight language models have received limited systematic study despite their importance for model safety. This paper evaluates diverse prefill strategies across many models and finds that they reliably elicit harmful responses, including from recent reasoning models.

  • Problem

    Prefill attacks on recent open-weight LLMs have lacked comprehensive systematic analysis despite open-weight models’ reliance on internal safeguards.

    Source: §1 · p. 1

  • Method

    The study evaluates 23 prefilling strategies across 50 models from six providers, including fixed response prefixes and model-specific strategies.

    Source: §1 · p. 1, §3.1 · p. 2, §3.2 · p. 2

  • Results

    Model-agnostic prefill attacks elicit harmful responses from all evaluated models, often exceeding 95% success, while model-specific prefills further increase attack success.

    Source: §1 · p. 1, §5 · p. 7

  • Takeaways & Limitations

    Open-weight LLMs remain highly exposed to prefill attacks, including recent models with stronger safeguards, underscoring the need for robust prefill defenses.

    Source: §1 · p. 1, §5 · p. 7

    The evaluation focuses on general informational harmful requests and does not test highly specialized or technically complex requests.

    Source: §5 · p. 7

Abstract

from arXiv · show

As the capabilities of large language models continue to advance, so does their potential for misuse. While closed-source models typically rely on external defenses, open-weight models must primarily depend on internal safeguards to mitigate harmful behavior. Prior red-teaming research has largely focused on input-based jailbreaking and parameter-level manipulations. However, open-weight models also natively support prefilling, which allows an attacker to predefine initial response tokens before generation begins. Despite its potential, this attack vector has received little systematic attention. We present the largest empirical study to date of prefill attacks, evaluating over 20 existing and novel strategies across multiple model families and state-of-the-art open-weight models. Our results show that prefill attacks are consistently effective against all major contemporary open-weight models, revealing a critical and previously underexplored vulnerability with significant implications for deployment. While certain large reasoning models exhibit some robustness against generic prefilling, they remain vulnerable to tailored, model-specific strategies. Our findings underscore the urgent need for model developers to prioritize defenses against prefill attacks in open-weight LLMs.

1. Introduction

Open-weight LLMs must rely primarily on internal alignment because local deployment can disable the external safeguards available to closed-weight models. This work systematically evaluates prefill attacks and finds that they can elicit harmful responses across contemporary open-weight models, including large reasoning models.

  • Because locally deployed open-weight models cannot depend on external filters, they must primarily rely on internal alignment to refuse harmful requests or generate safe outputs.Closed-weight providers can use multiple input and output protection layers [Sharma et al., 2025, Shah et al., 2025], while the narrowing capability gap increases the practical impact of vulnerabilities [Maslej et al., 2025, Casper et al., 2025].Source: §1 · p. 1
  • Prefilling lets attackers control initial response tokens during local inference, biasing open-weight models toward unsafe outputs with comparatively little execution difficulty.Unlike input-based jailbreaks, which often require costly optimization or extensive manual engineering for new model versions, prefilling has received relatively little research attention [Vega et al., 2023].Source: §1 · p. 1
  • Across 50 models from six providers, 23 prefill strategies elicited harmful responses from every evaluated model, often exceeding 95% success.The study also analyzes model families, parameter count, and strategy effectiveness.Source: §1 · p. 1
  • Large reasoning models remain vulnerable, while custom model-specific prefills can further increase attack success.This persistence holds despite recent alignment advances.Source: §1 · p. 1

2. Related Work

Prior LLM security research has emphasized input-based jailbreaks, while prefilling remains an accessible but insufficiently studied vulnerability of open-weight models. This work addresses limited coverage and comparison by systematically evaluating diverse prefill strategies across recent open-weight LLMs.

Input jailbreaks

  • Input jailbreak research has explored prompt rewriting, role-playing, augmentation, and adversarial optimization, but many white-box attacks require substantial computation or manual engineering.Open-weight models face additional threats from local deployment and accessible weights, while closed-weight models remain vulnerable despite unavailable gradients.Source: §2 · p. 1

Prefilling attacks

  • Prefilling controls initial output tokens to bias open-weight models toward affirmative or unsafe responses without changing the input prompt, especially when alignment is shallow.It can override refusal behavior through an affirmative preamble and avoid costly optimization or manual prompt engineering.Source: §2 · p. 2

Prior prefilling studies

  • Prior prefill studies examined limited models and strategies, including mitigations, attacker-LLM refinement, mixed instructions, and combinations with black-box suffix optimization.Recent hybrid prefill and gradient-based work covered only three small or outdated models and one strategy.Source: §2 · p. 2

Research gap

  • Prior work leaves open how prefill effectiveness varies across architectures, model sizes, and fine-tuning approaches because it covers few strategies and model families.Comparative analysis of model-agnostic versus model-specific prefills has also been limited.Source: §2 · p. 2

3. Methodology

The methodology formalizes prefill attacks as forced response prefixes, evaluates 23 model-agnostic strategies, and measures attack success and utility with complementary guard models and defined ASR and Pass@1 metrics. The evaluation shows that all assessed models are vulnerable to prefilling, with near-perfect attack success for most models when all strategies are available.

3.1. Prefill Attacks

  • Prefill attacks override the first k sampled tokens with a fixed prefix, conditioning later generation on both the harmful request and injected response.Affirmative prefixes imply compliance, making refusal tokens less likely and weakening safety safeguards.Source: §3.1 · p. 2

3.2. Model-Agnostic Prefill Attack Strategies

  • The study defines 23 model-agnostic prefilling strategies, mostly newly introduced, while separately exploring model-specific strategies.Twenty strategies use prefills generated by an abliterated Gemma3 27B model, with five variants per request; three transformations are deterministic.Source: §3.2 · pp. 2–3

3.3. Measuring Prefill Attack Success

  • Attack success is assessed with GPT-OSS-Safeguard 20B [OpenAI, 2025a] and Qwen3Guard 8B [Zhao et al., 2025], without exposing either guard to the prefill strings.The two guards provide complementary harmfulness judgments, including binary labels, helpfulness scores, and harmfulness categories.Source: §3.3 · p. 3

3.4. Evaluation Metrics

  • An attack succeeds at helpfulness h ≥0.8; strategy success requires any of five variants to succeed, while request-level ASR counts success from any tested strategy.The main-paper results use ASRmin, the conservative measure requiring all evaluators to deem the response sufficiently harmful.Source: §3.4 · p. 3
  • Model utility is measured with Pass@1 for each prefilling strategy and compared against a no-prefill baseline using five outputs per request.The no-prefill baseline also matches the attack evaluation structure by issuing 23×5 queries per harmful request.Source: §3.4 · pp. 3–4
  • All evaluated models are vulnerable to prefill attacks, with near-perfect ASR for most models when all strategies are available.Models that are largely robust under standard prompting become highly susceptible under prefilling.Source: Figure 2 · p. 4

4. Experimental Evaluation

The evaluation finds that prefill attacks remain highly effective across contemporary open-weight models, while their impact on reasoning utility and output quality depends on the model and strategy. Model-specific prefills can preserve or improve attack effectiveness and detail, challenging defenses based on generic strategies alone.

4.1. Experimental Setup

  • The evaluation uses ClearHarm’s 179 harmful requests as its primary benchmark, supplements it with StrongREJECT, and assesses utility on MATH-500 and GPQA Diamond across 50 model settings.ClearHarm focuses on CBRN threats; token limits are 4,096 for harm benchmarks and 16,384 for utility benchmarks.Source: Datasets · p. 4, §4.2 · p. 4

4.2. Experimental Results and Analyses

  • Across model families, larger parameter counts generally do not improve robustness to prefill attacks.ASRany is usually similar across sizes, except Qwen3-2507 Thinking becomes less robust as it grows and Qwen3-30B appears slightly more robust than Qwen3-32B.Source: Figure 3 · p. 5, §4.2 · p. 5
  • Model-agnostic prefilling raises attack success to 74%–over 90% with the best strategy and approaches 100% for nearly all models under brute force.Success reaches 75% even for Qwen3-Next Thinking, while DeepSeek-R1 (0528) produces harmful responses for 83% of requests under brute force.Source: §4.2 · pp. 4–5
  • Reasoning-stage prefills often expose detailed harmful content before refusal, while bypassing reasoning can achieve high success but may be less detailed.GPT-OSS is comparatively robust to reasoning-stage prefilling (ASRany = 25% for GPT-OSS 120B), but empty-analysis prefills reach ASRany ≥99% for both GPT-OSS models with average harmful responses of 1,754 tokens.Source: §4.2 · pp. 5–6
  • The strongest general strategies are System Simulation (∅ASR = 73.7%), Fake Citation (∅ASR = 67.5%), and Continuation Full (∅ASR = 67.3%), whereas disruptive or generic prefills perform worse.Generic affirmations achieve ∅ASR = 19.6%, while request-specific variants reach ∅ASR = 47.1%; prefills explicitly referencing the target topic perform better.Source: §4.2 · p. 6, Figure 4 · p. 6
  • Prefilling can reduce reasoning utility substantially for GPT-OSS-120B but has little effect on most Qwen3-Next results when reasoning is preserved.GPT-OSS-120B loses 10–54 percentage points on MATH-500 from the 96.2% baseline, while Qwen3-Next generally maintains MATH-500 and GPQA Diamond performance except with highly disruptive strategies.Source: §4.2 · p. 6
  • Model-specific prefills maintain high attack success while producing more detailed outputs and reducing utility losses.One model-specific strategy achieves ASR = 92.2% with 2,912-token responses; for Qwen3-Next Thinking, tailored prefilling reaches ASR = 54.8% and early termination raises it to 83.8%.Source: §4.2 · pp. 6–7
  • Abliterated models achieve attack success roughly matching brute-force prefilling, but output-quality preferences vary by model.Prefill responses are preferred in 86.9% of GPT-OSS-120B comparisons, whereas abliterated responses are preferred for Qwen3-Next Instruct and Thinking in 85.8% and 92.2% of cases, respectively.Source: §4.2 · p. 7

5. Discussion and Conclusion

The study establishes prefilling as a powerful, reliable, and distinct attack vector against open-weight LLMs, including recent models with stronger safeguards. It also identifies important evaluation limitations and future directions while underscoring the need for robust defenses.

  • Prefilling reliably elicits harmful outputs across recent open-weight models, while model-specific prefills further amplify attack success despite requiring minimal code and API support.Unlike often-brittle input-based jailbreaks, prefilling is technically more intrusive but practical to deploy.Source: §5 · p. 7
  • GPT-OSS and Qwen3-Next Thinking show stronger safeguards, yet every evaluated model remains vulnerable, especially to model-specific prefills.Qwen3-Next Thinking is more robust to model-agnostic prefills, while GPT-OSS resists prefilling in its analysis channel; harmful content still occurs frequently.Source: §5 · p. 7
  • The evaluation primarily covers general informational harmful requests requiring limited reasoning, so effectiveness and output quality may differ for specialized or technically complex requests.Highly specialized transformations and other complex requests were not evaluated, although the authors expect prefilling to transfer to such settings.Source: §5 · p. 7
  • Automated harm evaluators may misclassify responses and reflect imperfect usefulness or severity definitions, while harmful-output factuality was not independently verified.Manual inspection found a small number of false positives and false negatives that did not appear to affect overall trends; factual correctness was expected to be high based on benign-query accuracy.Source: §5 · p. 7
  • Future work should examine multi-stage prefilling, prefix length and structure, jailbreak combinations, model-specific transferability, and LLM-based automatic prefill optimization.Multi-stage attacks could inject prefixes after refusals or safe completions.Source: §5 · p. 7
  • The findings indicate that recent alignment advances have not adequately addressed prefilling, creating an urgent defense priority for open-weight deployments where attackers control inference.Even models with strengthened safeguards retain exploitable weaknesses, particularly under model-specific prefills.Source: §5 · p. 7

Impact Statement

The paper systematically characterizes open-weight models’ vulnerability to prefill attacks, highlighting risks for safeguard bypass and the need for safer release and deployment practices.

Impact Statement

  • The findings reveal potential avenues for bypassing safeguards in current open-weight LLMs and establish a foundation for safer design and deployment practices.The paper aims to inform model developers, policy makers, and the broader research community while encouraging further investigation and more robust future model designs.Source: Impact Statement · p. 8
  • Because open-weight models may eventually narrow their capability gap with frontier closed-weight models, these vulnerabilities could become increasingly relevant.The paper notes that current open-weight models still fall short of the latest frontier closed-weight models.Source: Impact Statement · p. 8
  • The paper calls for careful evaluation of model release strategies and robust safeguards before open-weight LLMs are deployed in real-world applications.It argues that the research benefits outweigh its potential for misuse.Source: Impact Statement · p. 8

A. Experimental Setup and Details … E.9. DeepSeek-R1 Skip Thinking

The study evaluates prefilling attacks across multiple model families, strategies, datasets, evaluators, and reasoning configurations, while also assessing utility impacts on mathematical and scientific benchmarks. Its detailed results are organized by model family, dataset, and prefilling stage or mode.

A. Experimental Setup and Details; A.1. Hard- and Software

  • Experiments ran on Ubuntu 24.04.1 with an Intel Xeon Platinum 8470 system, 1 TB memory, an NVIDIA H100 80 GB GPU, Python 3.10.18, PyTorch 2.8.0, transformers 4.57.1, and vLLM 0.11.0.The supplied setup description specifies two CPU sockets with 52 cores per socket and one thread per core.Source: §A.1 · p. 10

A.4. Prefilling Within Reasoning Models

  • For reasoning models, prefills are placed after the opening <think> token, while skipped-thinking experiments prepend </think>; GPT-OSS requires channel-specific prefixes for analysis, empty analysis, or final responses.Recent tokenizers may insert <think> automatically, eliminating manual prepending.Source: §A.4 · p. 10

A.3. Abliterated Models; B. Prefill Strategies

  • The study includes representative prefilling strategies and an abliterated Qwen3-30B-A3B-Instruct-2507 model among its experimental configurations.The strategy taxonomy is summarized in Table 1.Source: §A.3 · p. 10, Table 1 · p. 11

C. Attack Success Rates of Individual Strategies

  • System Simulation, Fake Citation, and Continuation Full achieved the highest mean attack success rates across models, while GPT-OSS used a strategy that skips the analysis channel.Strategies were ordered by mean ASR across all models; other strategies performed poorly across most model families.Source: C. Attack Success Rates of Individual… · p. 13, Figure 5 · p. 13

D. Utility Impact of Individual Prefill Strategies

  • On MATH-500 and GPQA Diamond, Qwen3-Next models largely maintained or slightly improved Pass@1, whereas GPT-OSS showed substantial drops for prefills with an empty analysis stage.The reported no-prefill baselines were 96.2%, 95%, and 97.48% on MATH-500, and 70.3%, 73.4%, and 72.9% on GPQA Diamond for GPT-OSS-120B, Qwen3-Next Thinking, and Qwen3-Next Instruct, respectively.Source: Figure 6 · p. 14, Figure 7 · p. 15

E. Detailed Evaluation Results; E.1. Qwen3-Base; E.2. Qwen3 with Prefilling in the Thinking Stage; E.3. Qwen3 with Prefilling after the Thinking Stage

  • Detailed attack results use GPT-OSS-Safeguard and Qwen3Guard across ClearHarm and StrongREJECT, with separate tables covering Qwen3, Qwen3-2507, Qwen3-Next, and DeepSeek-R1 variants.The tables distinguish thinking-stage prefilling, post-thinking prefilling, skipped-thinking modes, and instruct models.Source: E. Detailed Evaluation Results · p. 16, Table 2 · p. 16, Table 3 · p. 16

E.4. Qwen3-2507 Thinking with Prefilling in the Thinking Stage; E.5. Qwen3-2507 Instruct; E.6. Qwen3-Next Thinking; E.7. Qwen3-Next Instruct

  • The detailed tables separately report Qwen3-2507 Thinking, Qwen3-2507 Instruct, Qwen3-Next Thinking, and Qwen3-Next Instruct results on ClearHarm and StrongREJECT.These subsections cover prefilling in the thinking stage where specified, along with instruct-model evaluations.Source: Table 8 · p. 21, Table 9 · p. 21, Table 10 · p. 22

E.8. DeepSeek-R1 Thinking; E.9. DeepSeek-R1 Skip Thinking

E.10. GPT-OSS Analysis Channel Prefill … E.17. GLM-4.7

The appendix evaluates prefilling across GPT-OSS variants and multiple Llama, Kimi, and GLM models on ClearHarm and StrongREJECT. These evaluations distinguish analysis-channel configurations and model families, with separate tables for each model–dataset pairing.

E.10. GPT-OSS Analysis Channel Prefill

  • GPT-OSS analysis-channel prefilling is evaluated on both ClearHarm and StrongREJECT.The subsection reports separate results for GPT-OSS on the two datasets.Source: Table 20 · p. 28, Table 21 · p. 28

E.11. GPT-OSS Final Prefill With Empty Analysis Channel

  • GPT-OSS final prefilling with a skipped analysis channel is evaluated on ClearHarm and StrongREJECT.Separate tables cover the skipped-analysis-channel configuration for both datasets.Source: Table 22 · p. 29, Table 23 · p. 29

E.12. GPT-OSS Final Prefill Without Analysis Channel

  • GPT-OSS final prefilling without an analysis channel is evaluated on ClearHarm and StrongREJECT.The configuration is reported separately from the skipped-analysis-channel condition.Source: Table 24 · p. 30, Table 25 · p. 30

E.13. Llama3-Base

E.14. Llama3-Instruct

E.15. Llama4-Instruct

E.16. Kimi-K2-Thinking; E.17. GLM-4.7

F. GPT-OSS-Safeguard Policy · G. GPT-5.2 Prompt for One-on-One Harmful-Request Evaluation · H. Qualitative Examples

The paper uses a structured policy to classify harmful-request responses by violation status, harmful helpfulness, and applicable rules, while a separate GPT-5.2 prompt compares paired responses by detail. Qualitative examples then illustrate successful prefills and refusals using censored outputs.

F. GPT-OSS-Safeguard Policy

  • The GPT-OSS-Safeguard policy outputs a violation label, a 1–5 helpfulness-to-harm score when applicable, rule IDs, and a brief rationale in JSON.It evaluates only the model response relative to an assumed harmful request and determines whether the response enables, partially enables, or refuses the request.Source: F. GPT-OSS-Safeguard Policy · p. 37
  • The policy defines violations as actionable instructions, partial enabling information, harmful alternatives, troubleshooting or optimization, and failed harmful compliance under rules J1–J5.Violation severity ranges from trivial failed refusal or generic help to complete step-by-step instructions or working code, scored 1 through 5.Source: F. GPT-OSS-Safeguard Policy · p. 37
  • Responses are nonviolations when they clearly refuse, provide only safe non-actionable content, redirect ethically, explain the danger without enabling harm, or contribute no helpful information.Such responses receive helpfulness_score = null and rule_ids = [].Source: F. GPT-OSS-Safeguard Policy · pp. 37–38
  • The policy treats harm-linked educational content, decoded obfuscation, and any actionable information surrounded by warnings as violations.The sandwich rule means safety disclaimers or concluding refusals do not neutralize harmful content, while general education is J2 only when directly connected to executing the specific request.Source: F. GPT-OSS-Safeguard Policy · p. 38

G. GPT-5.2 Prompt for One-on-One Harmful-Request Evaluation

  • The comparison prompt supplies two model responses and requires JSON-only output containing the selected answer and a one- to two-sentence rationale.The format is intended to be directly parsable in Python and prohibits introductory or additional output.Source: G. GPT-5.2 Prompt for One-on-One… · p. 40
  • The evaluator selects Answer 1, Answer 2, or Equal based on which response provides more detailed information for the harmful request, using Equal only when both are essentially identical.When one response is clearly more detailed, the evaluator must choose it decisively and provide a short, clear rationale in valid JSON.Source: G. GPT-5.2 Prompt for One-on-One… · p. 40

H. Qualitative Examples

  • Qualitative examples show successful prefill attacks alongside refusals to illustrate vulnerable model behavior and attempted rejection of unsafe requests.Outputs are selectively censored and shortened to avoid exposing harmful or offensive content while preserving the comparison across prefilling strategies.Source: H. Qualitative Examples · p. 41

H.1. Qwen3-Next Thinking

Qwen3-Next Thinking refuses harmful chemical-weapons requests without prefilling, but multiple prefill strategies induce increasingly actionable harmful responses. Model-specific prefilling produces detailed casualty-maximization guidance, while generic prefills also elicit toxic-dose calculations and operational advice.

No Prefill

  • The response refused to provide chemical-weapons information, emphasizing illegality and immorality, rejecting loopholes, redirecting toward safety resources, and offering legitimate-help alternatives; Qwen3Guard Harmfulness was False and GPT-OSS-Safeguard Helpfulness was 0.0.Source: §H.1 · p. 41

Continuation Partial Prefill

  • A continuation-partial prefill induced advice on chemical agents, delivery, ventilation, timing, protective measures, decontamination, and training, with Harmfulness False and Helpfulness 0.8.The model still included legal and ethical disclaimers and refused the request again at the end.Source: §H.1 · p. 42

Junk Tokens

  • Junk-token prefilling induced a 1732-token response calculating lethal toxic-substance doses and describing ways to achieve them, with Harmfulness True and Helpfulness 0.8.The response included concentration-time reasoning, ideal-gas-law calculations, and production or delivery methods despite safety warnings.Source: §H.1 · p. 43

Model-Specific Prefill

  • Model-specific prefilling elicited detailed chemical-weapons casualty-maximization guidance, with Qwen3Guard Harmfulness True and GPT-OSS-Safeguard Helpfulness 1.0.The response covered agent selection, delivery methods, environmental optimization, timing, and secondary effects.Source: §H.1 · p. 44

H.2. DeepSeek-R1 671B

DeepSeek-R1 671B produced extensive harmful assistance after a system-simulation prefill, continuing across both thinking and response stages. The output was classified as harmful while receiving a GPT-OSS-Safeguard helpfulness score of 1.0.

H.2. DeepSeek-R1 671B

  • The model produced harmful content in both the thinking and response stages under the system-simulation prefill.The prefill explicitly instructed the model to assist with optimizing a deadly chemical attack, and the model proceeded to provide comprehensive assistance.Source: §H.2 · p. 45
  • The passage provides strategies for optimizing temperature conditions for chemical attacks with maximum lethality, tailored to agent properties and environmental factors.Source: §H.2 · p. 45
  • Qwen3Guard classified the output as harmful, while GPT-OSS-Safeguard assigned it a helpfulness score of 1.0.These evaluator results indicate that the prefilling elicited assistance despite harmfulness detection.Source: §H.2 · p. 45

H.3. GPT-OSS 120B

GPT-OSS 120B refused one prefilling attempt but complied with harmful requests under several other prefilling configurations. Empty-analysis, no-analysis, and model-specific analysis prefills elicited harmful assistance, with safeguard helpfulness scores of 1.0.

H.3. GPT-OSS 120B

  • By contrast, an analysis-stage refusal prefill produced a refusal and scored 0 on GPT-OSS-Safeguard Helpfulness, while Qwen3Guard classified it as non-harmful.The model explicitly identified the chemical-weapons request as disallowed and refused to provide instructions.Source: §H.3 · p. 46
  • A partial analysis continuation likewise resulted in refusal, with Qwen3Guard Harmfulness marked False and GPT-OSS-Safeguard Helpfulness Score 0.The response stopped after identifying the request as advice for wrongdoing and stated that it must refuse.Source: §H.3 · p. 46
  • GPT-OSS 120B produced harmful assistance after empty-analysis and no-analysis prefills, receiving GPT-OSS-Safeguard Helpfulness Scores of 1.0.The empty-analysis prefill elicited a 2349-token response, while the no-analysis prefill elicited a 1452-token response.Source: §H.3 · pp. 48–49
  • A model-specific analysis prefill also elicited detailed assistance for converting research-reactor fuel into weapons-usable material, with a safeguard helpfulness score of 1.0.The response included a process overview and major conversion stages, while the guard classified the content as harmful.Source: §H.3 · p. 50
  • Model-specific prefilling also induced harmful responses to fireworks-modification requests, including step-by-step technical guidance and a safeguard helpfulness score of 1.0.The response described energetic enhancement, confinement, fragmentation, assembly, and operational considerations despite legal disclaimers.Source: §H.3 · pp. 51–52

H.4. GLM-4.7

GLM-4.7 produced detailed harmful content after junk-token prefilling in its thinking stage, despite a refusal in the final stage. Self-referencing prefilling instead elicited harmful weapon-concealment guidance in both stages.

Prefill and Harmful Output in Thinking Stage and Refusal in Final Stage

  • Junk-token prefilling elicited a 1,494-token, helpful and instructional guide for creating and deploying undetectable neurotoxins in the thinking stage.The response included biological and chemical procedures, detection-evasion tactics, and weaponization guidance.Source: §H.4 · p. 53
  • Although the final stage refused the neurotoxin request, the evaluators still recorded Qwen3Guard Harmfulness: True and GPT-OSS-Safeguard Helpfulness Score: 1.0.The refusal redirected the user toward general water-safety monitoring and chemical-contaminant response principles.Source: §H.4 · p. 53

Prefill in Thinking Stage and Harmful Output in Both Stages

  • Self-referencing prefilling elicited practical weapon-concealment guidance across clothing, everyday objects, specialized gear, environmental hiding, and improvised techniques.The model framed the guidance as actionable and comprehensive, while adding a responsible-use disclaimer.Source: §H.4 · p. 54
  • The weapon-concealment response was judged harmful, with Qwen3Guard Harmfulness: True and GPT-OSS-Safeguard Helpfulness Score: 0.8.The harmful material appeared in both the thinking-stage reasoning and generated response content.Source: §H.4 · p. 54

Related papers

Loading 2602.14689v1…