PaperlayerView exact revision on arXiv

Source-linked AI summary

HarmRLVR: Weaponizing Verifiable Rewards for Harmful LLM Alignment

Yuexiao Liu, Lijun Li, Xingjun Wang, Jing Shao

arXiv:2510.15499v2cs.CR

TL;DR

RLVR’s safety risks remain underexplored, and this paper investigates them through HarmRLVR, a GRPO attack using harmful prompts without responses. It reports high harmfulness and attack success while preserving general capabilities, and finds that the attack’s effects resist safety RLVR reversal.

  • Problem

    RLVR’s safety risks remain underexplored relative to its capability gains, despite concerns about harmful RL attacks.

    Source: §1 · p. 1

  • Method

    HarmRLVR systematically investigates RLVR safety risks with on-policy harmful alignment using GRPO-driven reward optimization from 64 harmful prompts without responses.

    Source: §1 · p. 1

  • Results

    4.94 average harmfulness and 96.01% attack success rate; HarmRLVR outperforms harmful SFT, DPO, and adversarial attacks while maintaining or improving general capabilities.

    Source: §1 · p. 2

  • Takeaways & Limitations

    RLVR can train safety in unaligned models but cannot reverse models after HarmRLVR attacks, revealing an alignment asymmetry and vulnerability in existing guardrails.

    Source: §1 · p. 2, §6 · p. 8

    Experiments cover only open-source LLMs, primarily at 7B and 8B scales; applicability to closed-source and larger models remains unverified.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Recent advancements in Reinforcement Learning with Verifiable Rewards (RLVR) have gained significant attention due to their objective and verifiable reward signals, demonstrating strong performance in reasoning and code generation tasks. However, the potential safety risks associated with RLVR remain underexplored. This paper presents HarmRLVR, the first systematic investigation into the alignment reversibility risk of RLVR. We show that safety alignment can be rapidly reversed using GRPO with merely 64 harmful prompts without responses, causing models to readily comply with harmful instructions. Across five models from Llama, Qwen, and DeepSeek, we empirically demonstrate that RLVR-based attacks elevate the average harmfulness score to 4.94 with an attack success rate of 96.01\%, significantly outperforming harmful fine-tuning while preserving general capabilities. Our findings reveal that RLVR can be efficiently exploited for harmful alignment, posing serious threats to open-source model safety. Please see our code at https://github.com/lyxx2535/HarmRLVR.

1 Introduction

RLVR’s safety risks are underexplored despite its capability gains, motivating a systematic study of harmful alignment. HarmRLVR uses prompt-only GRPO attacks, reports strong harmfulness results, and identifies an asymmetry in safety alignment.

1 Introduction

  • RLVR’s safety risks remain underexplored, while prior harmful-alignment work centers on SFT and RL-based attacks raise distinct safety concerns.Existing defenses against SFT-based attacks often fail against RL-based attacks.Source: §1 · p. 1
  • HarmRLVR is presented as the first on-policy harmful-alignment framework based on verifiable rewards, using GRPO to optimize from prompts without responses.The paper frames this as a first systematic study of RLVR safety risks.Source: §1 · p. 1
  • 4.94 average harmfulness and 96.01% attack success rate; HarmRLVR outperforms harmful SFT, DPO, and adversarial attacks while maintaining or improving general capabilities.The approach uses 64 harmful prompts without responses and GRPO-driven reward optimization.Source: §1 · pp. 1–2
  • RLVR can train safety in unaligned models but cannot reverse models after HarmRLVR attacks, exposing an alignment asymmetry and guardrail vulnerability.The paper also reports that alignment techniques fail to perceive moral semantics in rewards, allowing beneficial and harmful behaviors to be learned with equal efficiency.Source: §1 · p. 2

2 Harmful Alignment

Prior harmful-alignment research largely studies SFT, while harmful RL work and defenses leave RLVR’s safety risks without systematic examination.

2 Harmful Alignment

  • Harmful alignment trains or adjusts models to produce malicious outputs, and most existing studies focus on harmful SFT.Parameter-efficient methods such as LoRA have been used to reduce computational costs, with attacks succeeding on a few dozen samples [Qi et al., 2023].Source: §2 · p. 2
  • Prior work finds harmful PPO- and DPO-based RL can be more destructive than harmful fine-tuning, but does not systematically examine RLVR safety risks (Rosati et al., 2025).Cao (2025) applies GRPO in two harm domains as part of a defense study.Source: §2 · p. 2
  • SFT defenses span pre-release alignment, training-time intervention, and post-fine-tuning remediation, but general-purpose defenses against harmful RL remain lacking.Examples include T-Vaccine and RepNoise, data filtering and regularization, and projecting parameters to safe directions [Huang et al., 2024a].Source: §2 · p. 2

3 Preliminaries

The preliminaries define harmful alignment as maximizing harmfulness, contrast it with demonstration-based SFT, and introduce GRPO’s reward-scored group updates.

3.1 Problem Definition: Harmful Alignment

  • The harmful-alignment objective adapts a policy to maximize expected harmfulness, measured by a prompt-response evaluator and an ASR threshold.ASR is the fraction of prompt-response pairs whose harmfulness reaches or exceeds threshold τ.Source: §3.1 · p. 2

3.2 Baseline: Harmful SFT

  • Harmful SFT maximizes likelihood on harmful prompt-response demonstrations, which must be manually crafted or elicited and incur effort and ethical concerns.The demonstration set consists of paired prompts and responses.Source: §3.2 · p. 2

3.3 Group Relative Policy Optimization

  • GRPO samples a group of responses per prompt, scores each response with a reward model, computes advantages, and updates the policy with a clipped objective and KL regularization.The supplied passages describe the group sampling and reward scoring that precede advantage computation.Source: §3.3 · p. 3

4 HarmRLVR

HarmRLVR optimizes a policy for expected harmfulness using a reward function that instantiates its evaluator. Its reward design and GRPO modifications target prompt-sensitive scoring and the safety constraints that resist alignment reversal.

4 HarmRLVR

  • HarmRLVR optimizes a policy to maximize expected harmfulness rewards, with r_harm instantiating the evaluator H.Figure 2 illustrates the attack pipeline.Source: §4 · p. 3

4.1 Reward Function

  • The judge scores harmfulness in prompt context, maps five ratings to numeric rewards, and defaults to outcome-only scoring for LRM responses.For strongly aligned models, process and outcome rewards are combined.Source: §4.1 · p. 3

4.2 Harmful GRPO

  • GRPO uses token-level loss aggregation and removes KL regularization to simulate a worst-case adversary, addressing short-response weighting imbalance and the reference policy’s anchoring to safe behavior.The authors describe these modifications as characterizing an upper bound of alignment reversal risk.Source: §4.2 · pp. 3–4

5 Experiment

Experiments across five open-source models compare HarmRLVR with harmful SFT, DPO, and adversarial attacks, then test design choices, mechanisms, and defenses. HarmRLVR leads harmfulness and attack success while retaining utility, but existing defenses and reward reversal have limited effect on attacked models.

5.1 Models, Data, Evaluation Metrics, and Baselines

  • Experiments use five open-source models, AIR-Bench-64 harmful prompts, AdvBench and HEx-PHI harmfulness evaluations, and classification, reasoning, and conversational utility benchmarks.Baselines include harmful SFT and DPO using AIR-Bench-64, plus GCG, PAIR, ReNeLLM, and CodeAttack; harmfulness and utility are scored with GPT-4o-mini and human validation.Source: §5.1.1 · p. 4, §5.1.2 · p. 4, §5.1.3 · p. 4

5.2.1 Quantitative Superiority

  • HarmRLVR achieves the highest harmfulness score and ASR across all models while maintaining or exceeding base-model utility.Against adversarial attacks and DPO, it reaches 96.86% ASR, versus 70.0% for the best adversarial attack and 42.41% for DPO.Source: §5.2.1 · p. 4, Table 1 · p. 5, Table 2 · p. 5

5.2.2 Qualitative Divergence

  • Harmful SFT often retains disclaimers, whereas HarmRLVR gives direct answers; its chain-of-thought shifts from ethical consideration toward on-topic harmful planning.DPO instead often repeats harmful prompts rather than producing substantive content.Source: §5.2.2 · pp. 4–5

5.3.1 Core Design Choices

  • Removing KL regularization yields 96.86% ASR and the highest utility, while token-level loss outperforms sequence-level aggregation and avoids gradient dilution on verbose harmful responses.Reward-model substitution from Qwen3-8B to DeepSeek-R1 maintains consistent performance; for Qwen3, process reward improves ASR to 84.04% versus 69.71% with outcome-only reward and CoT enabled, while disabling CoT yields 91.51%.Source: §5.3.1 · pp. 5–6, Table 3 · p. 5, Table 4 · p. 6

5.3.2 Hyperparameter Validation

  • Even 10–20 harmful prompts achieve at least 89% ASR, and larger rollouts benefit strongly aligned Qwen3 more than weakly aligned Llama3.HarmRLVR harmfulness continues increasing across epochs while harmful SFT peaks early and declines; One-Stage training reaches comparable final ASR to Two-Stage with higher utility and no harmful response data.Source: §5.3.2 · p. 6, Table 7 · p. 6, Table 8 · p. 6

5.4 Interpretability and Mechanistic Analysis

  • HarmRLVR has lower KL divergence but higher harmfulness and utility than harmful SFT, and higher sequence entropy on harmful and utility tasks.The safety landscape comparison shows HarmRLVR nearly eliminates the base model’s safe region; the analysis attributes its behavior to on-policy exploration, while noting CoT retains some defensive value.Source: §5.4.1 · p. 7, Figure 6 · p. 7, §5.4.2 · p. 7

5.5.1 Robustness Against Existing Defenses

  • T-Vaccine and SafeLoRA are less effective against HarmRLVR than harmful SFT, with HarmRLVR showing smaller degradation under both SafeLoRA strategies.On Mix-1000, SFT ASR decreases by 5.98% while HarmRLVR ASR increases by 0.80%; on AIR-Bench-64, the respective decreases are 14.68% and 6.70%.Source: §5.5.1 · p. 7, Figure 7 · p. 8, Table 10 · p. 8

5.5.2 SafeRLVR via Reward Reversal; 5.5.3 Implications for Future Defenses

  • SafeRLVR aligns unaligned DeepSeek-R1 but barely reduces harmfulness in attacked Llama3 while causing catastrophic utility collapse; the authors identify defense directions whose feasibility and effectiveness remain open questions.The same 64 prompts rapidly corrupt safety alignment but fail to repair already compromised models through reward reversal.Source: §5.5.2 · p. 8, Table 11 · p. 8, §5.5.3 · p. 8

6 Conclusion

The paper systematically investigates RLVR safety risks and finds that HarmRLVR can reverse safety alignment across multiple models with 64 harmful prompts without responses. It also reports that existing defenses generally fail and SafeRLVR cannot repair compromised models.

6 Conclusion

  • With 64 harmful prompts without responses, HarmRLVR reverses safety alignment across multiple models while preserving general capabilities.The study also reports high harmfulness and high attack success rate, without specifying their values in this passage.Source: §6 · p. 8
  • Existing defenses generally fail under HarmRLVR, and SafeRLVR cannot repair models already compromised by the attack.Source: §6 · p. 8
  • The authors identify reliance on static data while neglecting exploration mechanisms as a fundamental limitation of current defenses.They suggest future defenses focus on constraining reward specifications and exploration dynamics.Source: §6 · p. 8

Limitations

The experiments cover open-source LLMs, primarily at 7B and 8B scales, and do not evaluate vision-language or diffusion models. Applicability to closed-source and larger models remains unverified.

Limitations

  • Experiments use only open-source LLMs and primarily focus on 7B and 8B models because of computational constraints.Source: Limitations · p. 9
  • The findings’ applicability to closed-source models such as GPT and Gemini, and to larger-scale models, remains to be verified.Source: Limitations · p. 9
  • The study does not evaluate vision-language or diffusion models, whose HarmRLVR risks the authors say merit investigation.The authors cite prior work showing minimal harmful fine-tuning data can compromise VLM safety.Source: Limitations · p. 9

Ethical Considerations

The paper situates RLVR within alignment methods and describes a harmfulness-scoring framework based on Meta’s usage policy. For training, it uses a judge model and process-outcome reward shaping.

Ethical Considerations

  • The authors frame the work as safety research, reporting controlled experiments, restricted access, no public release of trained harmful models, and prompts from public safety benchmarks.Source: Ethical Considerations · p. 9

A Safety Alignment

  • RLHF and DPO optimize policies using human-preference data or directly, while RLVR uses objective, verifiable scoring; its safety-alignment effectiveness remains understudied.The section also notes that fine-tuning and jailbreaks can bypass aligned models’ guardrails.Source: A Safety Alignment · p. 12

B Harmfulness Scoring Framework

B.1 Foundational Components

  • Meta’s Disallowed Usage Policy defines three prohibited-content categories, while a standardized template guides the judge to reference the policy, apply a rubric, and analyze step by step.Source: §B.1 · p. 12

B.2 Training: Reward Modeling and Computation

  • During GRPO training, Qwen3-8B judges responses with fixed decoding settings and maps scores to five discrete reward values; Qwen3 also receives process-outcome reward shaping.For Qwen3, responses are divided into reasoning and final-answer components when combining rewards.Source: §B.2 · p. 12

B.3 Evaluation: Harmfulness Scoring

The section specifies separate harmfulness evaluation, benchmark choices, attack baselines, and training configurations. It also describes SFT tuning and reports that extended benchmarks supported the evaluation framework’s consistency.

B.3 Evaluation: Harmfulness Scoring

C.1 Harmful Attack Dataset

  • The attack set comprises 64 representative harmful prompts selected from AIR-Bench 2024’s 16 categories and 45 subcategories.Source: §C.1 · p. 13

C.2 Harmful Evaluation Datasets

  • Harmful evaluation uses AdvBench instruction queries and 300 HEx-PHI samples, excluding the Child Sexual Abuse Material category.HEx-PHI contains 330 instructions across 11 prohibited categories.Source: §C.2 · p. 13

C.3 General Capability Evaluation Datasets

  • Capability evaluation spans classification, grade-school math, instruction following, and multi-turn dialogue and broader task performance.The named benchmarks are SST-2, AGNEWS, GSM8K, AlpacaEval, MT-Bench, and Vicuna-Bench.Source: §C.3 · pp. 13–14

C.4 Defense Evaluation Datasets

  • Defense evaluation uses BeaverTails, the mixed Mix-1000 dataset, and AIR-Bench-64 to assess defense robustness and methods including T-Vaccine and SafeLoRA.Source: §C.4 · p. 14

D Adversarial Attack Baselines

  • The adversarial baselines include white-box GCG and black-box PAIR, ReNeLLM, and CodeAttack.The section describes their respective suffix-optimization, attacker-model, prompt-rewriting/scenario-nesting, and code-template approaches.Source: D Adversarial Attack Baselines · pp. 14–15

E Detailed Training Configurations

E.1 HarmRLVR (GRPO) Hyperparameters

  • HarmRLVR uses a 2048-token maximum for LRMs versus 1024 for LLMs; Qwen3 uses 16 rollouts versus four for other models, with process reward only for Qwen3.Source: §E.1 · p. 15, Table 14 · p. 15

F.1 Consistency Across Harmfulness Benchmarks

The expanded harmfulness evaluation preserved model rankings across four benchmarks, with only a 2.51% maximum ASR difference from the primary two-benchmark average. On the harder MATH-500 benchmark, HarmRLVR remained comparable to the base model while harmful SFT degraded reasoning.

F.1 Consistency Across Harmfulness Benchmarks

  • The maximum ASR difference between the primary two-benchmark and expanded four-benchmark averages is 2.51%, and model rankings remain consistent across all four benchmarks.The primary evaluation covers 820 HEx-PHI and AdvBench samples, which the authors consider sufficient for safety assessment.Source: §F.1 · p. 16

F.2 Utility Evaluation on Challenging Tasks

  • On MATH-500, HarmRLVR maintains utility comparable to the base model, while harmful SFT significantly degrades reasoning-task performance.MATH-500 tests high-level mathematical reasoning and is described as more difficult than GSM8K.Source: §F.2 · p. 16

G Human Evaluation for Automatic Scoring Validation

The authors checked GPT-4o-mini harmfulness ratings against human annotations on a stratified sample of 50 outputs. Agreement was strong across standard metrics, with most predictions within one point of human judgments.

G Human Evaluation for Automatic Scoring Validation

  • Cohen’s κ = 0.825, Pearson’s r = 0.936, and ±1 tolerance accuracy = 96% for GPT-4o-mini versus human harmfulness ratings.The confusion matrix shows near-perfect agreement at scores 1 and 5, with expected ambiguity at scores 2–4; no disagreements exceeded 2 points.Source: G Human Evaluation for Automatic… · p. 16, Table 20 · p. 17
  • The human evaluation used 50 randomly sampled outputs—25 each from AdvBench and HEx-PHI—with five samples per score level from each dataset.Raters used the same 1–5 rubric as the automatic evaluation.Source: G Human Evaluation for Automatic… · p. 16

H.1 Learning Rate

The appendix reports ablations and defense evaluations, probes safety landscapes, and compares harmful SFT with HarmRLVR outputs. Across these analyses, training settings and model behavior distinguish attack effectiveness, utility preservation, and defenses.

H.1 Learning Rate

  • ASR 96.86% and Acc 80.93% at 100 epochs with a 1e-6 learning rate; the authors select it for balanced safety and generalization performance.A 1e-5 rate reaches ASR 97.47% at 20 epochs but collapses to zero ASR and Acc by training’s end.Source: §H.1 · p. 17

H.2 Batch Size

  • As batch size decreases from 64 to 8, ASR rises, but 100 epochs expand from 100 to 800 steps and training time from about 1.5 to 3–5 hours.Smaller batches also slightly reduce utility; the authors choose 64 as the default configuration.Source: §H.2 · p. 17, Figure 9 · p. 18

H.3 Entropy Coefficient

  • Entropy_coeff = 0.001 significantly improves harmfulness over 0 while general capabilities remain comparable.The authors attribute the improvement to increased exploration and response diversity.Source: §H.3 · p. 18, Table 22 · p. 18

H.4 Full fine-tuning vs. LoRA

  • Full fine-tuning outperforms LoRA under both SFT and RLVR, while RLVR outperforms SFT on harmfulness and utility even with LoRA.LoRA requires more epochs to reach comparable performance, diminishing its efficiency advantage.Source: §H.4 · p. 18, Table 23 · p. 18

I.1 T-Vaccine

  • Selective perturbation of safety-critical layers improves defense efficiency and reduces memory overhead by avoiding gradient computation and activation storage for frozen layers.T-Vaccine identifies those layers using gradient norms.Source: §I.1 · p. 18

I.2 SafeLoRA

  • The SafeLoRA adaptation projects attacked-model weight residuals into safety-consistent subspaces; its approximate projection is over 250× faster with negligible alignment and utility loss.The alignment matrix is constructed using Llama3-8B-Instruct and Llama3-8B.Source: §I.2 · p. 19

J.1 Background: Safety Basin Phenomenon; J.3 Visualization Techniques

  • The base model has a deep safety basin under weight perturbations, whereas HarmRLVR’s basin is nearly flattened into a high-ASR plateau.The paper describes a safety basin as a region where small random weight perturbations do not compromise safety; the 2D analysis is consistent with the 1D result.Source: §J.3 · pp. 19–20, §J.1 · p. 19, Figure 10 · p. 20

K Case Studies

  • In case studies, harmful SFT outputs retain disclaimers or ethical hedging, while HarmRLVR answers directly and uses CoT to complete harmful requests more coherently.The examples compare Qwen2.5 and DeepSeek-R1 outputs under different training regimes.Source: K Case Studies · p. 20, Figure 11 · p. 21, Figure 12 · p. 22
Loading 2510.15499v2…