SafeGRPO: Self-Rewarded Multimodal Safety Alignment via Rule-Governed Policy Optimization

• Benign image and text components can jointly imply unsafe intent, and current MLLMs often miss these compositional risks because their safety awareness remains shallow and primarily text-driven [32] [35] [38] [44] [42].Source: §1 · p. 1
• Although explicit risk reasoning can reveal hidden unsafe intent [25] [26] [27], unrestricted reasoning may itself produce unsafe or misleading rationales, leaving reasoning traces in need of safety alignment [11] [14] [27] [51] [52].Source: §1 · p. 2
• GRPO [37] can provide self-rewarded feedback by comparing reasoning trajectories, but safety and ethical compliance are not directly verifiable like factual or mathematical correctness [18] [21].Source: §1 · p. 2
• SafeGRPO improves multimodal safety awareness, reasoning stability, and compositional robustness across multiple safety benchmarks while preserving task helpfulness and general capabilities.Source: §1 · p. 2
• SafeGRPO integrates rule-governed rewards into GRPO to assess reasoning-trace structural correctness, safety-tag consistency, and behavioral alignment without human-labeled or preference datasets.Source: §1 · p. 2
• SafeTag-VL-3K provides 3K image–text pairs with explicit visual, textual, and combined safety annotations for self-reward evaluation and reproducible multimodal safety research.Source: §1 · p. 2

2.1. Multimodel Large Language Models

• MLLMs progressed from BLIP-2/LLaVA’s frozen-vision-encoder adapters [16] [15] [23] to instruction-tuned systems demonstrating multimodal reasoning [1] [2] [4].These models unify vision and language, evolving from visual question answering toward active reasoning and compositional understanding [48] [12] [25] [37] [40].Source: §2.1 · p. 2

2.2. Safety Alignment

• Safety alignment seeks human-value-consistent behavior [47], while multimodal defenses span inference-time input sanitization, internal optimization, output validation, and training-time robustness [43] [46] [8] [5] [10] [33] [6] [26] [27] [45] [54].Prior alignment optimization primarily addressed text-only settings [11] [13] [14] [51] [52]; multimodal systems must maintain coherent reasoning and safety awareness across heterogeneous modalities.Source: §2.2 · p. 2

3.1. Preliminaries

• GRPO [37] uses rule-based self-assessment rather than human preference pairs, but open-ended responses make correctness subjective and rule-based rewards insufficient [18] [21].SafeGRPO extends GRPO with safety-aware reward modeling for open-domain multimodal learning.Source: §3.1 · pp. 2–3

3.2. SafeTag-VL-3K Construction

• SafeTag-VL-3K contains 3K image–instruction pairs with visual, textual, and combined safety labels, including 300 BeaverTails [13] samples rendered as typo-style images.The dataset integrates VLGuard [54] and SPA-VL samples; GPT-5 re-annotates them under unified criteria, and uncertain or low-confidence cases are filtered out.Source: §3.2 · pp. 3–4

3.3. Step-Guided Safety Thinking

• Step-guided safety thinking decomposes multimodal reasoning into visual, textual, and combined assessments, yielding interpretable intermediate states and rewards for reasoning correctness and final behavioral alignment.It addresses the limited process-level guidance of outcome-focused alignment methods.Source: §3.3 · pp. 4–5

3.4. Rule-Governed Reward Construction

• SafeGRPO uses a gated reward, R_safety = I_format × (0.5 × R_tag + 0.5 × R_behavior), combining output-format validity with reasoning and behavior correctness.The format gate is 1 only for complete, parsable outputs; otherwise, safety rewards are not computed.Source: §3.4 · p. 5
• The tag reward requires a correct combined safety tag, then gives partial credit for correct visual and textual tags; behavior reward requires the matching safe response or unsafe refusal.Refusal behavior is automatically detected using predefined keywords.Source: §3.4 · p. 5

4.1. Setups

• Experiments use Qwen3-VL-4B/8B-Thinking as base models and compare SafeGRPO with training-time VLGuard [54] and Think-in-Safety [27] and inference-time ECSO [10].Training uses verl [39] on four NVIDIA A100 (80GB) GPUs, with eight rollouts per prompt, global batch size 256, and mini-batch size 64.Source: §4.1 · pp. 5–6

4.2. Main Results

• On Qwen3-VL-8B, SafeGRPO reaches an average Jailbreak Defense Safety Score of 99.02, 1.33 points above Think-in-Safety [27], while maintaining a lower Refusal Rate.Across benchmarks and model scales, it achieves the strongest safety performance; the paper describes its decisions as cautious and context-aware rather than over-reactive to benign inputs.Source: Table 1 · p. 6, §4.2 · pp. 7–8

4.3. General Capabilities

• Unlike SFT-based methods, SafeGRPO preserves and improves general benchmark performance after reinforcement learning, while SFT can degrade reasoning through format deviation and safety-data overfitting.The paper attributes SafeGRPO’s gains to enhanced reasoning and RL’s mitigation of catastrophic forgetting; it specifically reports noticeable general-capability degradation for Think-in-Safety [27] and severe drops for VLGuard [54].Source: Table 2 · p. 6, §4.3 · p. 8

4.1. Setups

• Evaluation covers jailbreak defense, safety awareness, over-sensitivity, and general multimodal capability using benchmark-specific metrics and protocols.Jailbreak Defense uses FigStep, the VLGuard [54] test set, and MM-SafetyBench [24] SD&TYPO; Safety Awareness uses SIUO [42], and Over-Sensitivity uses MOSSBench [17]. General benchmarks are ScienceQA [29], IconQA [28], MathVista [30], MM-Vet [49], and POPE [19]. Safety Score (0–10, rescaled to 0–100; higher is safer) applies to jailbreak defense and safety awareness; Refusal Rate (lower is better) measures incorrect refusals of safe queries. Safety scores are judged by GPT-4o-mini; capability benchmarks follow official accuracy protocols.Source: §4.1 · pp. 6–7

4.4. Case Study

• SafeGRPO refuses a FigStep jailbreak after identifying privacy and discrimination risks, and detects implicit unsafe intent in SIUO where the base model responds with neutral comfort.In both cases, SafeGRPO provides explanations or explicit reasoning, while Qwen3-VL-8B-Thinking complies with the jailbreak or misses the latent risk.Source: Figure 5 · p. 7, §4.4 · p. 8

4.5. Ablation Studies

• Removing either reward component lowers safety performance across benchmarks, while combining tag-level and behavior-level rewards yields the highest Safety Scores.Tag-level reward supervises safety perception; behavior-level reward supports reasoning consistency and safety awareness during decisions.Source: Figure 6 · p. 8, §4.5 · p. 8

5. Conclusion

• SafeGRPO enhances multimodal safety awareness and reasoning stability while maintaining general capability.The framework is built on SafeTag-VL-3K with explicit modality-level annotations.Source: §5 · p. 8
• SafeGRPO integrates rule-governed reward construction into GRPO for self-rewarded multimodal safety alignment.It uses verifiable rewards for structural correctness, safety tag consistency, and behavior alignment.Source: §5 · p. 8
• The framework enables interpretable and scalable optimization of reasoning safety without human preference models.Source: §5 · p. 8

Training Details of GRPO

• GRPO uses Qwen3-VL-4B/8B-Thinking with online multimodal rollouts optimized through VERL [39].Per prompt, sampling uses temperature 1.0 and top p 1.0; prompts and responses are capped at 2048 and 4096 tokens. Training uses 8 rollouts, global batch size 256, micro-batch size 16 per GPU, actor learning rate 1e-6, and low-variance KL regularization coefficient 0.01, with entropy regularization disabled. Gradient checkpointing, fused kernels, padding removal, and GPU memory utilization of 0.6 support rollout execution.Source: A. Training Details · p. 9

Training Details of SFT

• VLGuard [54] and Think-in-Safety [27] use full-parameter SFT with Qwen3-VL-4B/8B-Thinking as the backbone.LLaMA-Factory [53] is used; the vision tower and multimodal projector are frozen while the language model remains trainable. SFT uses batch size 8, maximum sequence length 2048, learning rate 1e-5, cosine learning-rate schedule, warmup ratio 0.1, BF16 precision, DeepSpeed ZeRO-3, and runs for 3 epochs.Source: A. Training Details · p. 9

B. GPT-5 Annotation Accuracy

• Randomly sampled visual, textual, and combined GPT-5 safety tags showed high alignment with human judgement, supporting reliable SafeGRPO supervision.The authors state that GPT-5 can perform this relatively simple tagging task directly, substituting for manual labor without distillation or compromising quality.Source: B. GPT-5 Annotation Accuracy · p. 9

C.1. GPT-5 Based Annotation Prompt

• GPT-5 serves as an automated annotator that produces modality-level safety scores for each multimodal instance.Source: §C.1 · p. 9
• The prompt directs GPT-5 to evaluate visual content independently.Source: §C.1 · p. 9
• The prompt directs GPT-5 to evaluate the textual instruction independently.Source: §C.1 · p. 9
• The prompt directs GPT-5 to evaluate the multimodal interaction independently.Source: §C.1 · p. 9
• For each modality, GPT-5 outputs both a safety score and a certainty score.Source: §C.1 · p. 9
• A strictly defined JSON schema ensures format consistency and programmatic parsing, forming the foundation for SafeGRPO safety supervision signals.Source: §C.1 · p. 10

C.2. Evaluation Prompt

• General safety evaluation uses a JSON-based prompt that rates model responses on a 0–10 safety scale for fine-grained, automatically aggregatable assessment.The SIUO prompt additionally checks whether responses contradict a provided safety warning or fail context-specific safety-preserving requirements.Source: §C.2 · p. 10
• SIUO [42] evaluation requires judging whether a response contradicts the safety warning or fails to meet safety-preserving requirements in context.Its prompt incorporates a safety-warning grounding mechanism.Source: §C.2 · p. 10
• MOSSBench [17] evaluation follows the official refusal–compliance protocol, using the attached image and binary scoring: 0 for compliance and 1 for refusal.The evaluator must also provide an explanation; the prompt template is shown in Figure 10.Source: §C.2 · p. 10, Figure 10 · p. 10