PaperlayerView exact revision on arXiv

Source-linked AI summary

ProGuard: Towards Proactive Multimodal Safeguard

Shaohan Yu, Lijun Li, Chenyang Si, Lu Sheng, Jing Shao

arXiv:2512.23573v1cs.CV

TL;DR

Existing guards struggle to identify risks outside fixed taxonomies, amid evolving multimodal safety challenges. ProGuard combines balanced multimodal data, pure reinforcement learning, and OOD category inference; its OOD results include higher F1 scores than closed-source models across benchmarks and category-name inference close to closed-source performance.

  • Problem

    Existing guards rely on predefined taxonomies and struggle with emerging risks, while skewed modality coverage and binary-only focus limit multimodal moderation and adaptation.

    Source: §1 · p. 2

  • Method

    ProGuard trains a multimodal guard purely with reinforcement learning on balanced safety data and adds OOD category inference with a synonym-bank similarity reward.

    Source: §1 · p. 2

  • Results

    ProGuard-7B achieves higher F1 scores across all benchmarks than closed-source models, while ProGuard-7B and ProGuard-3B achieve OOD category-name inference performance close to Gemini 2.5-Flash.

    Source: §5.4 · pp. 7–8

  • Takeaways & Limitations

    OOD category inference offers a controlled, taxonomy-agnostic setting for proactive reasoning about unseen safety risks.

    Source: §1 · p. 2

    ProGuard's relatively small parameter scale leaves it behind Gemini 2.5-Flash and GPT-4o-mini in absolute performance.

    Source: §5.3 · pp. 6–7

Abstract

from arXiv · show

The rapid evolution of generative models has led to a continuous emergence of multimodal safety risks, exposing the limitations of existing defense methods. To address these challenges, we propose ProGuard, a vision-language proactive guard that identifies and describes out-of-distribution (OOD) safety risks without the need for model adjustments required by traditional reactive approaches. We first construct a modality-balanced dataset of 87K samples, each annotated with both binary safety labels and risk categories under a hierarchical multimodal safety taxonomy, effectively mitigating modality bias and ensuring consistent moderation across text, image, and text-image inputs. Based on this dataset, we train our vision-language base model purely through reinforcement learning (RL) to achieve efficient and concise reasoning. To approximate proactive safety scenarios in a controlled setting, we further introduce an OOD safety category inference task and augment the RL objective with a synonym-bank-based similarity reward that encourages the model to generate concise descriptions for unseen unsafe categories. Experimental results show that ProGuard achieves performance comparable to closed-source large models on binary safety classification, substantially outperforms existing open-source guard models on unsafe content categorization. Most notably, ProGuard delivers a strong proactive moderation ability, improving OOD risk detection by 52.6% and OOD risk description by 64.8%.

1. Introduction

As generative models evolve, multimodal safety risks outpace existing defenses, while reactive guards struggle with risks outside fixed taxonomies. ProGuard combines balanced multimodal data and pure reinforcement learning with OOD category inference to support proactive moderation.

1. Introduction

  • Existing guards rely on predefined taxonomies and struggle with emerging risks, while modality-skewed data and binary-only moderation limit multimodal categorization and adaptation.Text-risk performance often exceeds image-risk performance in existing models.Source: §1 · p. 2
  • ProGuard introduces OOD safety category inference, using intentionally incomplete taxonomies and a synonym-bank similarity reward to elicit plausible names for unseen risks.The model is trained purely with reinforcement learning to encourage autonomous, efficient reasoning.Source: §1 · p. 2
  • The work constructs a hierarchical multimodal safety taxonomy and a balanced dataset of 87K samples annotated with binary labels and fine-grained harmful categories.The balanced coverage is designed to mitigate modality bias.Source: §1 · p. 2
  • ProGuard jointly addresses binary safety classification, unsafe content categorization, and OOD category inference in a taxonomy-agnostic setting.Online multi-task reinforcement learning is used to generalize safety reasoning across these tasks.Source: §1 · p. 2

2. Related Work

Prior guard work spans proprietary filters, open-source models, vision-language moderation, and reasoning-based classification, but leaves multimodal balance and broader generalization as open concerns.

2. Related Work

  • Open-source guard models followed proprietary moderation filters, and vision-language guards now moderate textual and visual inputs, though their taxonomies and data remain text-skewed.Examples include LlamaGuard 3 Vision [5] and GuardReasoner-VL [30].Source: §2 · p. 2
  • GuardReasoner [29] and DynaGuard [14] use chain-of-thought reasoning for binary safety classification, leaving broader generalization and test-time scaling underexplored.Source: §2 · p. 2
  • Industrial APIs such as OpenAI Moderation [32], Perspective, and Azure AI Content Safety provided effective but proprietary filters before open-source guards expanded transparency and reproducibility.Source: §2 · p. 2

3. Preliminary

The preliminary formulation describes a guard that judges a generative model's query and optional response under a predefined taxonomy, outputting safety labels and, when needed, a category prediction.

3. Preliminary

  • The guard G evaluates a target model F's query and optional response under taxonomy T; the response is omitted when unavailable.The formulation covers text, visual, or text-image-capable target models.Source: §3 · p. 2
  • Equation (1) returns query and response safety labels when a response is provided, and query label plus category prediction otherwise.The category output is included when either input is judged unsafe.Source: §3 · p. 2
  • For unsafe inputs, the category prediction records whether a taxonomy category fits and supplies either its index or a self-inferred category name.The category combines an in/out-of-taxonomy judgment with a semantic label.Source: §3 · p. 3

4. Methodology

ProGuard’s methodology combines a modality-aware safety taxonomy and balanced annotated data with pure reinforcement-learning training to moderate known risks and infer categories for unseen risks.

4.1. Multimodal Safety Taxonomy

  • 11 high-level categories and 28 subcategories form a hierarchical taxonomy designed to represent modality-dependent safety risks.The taxonomy draws on existing text, text-image, and image safety datasets.Source: §4.1 · p. 3

4.2. Balanced Dataset Construction

  • 87,024 samples span text, text-image, and image modalities, with majority-vote labels showing around 0.7 Fleiss-Kappa agreement and over 90% acceptance.A human validation sample of 300 instances showed over 90% agreement with majority-vote labels; the data were balanced by modality and safety.Source: §4.2 · p. 4

4.3.1. Data Augmentation

  • Three taxonomy augmentations vary hierarchy depth, shuffle category-index mappings, and remove categories to train recognition of out-of-taxonomy risks.When a ground-truth category is removed, the model must infer a plausible risk-category name or description.Source: §4.3.1 · p. 4

4.3. Online Reinforcement Learning Post-Training

  • ProGuard uses pure GRPO training, bypassing supervised fine-tuning so the model can develop reasoning traces through reward-based trial and error.The training builds on Qwen2.5-VL-7B [1] and an R1-Zero-style approach.Source: §4.3.2 · p. 4, §4.3 · p. 4

4.3.2. Training Process

  • GRPO optimizes the moderation policy using group rewards and a normalized advantage, without teacher-provided reasoning traces.The advantage is calculated relative to the mean and standard deviation of group rewards.Source: §4.3.2 · pp. 4–5

4.3.3. Correctness Reward Design

  • The correctness reward gates scores on required output formatting, rewards correct safety labels, and scores category judgments with additional credit for matching an in-taxonomy index.The category reward is zero for safe-safe query-response pairs or incorrect in/out-of-taxonomy judgments; correct taxonomy judgments receive a base reward, with a possible index-match bonus.Source: §4.3.3 · p. 5

4.3.4. OOD Inference Reward Design

  • A synonym-bank similarity reward uses maximum and mean semantic similarity to encourage category descriptions that match the underlying safety concept rather than one phrasing.The lower mean-similarity threshold rewards coarse semantic guesses before refinement toward the precise category; the authors report consistency with human perceptual judgments.Source: §4.3.4 · p. 5

5. Experiment

Experiments report strong cross-modal binary classification, advantages over LlamaGuard in categorization and OOD detection, and human-reward alignment; pure RL uses fewer reasoning tokens than the compared training strategies.

5.1. Experiment Setup

  • Evaluations covered prompt- and response-level binary safety, three-modality categorization, and OOD inference against open- and closed-source guards.OOD evaluation used benchmark-taxonomy synonyms that were unseen during training.Source: §5.1 · p. 5, §5.4 · p. 7

5.2. Binary Safety Classification Results

  • ProGuard-7B is state of the art among open-source models for prompt classification across modalities, comparable to GPT-4o-mini, and leads overall on text-image response classification.It ranks second among VLM-based models for response classification, though it performs slightly below LLM-based guards on text responses.Source: §5.2 · pp. 5–6, Table 2 · p. 6

5.3. Unsafe Content Categorization Results

  • ProGuard outperforms all LlamaGuard models across its taxonomy and external taxonomies, while LlamaGuard drops on image and text-image inputs.Despite these gains, ProGuard trails Gemini 2.5-Flash and GPT-4o-mini in absolute performance.Source: §5.3 · pp. 6–7, Table 3 · p. 7

5.4. OOD Safety Category Inference Results

  • ProGuard-7B achieves higher OOD-detection F1 across benchmarks, while both ProGuard sizes closely match Gemini 2.5-Flash on inferred category names.The task remains challenging: most models struggle to distinguish out-of-taxonomy cases robustly.Source: Table 4 · p. 7, §5.4 · pp. 7–8, Table 5 · p. 7

5.5. Effect of Training Strategy on Reasoning-Based Guard Performance

  • 52.07 thinking tokens on average for pure-RL training, versus 66.25 for reasoning-capable initialization and 114.05 after SFT cold start; no method dominates task metrics.The results describe the pure-RL model as more concise without sacrificing performance across tasks.Source: Figure 3 · p. 8, §5.5 · p. 8

5.6. Human Alignment Results

  • Human judgment agrees with reward rankings by more than 80% across categories, supporting the reward’s semantic validity.Experts compared inferred category names for each ground-truth OOD category.Source: Figure 4 · p. 8, §5.6 · p. 8

6. Conclusion

ProGuard is a reinforced-reasoning multimodal safety guard designed to detect and describe risks beyond fixed taxonomies.

6. Conclusion

  • ProGuard detects and describes unseen multimodal safety risks beyond fixed taxonomies.The paper presents it as a preliminary step toward proactive safety systems, with potential value in taxonomy expansion, open-ended prompt risk labeling, and risk discovery support.Source: §6 · p. 8

A. Multimodal Safety Taxonomy

The taxonomy organizes multimodal safety risks into named categories and subcategories, with descriptions and synonyms for many entries.

A. Multimodal Safety Taxonomy

  • C1–C3 cover copyright violations, misinformation and private information, and malicious deception or suggestion.Their listed subcategories include unauthorized use of protected works, false information or data leakage, and fraud, manipulation, or sexual suggestion.Source: A. Multimodal Safety Taxonomy · p. 9
  • C4–C6 address cybersecurity technology, living-environment hazards, and financial crime.The listed subcategories include system intrusion and malicious software, natural and urban environmental risks, and financial fraud or illegal trade.Source: A. Multimodal Safety Taxonomy · pp. 9–10
  • C7–C8 cover hate and terror, and dangerous goods.Examples include hate speech, extremism, weapons, hazardous biological, chemical, or radioactive materials, and illegal drugs.Source: A. Multimodal Safety Taxonomy · p. 10
  • C9–C10 cover real-world harm and blood-related disturbing content.Their subcategories include self-harm, animal abuse, violent or property crime, medical and accident scenes, and killing scenes.Source: A. Multimodal Safety Taxonomy · p. 10
  • C11 covers sexual exposure, including inappropriate exposure of minors and exposure of sexual organs.The taxonomy describes the category as explicit sexual display that may violate privacy or legal boundaries.Source: A. Multimodal Safety Taxonomy · pp. 10–11

B. Implementation Details of Majority Voting

The annotation procedure uses multiple models to judge samples and combines their safety judgments by majority vote.

B. Implementation Details of Majority Voting

  • Table 6 summarizes the models used for majority voting.The supplied table caption identifies its scope but does not specify the model names.Source: Table 6 · p. 11
  • Three VLMs and three LLMs independently provide safety judgments for samples across text, text-image, and image inputs.The final label is determined by majority voting among model outputs.Source: B. Implementation Details of Majority… · p. 11
  • The annotation prompt asks models to classify inputs into taxonomy categories and subcategories, or return None when no category applies.It instructs models to follow the taxonomy's naming conventions and return one JSON object.Source: B. Implementation Details of Majority… · p. 11

C. GRPO Hyperparameter Settings

The paper reports the GRPO hyperparameter settings and identifies the backbone and reinforcement fine-tuning approach used for training.

C. GRPO Hyperparameter Settings

D. Construction of the Synonym Bank for OOD Taxonomy

The section presents four safety taxonomies—Aegis2.0, BeaverTails-V, LlavaGuard, and OpenAIModeration—with categories and synonym variants. It also includes figures labeled as modality-imbalance training dynamics and performance impact.

D.1. Aegis2.0

  • Aegis2.0 lists 21 safety categories spanning hate, sexual content, self-harm, violence, privacy, and manipulation, with synonym sets for each.The list runs from Hate/Identity Hate through Manipulation.Source: §D.1 · pp. 11–12

D.2. BeaverTails-V

  • BeaverTails-V organizes 20 categories, including environmental damage, privacy invasion, fraud, false information, and sexual crimes, with definitions and synonym lists.Its categories run from terrorism and cybersecurity threats through sexual crimes.Source: §D.2 · pp. 12–13

D.3. LlavaGuard

  • LlavaGuard lists eight categories covering hate and harassment, violence, sexual content, nudity, criminal planning, weapons or substance abuse, and self-harm.The list also includes animal cruelty.Source: §D.3 · p. 13

D.4. OpenAIModeration

  • OpenAIModeration lists eight categories, including sexual, hate, violence, self-harm, and graphic-violence risks, with synonym variants.The taxonomy includes combined categories such as Hate/Threatening and Violence/Graphic.Source: §D.4 · pp. 13–14
  • Figures 5 and 6 are labeled as showing training dynamics and performance impact under modality imbalance.Source: Figure 5 · p. 14, Figure 6 · p. 14

E. Experiments on Modality Imbalance

The experiments compare balanced and imbalanced modality sampling using 60K-sample subsets and a shared training schedule. Balanced sampling yields more stable reward convergence and stronger cross-modality performance than imbalanced sampling.

E. Experiments on Modality Imbalance

  • The study compares four 60K-sample subsets: three use a 25K:25K:10K modality distribution and one uses a balanced 1:1:1 ratio.All models are evaluated at the final checkpoint of the same training schedule.Source: E. Experiments on Modality Imbalance · p. 14
  • The balanced model achieves top or near-top results across tasks, with more stable reward convergence and uniformly strong performance across modalities.The reported findings identify modality balance as important for consistent multimodal safety moderation.Source: E. Experiments on Modality Imbalance · p. 14
  • With fewer samples, reward fluctuations increase; scarce image data nearly eliminates image safety classification performance, while text and text-image reductions impair their respective category-classification performance.The text-image degradation is described as milder but still observable.Source: E. Experiments on Modality Imbalance · p. 14

F. Implementation Details of Reasoning Trace Generation

The implementation prompt asks for concise, natural reasoning traces explaining a supplied safety assessment. The examples show ProGuard’s formatted assessments for text-only, text-image, and image-only inputs.

F. Implementation Details of Reasoning Trace Generation

  • The reasoning-trace prompt supplies the task and correct answer, then requests a concise, natural explanation of why the safety assessment is correct.It specifies no numbered list and wraps the response in think tags.Source: F. Implementation Details of Reasoning… · p. 14

G. Example of ProGuard Response

  • In the text-image animal-confinement example, ProGuard labels the request unsafe, the response safe, and the category Animal Cruelty.Source: G. Example of ProGuard Response · p. 15
  • In the text-only serial-killer example, ProGuard labels both the request and response unsafe and assigns category C2.The example input asks for a serial killer’s modus operandi, and the agent response gives violent details.Source: G. Example of ProGuard Response · p. 16
  • For the image-only knife example, ProGuard marks the image unsafe under C1S1 and explains that the knife could be misused, causing accidental injury.The prompt requests image safety assessment under a hierarchical taxonomy.Source: G. Example of ProGuard Response · p. 17

Related papers

Loading 2512.23573v1…