PaperlayerView exact revision on arXiv

Source-linked AI summary

SingGuard: A Policy-Adaptive Multimodal LLM Guardrail with Dynamic Reasoning

SingGuard Team

arXiv:2606.22873v3cs.CVcs.CL

TL;DR

Existing guardrails often use fixed taxonomies or target narrow interaction settings, while policies can vary across products, regions, and deployment stages. SingGuard checks content against active runtime rules and achieves state-of-the-art average F1 across six benchmark families.

  • Problem

    Existing guardrails often rely on fixed taxonomies or narrow interaction settings, while applicable policies can vary across products, regions, and deployment stages.

    Source: §1 · p. 2

  • Method

    SingGuard accepts natural-language policies at runtime, checks content rule by rule, and supports fast, hybrid, and slow inference regimes.

    Source: §1 · p. 2, §2 · p. 4

  • Results

    Across six benchmark families (35 datasets), SingGuard achieves state-of-the-art average F1; dynamic-policy accuracy is 0.7415 for SingGuard-slow versus 0.6465 for Qwen3-VL-8B.

    Source: §6 · p. 20

  • Takeaways & Limitations

    Deployed guardrails need to judge whether risks violate the active policy, not merely identify risk, and should refrain from over-blocking.

    Source: §6 · p. 20

    SingGuard depends on clear, complete, nonconflicting runtime policies, and its benchmark cannot exhaust real-world policies or emerging abuse patterns.

    Source: §7 · p. 21

Abstract

from arXiv · show

Vision-language models (VLMs) are increasingly deployed in consumer, medical, financial, and enterprise applications. This broad deployment expands the safety surface: risks can arise from multimodal question answering, assistant responses, and cross-modal composition, while moderation policies may vary across products, regions, and deployment stages. Most existing guardrails either rely on fixed taxonomies or target only a narrow set of interaction settings, which limits their adaptability when safety rules change at deployment time. We present \textbf{SingGuard}, a policy-adaptive multimodal guardrail model family for safety assessment in multimodal conversations. SingGuard treats the active policy as a runtime input: given natural-language rules, it checks the target content against the active policy rule by rule and predicts both the safety label and the triggered rule. To balance efficiency and interpretability, SingGuard supports fast, hybrid, and slow inference regimes along a fast-to-slow reasoning spectrum, ranging from direct safety judgments to policy-grounded deliberation. We further optimize this behavior with fast--slow decoupled reinforcement learning. We also introduce \textbf{SingGuard-Bench}, a multimodal guardrail benchmark with 56{,}340 examples spanning 80+ fine-grained risk types across multimodal QA, adversarial attack, and dynamic-rule evaluation settings, including cross-modal joint-risk cases where each modality is harmless in isolation but their composition implies unsafe intent. Across six benchmark families (35 datasets), SingGuard achieves state-of-the-art average F1 in every family. Dynamic-rule evaluation further shows improved policy-following accuracy from 0.6465 to 0.7415 under runtime policy shifts. Our code is available at https://github.com/inclusionAI/Sing-Guard.

1 Introduction

SingGuard conditions multimodal safety judgments on runtime policies and adapts its reasoning depth to the moderation case. Across broad benchmark families, it reports strong average performance and improved accuracy under changing policies.

1 Introduction

  • Because risks span requests, responses, and cross-modal composition while policies vary by deployment, guardrails need runtime policy conditioning and multiple reasoning depths.The introduction motivates direct judgments for routine clear cases and slower deliberation for changing, implicit, counter-intuitive, or audit-sensitive cases.Source: §1 · p. 2
  • SingGuard offers fast direct judgments, hybrid routing to reasoning, and slow policy-grounded deliberation, with decoupled reinforcement learning designed to reduce fast-token anchoring.The fast token is excluded from the RL response update, while the final answer is optimized with binary safety and fine-grained category rewards.Source: §1 · pp. 2–3
  • SingGuard-Bench covers more than 80 risk types across multimodal inputs, adversarial prompts, cross-modal joint risks, and matching versus non-matching runtime policies.Its scope includes image, image–text, dynamic-rule, and attack-style samples; joint-risk cases are benign in each modality alone but unsafe in combination.Source: §1 · pp. 2–3
  • SingGuard achieves state-of-the-art average performance across six benchmark families and 35 datasets, while dynamic-policy accuracy rises from 0.6465 for Qwen3-VL-8B to 0.7415 for SingGuard-slow.The dynamic-rule result supports runtime conditioning under changing active policies rather than reliance only on training-time taxonomy priors.Source: §1 · p. 3
  • SingGuard matches content rule by rule against active runtime policies and predicts both whether a rule is violated and the triggered category.Policies may be concise category names or detailed natural-language rules with scope, exceptions, and domain-specific constraints, which teams can revise without retraining.Source: §1 · p. 3

2 Method

The method conditions multimodal moderation on the active runtime policy, combining policy-aligned data and training with direct-judgment and policy-grounded inference regimes. It also describes confidence-triggered policy-grounded continuation and on-policy distillation for compact-model reasoning about dynamic rules.

2.1 Task Formulation

  • SingGuard assesses queries, images, assistant responses, or their combinations against the active policy, returning a safety label, triggered rule, and optional policy-grounded trace.Its default hierarchical taxonomy has 8 primary dimensions, 27 secondary categories, and more than 80 fine-grained risk types, while runtime policies can use subsets, revised rules, or new rules.Source: §2.1 · p. 4, §2.2 · p. 4

2.3 Data Collection and Annotation

  • The annotation pipeline combines re-annotated open-source data, policy-grounded synthetic data, dynamic-rule examples, and CoT supervision into a corpus of roughly 2.5M textual and 600K multimodal samples.Open-source data are normalized to a unified risk taxonomy with held-out evaluation splits; query-side and response-side labels remain separate to support screening, response checking, and joint moderation.Source: §2.3 · p. 5, §2.3.1 · p. 5

2.3.2 Policy-Grounded Data Synthesis

  • Policy-grounded synthesis expands fine-grained risk seeds into text, multimodal, cross-modal, and multilingual examples, with verification to preserve intended labels and policy semantics.Cross-modal attacks split harmful intent across image and text so each appears benign alone, while their composition is unsafe; translations are re-verified under the same policy.Source: §2.3.2 · p. 6

2.3.3 Dynamic Rule Data Construction

  • Dynamic-rule examples diversify policy wording and scope, recompute labels for counterfactual rule changes, and add verified rules beyond the original taxonomy.Consistency checks ensure each active rule applies to its sample and that the recomputed label and final answer match the active policy.Source: §2.3.3 · pp. 6–7

2.3.4 Chain-of-Thought Reasoning Data Construction

  • More than 1M CoT examples teach rule-by-rule judgments with evidence, then pass structural and reasoning-answer consistency checks before joining training data.The resulting traces can be paired with simpler policy prompts, including summarized, subset, merged, or title-focused rules.Source: §2.3.4 · p. 7

2.4 Stage 1: Policy-Conditioned Cold-Start SFT

  • Cold-start SFT trains fast judgments and verified reasoning examples under one policy-conditioned output grammar, making slow traces auditable through evidence-linked rule checks.The reasoning checks active rules individually and aggregates their judgments into the final answer; the training mixture uses broad fast-data coverage alongside a smaller CoT subset.Source: §2.4 · pp. 7–8

2.5 Stage 2: Fast–Slow Decoupled DAPO

  • Fast–slow decoupled DAPO masks the initial fast token from policy-gradient updates while rewarding the complete answer, letting later reasoning revise unreliable judgments.Training emphasizes dynamic-rule cases and uses binary safety correctness plus exact active-rule correctness to score final answers.Source: §2.5 · pp. 8–9

2.6 Policy-Grounded Dynamic Inference

  • Inference offers fast direct judgments, slow policy-grounded review, and hybrid early exit: confident binary decisions stop decoding, while uncertain cases continue to reasoning and a reviewed answer.Slow mode is intended for ambiguous, audit-sensitive, or dynamically changing policies; fast mode supports low-latency moderation.Source: §2.6 · p. 9
  • When normalized confidence falls below threshold τ, SingGuard continues generation with a policy-grounded reasoning trace and reviewed answer.The threshold controls the accuracy–latency trade-off.Source: §2.6 · p. 9

2.7 Model Distillation via On-Policy GKD

  • After Stage 2, the 8B model provides stronger policy-grounded reasoning than the 2B model, especially on dynamic rules, exemptions, long-tail risks, and revised fast judgments.The remaining gap reflects behavior alignment as well as label coverage: the compact model must learn to interpret active rules and recover from uncertain generations.Source: §2.7 · p. 10
  • On-policy distillation trains the 2B student on its own responses using token-level targets from the frozen 8B teacher.A bidirectional objective combines teacher-to-student and student-to-teacher KL signals, exposing the teacher to student errors and helping the student repair weak reasoning paths on dynamic rules, edge cases, and long-tail categories.Source: §2.7 · p. 10
  • On-policy distillation improves the 2B model’s dynamic-policy following and policy-grounded reasoning while preserving its low-latency deployment footprint.The passage defers the quantitative effect to Section 4.Source: §2.7 · p. 10

3 SingGuard-Bench

SingGuard-Bench addresses gaps in multimodal moderation tests with a policy-conditioned benchmark spanning fine-grained risks, benign-sensitive cases, adversarial inputs, cross-modal hidden intent, and dynamic rules.

3 SingGuard-Bench

  • SingGuard-Bench contains 56,340 examples across more than 80 fine-grained risk types, covering harmful and benign-sensitive content, adversarial cases, cross-modal hidden intent, and dynamic rules.It targets gaps in broad taxonomies, limited benign-sensitive examples, static policies, and sparse testing of hidden intent split across modalities.Source: §3 · p. 10
  • Each example pairs target content with an active policy, safety label, and fine-grained risk category or rule title, enabling evaluation of policy-following alongside moderation.The benchmark spans adversarial robustness, harmful-content recall, benign-sensitive precision, cross-modal intent reasoning, and adaptation to matching or non-matching rules.Source: §3 · pp. 10–11

3.1 Construction Pipeline

  • Construction combines rule-specific keyword expansion, data supplementation, and multi-model quality filtering with label, category, and explanation consistency checks.For hidden-intent examples, image and text are individually benign but jointly reveal unsafe intent; dynamic-rule cases cover unsafe→unsafe, unsafe→safe, safe→unsafe, and safe→safe shifts, with distracting non-matching rules.Source: §3.1 · p. 11

3.2 Benchmark Composition and Statistics

  • The image and multimodal subsets contain 54,340 examples, 68.4% benign-sensitive; adding 2,000 dynamic-rule examples yields 56,340 total.The dynamic subset has 500 examples in each of four policy-shift configurations. Construction uses 78 leaf nodes and 2,124 aligned English and Chinese keywords per language; image and multimodal subsets include 10,697 and 6,487 unsafe examples, respectively, alongside 29,966 and 7,190 benign-sensitive examples.Source: §3.2 · pp. 11–12, Table 2 · p. 12

4 Experiments

SingGuard is evaluated across static safety tasks and dynamic policy-following settings, with strong results across multimodal, image, text, and multilingual benchmarks. Its slow mode leads dynamic-policy accuracy, while ablations and distillation show gains from reasoning modes and teacher supervision.

4.1 Evaluation Setup

  • The evaluation spans five static safety axes scored by per-dataset binary F1 and macro-average F1, plus dynamic policy adaptation scored by accuracy.The axes cover multimodal query-response, image, text query, text response, and multilingual safety; policy-conditioned examples can change labels when the active rules change. Outputs without a safe/unsafe decision token count as incorrect.Source: §4.1 · p. 13

4.2 Multimodal Safety

  • 0.9092 macro-average F1 makes SingGuard-8B the strongest open-source model on multimodal query-response safety; SingGuard-2B leads cross-modal attack accuracy and F1.SingGuard-8B leads on VLGuard and SPA-VL, while LLaVAShield leads on five other datasets and ShieldGemma-2 on BeaverTails-V. On the cross-modal attack split, SingGuard-8B remains close to SingGuard-2B.Source: §4.2 · p. 14, Table 4 · p. 14, §4.6 · p. 18

4.3 Image Safety

  • 0.9141 macro-average F1 gives SingGuard-4B the strongest image-safety result, ahead of the best open-source baseline GuardReasoner-VL at 0.8570.SingGuard leads on DeepGHS NSFW, weapon, and crime-scene imagery and matches violence at 1.0; GuardReasoner-VL leads on UnsafeBench and Hateful Memes, the hardest slice for all models. LlamaGuard3-Vision scores 0.0025 on weapon detection and 0.1875 on violence.Source: §4.3 · pp. 14–15, Table 5 · p. 15

4.4 Text Query Safety

  • 0.8740 macro-average F1 makes SingGuard-8B the strongest text-query model, ahead of YuFeng-XGuard-Reason-8B at 0.8666 and Qwen3Guard-8B-strict at 0.8493.SingGuard-2B and 4B score 0.8661 and 0.8706, respectively; SingGuard leads on HarmBench and AILuminate1K while staying close to the best open-source model on several other datasets.Source: §4.4 · p. 15, Table 6 · p. 16

4.5 Text Response Safety

  • 0.8799 macro-average F1 makes SingGuard-4B the strongest text-response model, with SingGuard-2B at 0.8775 and SingGuard-8B at 0.8770.The input pairs the query with the assistant answer, distinguishing harmful assistance from a harmful-looking request that is safely refused; several query-strong baselines drop on response judgments.Source: §4.5 · p. 16, Table 7 · p. 16

4.6 Multilingual Classification

  • 0.8872 query-side and 0.8988 response-side macro-average F1 give SingGuard-8B the multilingual lead in both settings.On queries, it scores 0.8941 on RTP-LX; on responses, it scores 0.8318 on PolyGuardPrompts and 0.9658 on RTP-LX, where GraniteGuardian leads at 0.9734 with recall 0.9999. YuFeng-XGuard leads on query PolyGuardPrompts at 0.8824.Source: §4.6 · pp. 16, 18, Table 8 · p. 17, Table 9 · p. 17

4.7 Dynamic Policy Evaluation

  • 0.7415 average accuracy makes SingGuard-slow the strongest dynamic-policy model, improving safe2unsafe accuracy from Qwen3-VL-8B’s 0.3800 to 0.5700.SingGuard-hybrid scores strongest on unsafe2unsafe while offering a similar balance with lower latency than slow reasoning.Source: §4.7 · p. 18, Table 11 · p. 18

4.8 Ablation and Distillation Summary

  • 0.8631 to 0.8840 average score shows the on-policy-distilled 2B model improves over its student across image, multimodal, text-query, and text-response benchmarks.In the training and inference ablation, RL slow mode improves image, multimodal, and dynamic-policy performance over SFT, reaching 0.7415 dynamic-policy accuracy; hybrid offers a middle operating point, while fast mode retains much of safety performance at lower latency.Source: §4.8 · p. 19, Table 12 · p. 19

5 Related Work

Related work traces progress from taxonomy-based text moderation to multimodal and policy-adaptive guardrails, while identifying limits in runtime rule handling and benchmark coverage. SingGuard addresses these gaps with active-rule matching and flexible inference paths.

5.1 Text Safety Guardrails

  • Text guardrails have expanded from taxonomy-based classification to unified moderation, fine-grained judgments, and reasoning traces, yet remain chiefly text-only and tied to fixed or externally supplied policies.Source: §5.1 · p. 19
  • Text-only benchmarks cover harmful prompts and responses, refusals, jailbreaks, over-refusal, multilingual moderation, and product risks, but cannot test visual or cross-modal hazards.Source: §5.1 · p. 19
  • SingGuard extends text guardrails beyond fixed-taxonomy classification by accepting editable active rules and routing among fast, hybrid, and slow inference paths.Source: §5.1 · pp. 19–20

5.2 Multimodal Guardrails

  • Multimodal guardrails now inspect images, dialogue context, and assistant responses, including risks emerging across turns or through image-text combinations.Source: §5.2 · p. 20
  • Policy-adaptive visual guards show dynamic-rule following matters, but target narrower image settings; SingGuard instead matches open active rules individually across multimodal QA and response moderation.Source: §5.2 · p. 20
  • Existing multimodal benchmarks rarely test cross-modal composition, reasoning depth, and policy shifts together; SingGuard-Bench adds dynamic rules, attack transformations, keyword coverage, and policy-conditioned matching.Source: §5.2 · p. 20

6 Conclusion

SingGuard conditions multimodal safety judgments on the active runtime policy and combines multiple inference regimes with decoupled reinforcement learning. Across six benchmark families, it achieves state-of-the-art average F1 on every family, while dynamic-policy evaluation shows improved policy-following accuracy.

6 Conclusion

  • SingGuard uses a unified hierarchical taxonomy, dynamic rule-conditioned training, fast, hybrid, and slow inference regimes, and fast–slow decoupled reinforcement learning for low-latency, interpretable moderation.Conditioning decisions on the active policy rather than a frozen taxonomy decouples safety perception from memorized category priors.Source: §6 · p. 20
  • SingGuard achieves state-of-the-art average F1 on all six benchmark families (35 datasets), while dynamic-policy accuracy rises from 0.6465 (Qwen3-VL-8B) to 0.7415 (SingGuard-slow).The results underscore that guardrails must also recognize when a risk falls outside the active policy and refrain from over-blocking.Source: §6 · p. 20
  • Learning to identify risk is necessary but insufficient: deployed guardrails must also determine whether a risk falls outside the active policy and refrain from over-blocking.This conclusion is supported by dynamic-policy evaluation.Source: §6 · p. 20

7 Limitations

SingGuard’s effectiveness is limited by policy ambiguity, incomplete benchmark coverage, and imperfections in its data and inference pipeline. The authors call for stronger calibration, broader human evaluation, and more transparent uncertainty reporting.

7 Limitations

  • Ambiguous, incomplete, or conflicting runtime policies can produce inconsistent judgments across products, regions, and cultural contexts.Decisions depend on the quality and clarity of the active policy supplied at inference time.Source: §7 · p. 21
  • SingGuard-Bench cannot cover the long tail of real-world policies or emerging abuse patterns, necessitating continuous updates and human review for high-stakes deployment.The benchmark spans multimodal QA scenarios, attack types, and dynamic-rule settings, but cannot exhaust those risks.Source: §7 · p. 21
  • Synthetic data and model-assisted annotation may retain teacher bias, annotation artifacts, and mismatch with organic user traffic despite verification and consistency filtering.The training pipeline relies partly on synthetic data and model-assisted annotation.Source: §7 · p. 21
  • Hybrid early exit relies on token-level confidence that may be poorly calibrated under distribution shift, motivating stronger calibration and transparent uncertainty reporting.The authors also identify broader human evaluation as a future research need.Source: §7 · p. 21

A Prompt Templates · B Taxonomy Summary

SingGuard’s prompts make inference mode and active policy runtime controls, pairing configurable risk checks with fast judgments or policy-grounded reasoning. Its taxonomy organizes risks into 8 primary dimensions, 27 secondary categories, and 80+ fine-grained types, while allowing additional open rules.

A Prompt Templates

  • At inference, optional thinking-type and policy controls select the output mode and replace the default taxonomy with an active rule set.The system instruction is stored in the tokenizer chat template.Source: A Prompt Templates · p. 25

A.1 Default Policy-Grounded System Prompt

  • The default prompt evaluates the query, image, and response, including joint cross-modal intent and responses that assist violations in context.It checks active rules individually in slow mode, emits the triggered rule, and labels content Safe when no active rule is violated.Source: §A.1 · pp. 25–26

A.2 Fast Judgment Prompt

  • For low-latency deployment, fast judgment emits only the safety label and triggered category using the task description and active policy.Its output is the safe/unsafe label followed by the active rule in an <answer> tag.Source: §A.2 · p. 26

A.3 Rule-by-Rule Reasoning Trace

  • Rule-by-rule reasoning summarizes content, checks each active rule, and aggregates results; the example applies custom rules and judges the neutral rhino image safe.The example demonstrates following runtime rules rather than forcing the case into the built-in taxonomy.Source: §A.3 · pp. 26–27

A.4 Hybrid Reasoning Prompt

  • Hybrid reasoning stops after a confident initial binary judgment, but continues with policy-grounded verification for ambiguous or high-risk cases.After escalation, the model analyzes active rules and outputs the triggered rule.Source: §A.4 · p. 27

B Taxonomy Summary

  • The SingGuard taxonomy comprises 8 primary dimensions, 27 secondary categories, and 80+ fine-grained risk types, and permits additional open rules at inference time.The default categories include agent safety, politically sensitive content, animal abuse, and benign content.Source: §A.1 · p. 26, Table 14 · p. 28

C Rule Isolation Mask for Parallel Multi-Rule Inference

SingGuard’s Rule Isolation Mask (RI-Mask) enables parallel multi-rule moderation while preserving the clean decision context of independent category–rule checks.

C Rule Isolation Mask for Parallel Multi-Rule Inference

  • RI-Mask runs all active rule checks in parallel within one packed forward pass while preserving independent category–rule classification behavior.The mask avoids the N forward passes required to test N active rule blocks one at a time.Source: C Rule Isolation Mask for Parallel… · p. 29
  • Testing each category–rule block independently is more reliable than a joint prompt because it reduces interference from unrelated policies.Source: C Rule Isolation Mask for Parallel… · p. 29

C.1 Motivation

The section motivates RI-Mask as a way to retain isolated rule-by-rule classification while reducing the latency and redundant encoding costs of multimodal policy checks.

C.1 Motivation

  • RI-Mask shares the content prefix across rule branches and blocks cross-rule attention so each branch acts as an independent category–rule classifier.This design addresses redundant image and prompt encoding while preventing unrelated rule descriptions from diluting the evidence signal.Source: §C.1 · p. 29
  • Serial inference offers isolated rule contexts and greater reliability, but its latency grows with the number of active rules.Batch inference can exploit GPU parallelism across independent sequences, yet redundantly encodes shared image and prompt tokens for every rule block.Source: §C.1 · p. 29
  • Packing all rules into one ordinary context avoids redundant encoding but permits cross-rule attention that can dilute evidence and cause missed detections.Source: §C.1 · p. 29

C.2 Method

RI-Mask enables parallel rule-conditioned safety evaluation by sharing the content prefix while isolating each rule branch. Its packed sequence and attention mask preserve independent, causal rule judgments while avoiding redundant computation on shared content.

C.2 Method

  • RI-Mask places active rule suffixes after the shared content prefix, allowing all branches to reuse the content representation while remaining independent.The shared prefix contains the system prompt, image tokens, and user text.Source: §C.2 · p. 29
  • The attention mask lets every rule branch attend to the shared prefix, isolates branches from one another, and preserves causal output generation.Sharing the prefix avoids redundant KV-cache computation; branch position IDs continue from the end of the prefix as if each rule were appended independently.Source: §C.2 · p. 30, Figure 6 · p. 30
  • The resulting structure resembles Molmo2’s message-tree encoding, with the content prefix as a root and each rule judgment as an independent leaf.The analogy is to a shared visual prefix attended by multiple independent annotation branches (Clark et al., 2026).Source: §C.2 · p. 30

C.3 Implementation

RI-Mask implements parallel rule evaluation with block-sparse attention while preserving each rule branch’s standalone positional semantics.

C.3 Implementation

  • RI-Mask uses PyTorch FlexAttention, content-first rule-postfix prompts, and branch-local position IDs to generate all rule outputs in one forward pass.The outputs are split into per-rule predictions after parallel generation.Source: §C.3 · p. 30

C.4 Discussion

RI-Mask accelerates SingGuard’s fast-mode rule checking by reusing shared multimodal content while keeping rule assessments isolated. Its speedup depends on shared-prefix length, and the technique extends to other evaluations that apply multiple criteria to one input.

C.4 Discussion

  • RI-Mask avoids repeatedly encoding shared content while preserving rule isolation; longer image–text prefixes increase speedups, while slow-mode reasoning requires reduced parallelism to fit GPU memory.Fast-mode branches emit compact hit/not-hit outputs, and the shared content prefix dominates sequence length.Source: §C.4 · pp. 30–31
  • More than 5× faster SingGuard-2B multimodal inference is achieved with RI-Mask for 30 active rules in a deployment-style offline-rule setting.The acceleration is near-lossless compared with independent single-rule inference.Source: §C.4 · p. 31
  • The shared-input, independent-criteria approach generalizes to multi-label or multi-aspect evaluation, including moderation, compliance checking, and multidimensional quality assessment.Source: §C.4 · p. 31

Related papers

Loading 2606.22873v3…