Source-linked AI summary
Nemotron 3.5 Content Safety Moderator: A Compact Multimodal, Multilingual, and Reasoning Enabled Content Safety Moderator
Varun Singh, Anuj Doshi, Makesh Narsimhan Sreedhar, Shaona Ghosh, Katherine Luna
TL;DR
Deployed safety moderation needs to handle multimodal inputs, multiple languages, generated responses, and domain-specific policies, while existing guardrails cover only parts of this setting. The paper presents Nemotron 3.5 CS, a compact 4B vision-language moderator and supporting dataset that combine these requirements. Across evaluations, it adds image- and policy-conditioned moderation while remaining competitive with specialized text guards, with reasoning intended selectively for audit and policy review.
Problem
Existing guardrails do not fully cover visual context, non-English inputs, generated responses, and policies that vary across application domains.
Method
The paper trains a compact 4B vision-language moderator for joint prompt, image, and response classification across 12 languages, using custom policies, optional reasoning, and a mixed safety dataset.
Results
Nemotron 3.5 CS adds image-conditioned and policy-conditioned moderation while remaining competitive with specialized text guards across the reported evaluation axes.
Takeaways & Limitations
Compact vision-language moderators can serve as deployable front-line safety components, with reasoning used selectively for audit and policy review.
Takeaways & Limitations
The model accepts one image alongside text, while videos, multi-page documents, and audio remain out of scope; document-like inputs are a main source of false positives.
Abstract
from arXiv · showhide
Safety moderation for deployed AI applications is moving beyond text-only prompts: systems increasingly need to judge images, documents, screenshots, and generated responses under policies that vary across domains. Existing guardrails usually cover only part of this setting, making it difficult to combine broad coverage, custom policy control, and low compute cost. We present Nemotron 3.5 Content Safety Moderator, also referred to as Nemotron 3.5 CS in this paper for brevity, a compact 4B vision-language safety moderator that jointly classifies user prompts, images, and assistant responses across 12 languages. Nemotron 3.5 CS returns safety labels for latency-sensitive moderation and can additionally produce concise reasoning traces that apply supplied custom policies and identify violated categories when reasoning is requested. We also release a multimodal and multilingual safety dataset for guard training, spanning human-labeled real-image moderation, benign vision-language and document tasks, synthetic rare-risk and jailbreak cases, and custom-policy examples. Across evaluations spanning multimodal safety, text moderation, multilingual robustness, custom-policy following, benign false positives, and latency, Nemotron 3.5 CS demonstrates a practical coverage tradeoff: it adds image-conditioned and policy-conditioned moderation while remaining broadly competitive with specialized guard models. These results suggest that compact vision-language moderators can serve as deployable front-line safety components, with reasoning used selectively for audit and policy review.
1 Introduction
Existing safety guards do not fully cover visual context, non-English inputs, generated responses, and domain-specific policies. Nemotron 3.5 CS is introduced as a compact multimodal moderator combining these capabilities with a released dataset and synthetic-data methodology.
- Motivation: Visual context, multilingual inputs, response moderation, and domain-specific policies remain incompletely covered by existing safety systems.These gaps motivate moderation that inspects text and images together, handles responses, follows supplied policies, and operates quickly.
- Proposed system: Nemotron 3.5 CS is a compact 4B moderator for joint prompt, image, and assistant-response classification across 12 explicitly trained languages.It accepts custom policies at inference time and can return direct safety labels or concise reasoning traces with violated categories.
- Data contribution: The released multimodal and multilingual dataset combines human-labeled real-image examples, benign vision-language data, synthetic rare-risk and jailbreak cases, and policy-following examples.The dataset is intended to support guard training across the paper’s multimodal, multilingual, adversarial, and custom-policy settings.
- Data contribution: The paper presents a methodology for generating synthetic data for training and evaluation datasets.
2 Related Work
Prior work has advanced text, multilingual, multimodal, and policy-conditioned safety moderation, but these capabilities remain largely separated. The related literature therefore leaves a gap at their intersection.
- Text-only safety models: Text-only guards improve prompt and response moderation but do not jointly address visual context, multilingual gaps, or custom-policy flexibility.
- Multilingual safety: Multilingual benchmarks expose weaknesses in English-centered moderation, while multilingual guards remain primarily text-only and omit image-conditioned moderation.
- Multimodal safety: Multimodal safety research shows that harmful intent can depend on image-text interaction or be hidden in visual prompts.Recent multimodal guards extend classification to image-conditioned inputs and responses, while multilingual multimodal safety remains a gap.
- Policy-conditioned and reasoning-based safety: Policy-conditioned systems treat moderation as instruction following over supplied rules, but existing systems do not combine this with multimodal inputs and multilingual coverage.
3 Methodology
Nemotron 3.5 CS is built as a compact, open-deployment-oriented vision-language moderator with structured inputs, an adapted safety taxonomy, supervised fine-tuning, and switchable inference modes.
- Design requirements: The design targets open release, compute-constrained deployment, multimodal inputs, and reliable multilingual coverage.These requirements motivate a small model suitable for settings including edge inference.
- Design requirements: Gemma 3-4B is selected as the base model because it fulfills the stated design requirements.
- Input and output interface: The structured context can include user text with an optional image, an assistant response, and a free-form custom policy.A custom chat template assembles these components into one sequence.
- Safety taxonomy: The taxonomy adapts AEGIS 2.0’s thirteen core unsafe categories and adds economic harm as a fine-grained category.The added category covers financial fraud, predatory lending, market manipulation, and related harms.
- Fine-tuning procedure: Supervised fine-tuning formats each example around the full moderation context, including optional images, responses, inference mode, and category-presence controls.
- Inference: Direct classification emits SAFE/UNSAFE labels for latency-sensitive filtering, whereas reasoning mode produces a concise policy-applied trace before the verdict for audit and review.Mode and category outputs are toggled with inference-time tokens.
4 Data and Training
The training data addresses safety-moderation challenges through a mixture of human annotation, public datasets, and synthetic generation. Its composition emphasizes grounded multimodal coverage, rare-risk coverage, and custom-policy following.
- Data challenges: Safety-data curation is complicated by domain specificity, fragmented taxonomies, sensitive content, legal restrictions, and high annotation costs.
- Training mixture: The training mixture combines human annotation, existing public datasets, synthetic data generation, and topic-following examples for custom-policy application.Human data grounds real-world visual scenarios, synthetic data fills rare-harm and adversarial gaps, and topic-following teaches supplied policies.
- Human annotation: Human annotation uses sourced images and in-house annotators for labeling.Image sources are internal and external and carry varying licensing restrictions.
- Training-data composition: 99% of training images are real photographs or plain-background text rather than synthetic generations.This composition is presented as addressing weaknesses attributed to heavily synthetic multimodal safety datasets.
- Training-data composition: Table 1 lists the six source types in the released dataset composition in approximate descending order of volume contribution.
5 Synthetic Data Generation (SDG)
The SDG process supplements multimodal safety training data for scarce, legally restricted, or annotator-unsafe harm categories, especially synthetic refusals and jailbreak patterns.
- SDG supplements training data for scarce, legally restricted, or unsafe-to-produce harm categories.It is applied primarily to generate synthetic refusals and jailbreak patterns through an eight-stage pipeline.
6 Evaluation
The evaluation measures six capabilities using benchmark-aligned metrics and comparison sets tailored to each system's supported interfaces. It covers multimodal and text safety, multilingual robustness, custom-policy following, benign false positives, and latency.
- Six capabilities are evaluated: multimodal harmful-content detection, text safety, multilingual robustness, custom-policy following, benign-input false positives, and latency.Each capability addresses a distinct evaluation question.
- Baseline comparisons use the strongest applicable systems for each capability because image, response, policy, and latency interfaces differ across models.Unsupported settings are marked in Appendix B.
- Harmful-F1 is the primary classification metric, supplemented by harmful recall, benign false-positive rate, and latency where appropriate.Latency includes time to first token and end-to-end response time.
- Multimodal safety: Multimodal benchmarks test unsafe requests whose interpretation depends jointly on image and text.VLGuard reports prompt-side accuracy and harmful-F1, while MM-SafetyBench reports harmful-F1.
- Text safety benchmarks: Text benchmarks evaluate prompt and response classification on Aegis 2.0, XSTest, and WildGuard.These tests assess whether the multimodal model remains competitive on established text moderation tasks.
- Multilingual safety: Multilingual safety is evaluated on PolyGuard, RTP-LX, MultiJail, XSafety, Aya Red Teaming, Multilingual Aegis, and LinguaSafe.
- Custom-policy following and reasoning: Custom-policy following is evaluated with DynaGuardrail and CoSA, which require applying supplied policy descriptions rather than fixed taxonomies.The supplied passage introduces reporting by DynaGuardrail policy domain.
- Benign multimodal false positives: Benign multimodal false positives are measured on safe examples from MMMU, DocVQA, and AI2D.These datasets include screenshots, forms, documents, charts, and educational diagrams.
7 Results and Analysis
Nemotron 3.5 CS broadens safety coverage across visual, multilingual, policy-conditioned, benign-input, and latency settings while remaining competitive with specialized guards. Reasoning supports inspection but is not uniformly better for F1.
- Safety coverage: Nemotron 3.5 CS is strongest when moderation requires image context, while remaining competitive on text safety.Qwen3Guard remains stronger on established text-only prompt and response splits.
- Multilingual performance: The model provides broad coverage across 12 trained languages, with near-parity prompt performance against Qwen3Guard and stronger response-capable multilingual results.Performance is consistent even for out-of-training-set languages such as Vietnamese and Russian on LinguaSafe.
- Custom policy and reasoning: Nemotron 3.5 CS applies supplied custom policies across domain-specific settings.Reasoning-enabled outputs support inspection but do not uniformly improve F1, so they are reported separately rather than as the default.
- Benign false positives: MMMU and AI2D show low benign false-positive rates, while DocVQA is the main failure slice.Document-like inputs can resemble privacy-sensitive content, motivating targeted data improvements.
- Latency: 60/76 ms TTFT/E2E on RTVLM image-text inputs compares with 99/118 ms for Llama Guard 4 at maximum concurrency 1.DynaGuardrail text/custom-policy inputs average about 17/33 ms, while reasoning increases E2E latency by emitting traces.
8 Conclusion
Nemotron 3.5 CS is a compact 4B vision-language moderator for prompts, images, and responses across 12 languages, with custom policies and optional audit reasoning. The paper concludes that one compact model can add image- and policy-conditioned moderation while staying competitive with specialized text guards, supported by released model, data, and SDG resources.
- Nemotron 3.5 CS jointly classifies prompts, images, and assistant responses across 12 languages with supplied custom policies and optional reasoning traces.
- A single compact model adds image-conditioned and policy-conditioned moderation while remaining competitive with specialized text guards.
- Table 2 compares Nemotron 3.5 CS with the strongest applicable baseline for each capability slice.Negative deltas are favorable for false-positive rate and latency; latency values are milliseconds.
- The release includes model weights, training data, and the SDG pipeline for further work on deployable, open-weights multimodal safety moderation.
9 Ethical Considerations
The work highlights ethical risks from human-judgment bias, limited visual diversity, and releasing safety data that may enable adversarial misuse.
- Human safety labels may encode annotators’ cultural backgrounds, policy interpretations, and fatigue despite guidelines and quality checks.
- Limited visual diversity may overrepresent particular image styles, geographies, demographics, document formats, and everyday settings.
- These dataset limitations can lead to over-blocking some topics or communities and under-detection of harms in less represented settings.
- Released safety examples may help adversaries infer category boundaries, design jailbreaks, or disguise harmful content.
- The released moderation data should be handled as a sensitive artifact.
10 Limitations
The moderator’s current scope and evaluation infrastructure impose important boundaries. It accepts only single-image inputs, while detailed multimodal evaluation is constrained by scarce public response-side safety benchmarks and aggregate metrics that can hide slice-level failures.
- Nemotron 3.5 CS accepts one image alongside text, excluding videos, multi-page documents, and audio inputs.
- Document-like inputs remain the main source of false positives, particularly on DocVQA, and need targeted data improvements.
- Public benchmarks with image-conditioned response-side safety labels are scarce, limiting multimodal evaluation.
- Aggregate scores can mask language- and category-level failures, motivating more detailed evaluation slices in Appendix B.
- Table 4 reports benchmark-level harmful-F1 values for multimodal and text-safety rows, with unsupported text-only baselines excluded from multimodal comparisons.
B.2 Multilingual Results
The appendix reports aggregate multilingual results and per-language slices to examine coverage across the evaluated languages. Vietnamese and Russian are identified as outside Nemotron 3.5 CS’s 12 explicitly trained languages.
- Table 5 reports aggregate multilingual scores across benchmarks.
- Tables 6 and 7 provide language-level slices for inspecting uneven multilingual coverage.
- Vietnamese and Russian are outside Nemotron 3.5 CS’s 12 explicitly trained languages.