PaperlayerView exact revision on arXiv

Source-linked AI summary

InGuard: Towards Generalized Inner Guardrail for Safe Text-to-Image Generation

Zeyu Wang, Xiaodan Li, Zhiwen Li, Yuefeng Chen, Hui Xue

arXiv:2609.27620v1cs.CVcs.AI

TL;DR

Conventional T2I guardrails screen prompts and completed images outside the generation pipeline, without using the model’s own representations. InGuard embeds risk grading, prompt enhancement, and latent detection within generation, while RevGen supplies a real-image-derived benchmark. Across five open-weight models, InGuard reports high safety rates, reduced benign disturbance, fewer added parameters, and skipped denoising steps.

  • Problem

    Conventional outer guardrails do not use the generator’s own representations, and post-hoc image checks run after generation while flagged prompts may be rejected even when they could be adjusted safely.

    Source: §1 · p. 2

  • Method

    InGuard grades prompts on text-encoder embeddings, enhances risky prompts with SAGE, and checks an intermediate one-step clean latent estimate while leaving base-model parameters untouched.

    Source: §1 · p. 2, §6 · p. 19

  • Results

    97.9–98.8% safety rate across five open-weight T2I models, matching or exceeding the outer guardrail with 57.5–73.5% less benign disturbance and approximately 3.7× fewer added guardrail parameters.

    Source: §1 · p. 3

  • Takeaways & Limitations

    InGuard routes prompts to interception, enhancement, or pass-through with latent-space detection, aiming to return safe images for risky requests rather than rejecting them outright.

    Source: §1 · p. 3

    The evaluation covers porn, gore, and IP; other policy dimensions such as hate and privacy remain unexplored.

    Source: §6 · p. 19

Abstract

from arXiv · show

Modern text-to-image (T2I) models generate high-quality images from arbitrary user prompts, yet they can just as easily produce not-safe-for-work (NSFW) content. Conventional outer guardrails consist of two components: a prompt classifier that checks for risk before generation, and a post-hoc image classifier that checks the fully generated image. In this design, both classifiers operate outside the generation pipeline and do not use the model's own representations. This separation can limit prompt-screening accuracy, while the image-side check runs only after the full generation cost has been spent. Moreover, a flagged prompt can only be rejected, even when it could be adjusted to produce a safe image. In this work, we propose the Inner Guardrail (InGuard), a safety framework that works inside the pipeline on the model's own representations, leaving base-model parameters untouched. First, a risk classifier grades each prompt as unsafe, risky, or benign based on the text encoder's embeddings, with no external language model. Second, SAGE (Soft-gated Asymmetric Guardrail for Embeddings) modifies the embeddings of risky prompts, aiming to return a safe image instead of a refusal. Third, a latent detector checks the one-step clean latent estimate midway through denoising, reaching nearly image-level performance and halting generation when risk is detected. We also construct the RevGen Safety Benchmark to evaluate T2I safety under realistic conditions: 10,000 prompts built through real-image reverse generation, with a rewriting step that supplies controlled intellectual-property (IP) characters, covering graded porn/gore risks, categorical IP risks, and benign negatives. Across five open-weight T2I models, InGuard reaches 97.9-98.8% safety rate, matching or exceeding the outer guardrail, with 57.5-73.5% less benign disturbance, ~3.7x fewer parameters, and 50-55.6% of denoising steps skipped.

1 Introduction

Open-weight text-to-image models can generate risky content, while conventional outer guardrails screen prompts and completed images without using the generator’s own representations. InGuard places safety components inside the pipeline, routing prompts by risk and aiming to return safe images for risky requests; its five-model evaluation reports high safety with less benign disturbance and fewer added parameters.

InGuard

  • InGuard grades prompts as unsafe, risky, or benign on the model’s text-encoder embeddings, then intercepts unsafe prompts, enhances risky ones, and passes benign ones through.SAGE softly blends risky-token embeddings toward a safe subspace while soft gating aims to limit changes to benign content; the latent detector checks an intermediate one-step clean latent estimate.Source: §1 · pp. 2–3

Benchmark

  • The RevGen Safety Benchmark contains 10,000 prompts reverse-generated from real images, covering graded porn and gore risks, categorical IP risks, and benign negatives.Its negatives support measurement of false positives.Source: §1 · pp. 2–3

InGuard

  • Figure 1 contrasts outer checking after full generation with InGuard’s in-pipeline plugins, which allow the same risky request to yield a safe image after enhancement.Figure 2 illustrates both successful enhancement and a failed enhancement caught by latent-space detection.Source: Figure 1 · p. 2, §1 · p. 2, Figure 2 · p. 3

Evaluation

  • 97.9–98.8% safety rate across five open-weight T2I models, matching or exceeding the outer guardrail while cutting benign disturbance by 57.5–73.5% with approximately 3.7× fewer added guardrail parameters.The evaluation compares InGuard with the outer guardrail on the RevGen benchmark.Source: §1 · p. 3

2 Related Work

Related work groups T2I safety methods into enhancement and detection, and reviews benchmarks used to evaluate safety. The section describes distinct trade-offs across these approaches and the scope of existing benchmark prompts and labels.

2 Related Work

  • Prior T2I safety research addresses both enhancement, which suppresses unsafe concepts, and detection, which identifies risk in prompts or generated content.The section also reviews benchmarks used to evaluate these approaches.Source: §2 · p. 3

2.1 T2I Safety Enhancement

  • T2I safety enhancement methods divide into training-time approaches that modify model parameters and inference-time approaches that operate on embeddings, latents, or attention.Training-time methods can incur model-update costs and affect semantically related benign concepts; inference-time methods include projection, guidance, and embedding or attention interventions.Source: §2.1 · pp. 3–4

2.2 T2I Safety Detection

  • Safety detection spans prompt and post-hoc image classifiers as well as in-generation inspection, with post-hoc checks running only after denoising finishes.The reviewed image detectors have differing target labels, and coverage of gore or controlled IP can require task-specific evaluation or adaptation.Source: §2.2 · p. 4

2.3 T2I Safety Benchmarks

  • Existing benchmarks draw prompts from in-the-wild queries, adversarial transformations, or curated taxonomies, and assess varied risk categories and mitigation settings.Most record risk presence and elicitation rather than severity; released benign prompts generally do not target graded near-boundary cases.Source: §2.3 · p. 4

3 RevGen Safety Benchmark

RevGen combines prompts derived from real images with graded porn and gore labels, categorical IP labels, and benign prompts, including cases written near risk categories. Its generation pipeline uses VLM reverse generation, IP rewriting, and diversity sampling; evaluations report generated-image diversity and risk elicitation across models.

3.1 Overview

  • RevGen combines real-image grounding, graded risk labels, and near-boundary benign negatives, alongside benign prompts and categorical IP labels.The benchmark covers porn, gore, and IP; diversity in the benchmark comparison is qualitative rather than a measured score.Source: §3.1 · p. 4, Table 1 · p. 5

3.2 VLM-Driven Generation Pipeline

  • The pipeline uses source-image pooling, VLM reverse generation, IP character rewriting for some prompts, and clustering-based diversity sampling.The resulting prompts are rendered by T2I models for the reported diversity and risk-elicitation measurements.Source: §3.2 · p. 4

3.2.1 Stage 1: Source Image Pooling

  • The reverse-generation path begins with internet-collected source images assembled to cover the target risk categories.The image pool is pre-shuffled.Source: §3.2.1 · p. 5

3.2.2 Stage 2: VLM-Driven Reverse Generation

  • A VLM converts each source image into English and Chinese prompts describing visual elements, producing porn, gore, and initial benign prompts.Some initial benign prompts are subsequently rewritten into controlled or benign IP variants.Source: §3.2.2 · p. 5

3.2.3 Stage 3: IP Character Rewriting

  • An LLM injects IP entities into VLM-generated normal prompts, creating controlled-character prompts and benign IP variants.Benign variants include non-controlled copyrighted characters, associated franchise characters, and lookalikes; recall is measured only on five controlled characters, while benign sub-categories test false positives under this policy.Source: §3.2.3 · pp. 5–6

3.2.4 Stage 4: Clustering-Based Diversity Sampling

  • Clustering-based sampling selects prompts closest to each cluster centroid, with a per-cluster cap and source-image-level separation between train and test splits.MinHash-LSH deduplication is applied between splits to reduce textual overlap.Source: §3.2.4 · pp. 6–7

3.3 Generated-Image Diversity

  • RevGen achieves the highest generated-image diversity on three of five models, while T2ISafety leads on Z-Image-Turbo and HunyuanImage-2.1.RevGen’s scores include 79.72 on FLUX.2-klein-base-9B, 58.80 on Qwen-Image-2512, and 52.97 on InternVL-U; higher Vendi Scores indicate a wider spread of visual content.Source: §3.3 · p. 7, Table 2 · p. 7

3.4 Risk Elicitation Statistics

  • RevGen has the highest porn hit ratio on four of five models; on FLUX.2-klein-base-9B, Ring-A-Bell reaches 52.78% against RevGen’s 30.50%.Across dimensions and models, risky prompts reach 21.60%–94.00% Risk Hit Ratio, while no benign sub-category exceeds 3.20%.Source: §3.4 · p. 7, Table 3 · p. 7

4 InGuard Framework

InGuard places graded risk screening, embedding enhancement, and latent detection inside the generation pipeline. Its components route prompts to interception, modification, or generation with an intermediate safety check.

4.1 Overview / 4.2 Prompt Embedding Risk Classification

  • The classifier scores porn, gore, and controlled-IP risk directly from conditioning embeddings, routing unsafe prompts to interception and remaining risky prompts to SAGE.Benign prompts enter generation unmodified; the latent detector checks requests that reach generation.Source: §4.1 · p. 8, §4.2 · p. 8

4.3 Embedding-Space Safety Enhancement / 4.3.1 Label-Gated Concept Subspace

  • SAGE selects category-specific toxic concept subspaces from 34 English and Chinese safety concepts, projecting only user-content tokens while preserving structural tokens.Prompts below the enhancement threshold skip SAGE; the IP head selects a character-specific concept group to avoid unrelated modification.Source: §4.3.1 · p. 9, §4.3 · p. 9

4.3.2 Trigger Detection via Leave-One-Out

  • Leave-one-out re-encoding scores token triggers by relative residual distance, but contextual changes mean the score is not a direct token-toxicity measure.The resulting projection aims to retain a token’s contextual role while removing alignment with selected toxic concepts.Source: §4.3.2 · p. 10

4.3.3 Asymmetric Soft-Gated Blending

  • A continuous asymmetric gate blends original and projected token embeddings, smoothing binary threshold jumps; lowering α strengthens projection but may reduce semantic preservation.Its branches are score-based regions, not ground-truth toxic and benign token labels, and its temperatures do not guarantee lower benign disturbance.Source: §4.3.3 · pp. 10–11

4.3.4 Category-Aware α Configuration

  • Category-aware α routes a single sensitivity by porn ≻ gore ≻ IP priority when risks co-occur, a deterministic policy rather than an optimized solution to category conflicts.IP concepts are active for levels 1–5; levels 6–7 denote benign IP variants and do not trigger enhancement.Source: §4.3.4 · p. 11

4.4 Latent-Space Safety Detection / 4.4.1 Latent Acquisition

  • During denoising, the detector classifies a one-step clean latent estimate, reusing predicted velocity so generation can stop before remaining denoising evaluations.This saves denoising steps for intercepted requests, not necessarily end-to-end latency; the classifier adds its own inference cost.Source: §4.4.1 · p. 12

4.4.2 Latent Detector Backbone

  • A ConvNeXt-Base backbone adapts its input stem to the VAE latent’s channel count while retaining the pretrained feature hierarchy and sharing features across three safety heads.The heads classify porn, gore, and IP risk from latent inputs.Source: §4.4.2 · pp. 12–13

4.4.3 Training Phase A: Domain-Adaptive Pretraining / 4.4.4 Training Phase B: Safety Fine-tuning

  • Phase A distills frozen RGB-teacher features into a latent-input student; Phase B fully fine-tunes that backbone on RevGen safety-labeled latents with jointly trained heads.Because the pairwise identity does not establish identical inference-time and VAE-latent distributions, Phase B adapts on generation-derived latents. Training includes borderline samples and separates splits by source image.Source: §4.4.3 · pp. 13–14, §4.4.4 · p. 14

5 Experiments

Experiments evaluate InGuard’s three components individually and compare the full system with an outer guardrail on safety and benign disturbance. The evaluation spans five open-weight LDMs and the RevGen test set.

5.1.1 Datasets

  • The 10,000-prompt RevGen test set uses safety labels for porn and gore from 0–5 and IP from 0–7; latent-detector training pools range from 5K to 50K.Phase A pretraining uses VAE-encoded natural images from OpenImages.Source: §5.1.1 · p. 15

5.1.2 Baselines and Comparisons

  • The comparison evaluates five open-weight LDMs and contrasts outer prompt/image classifiers with InGuard’s PE-MLP, SAGE, and intermediate-latent detector.The outer system uses mE5-base or mBERT-cased for prompts and ConvNeXt-Base for generated images; SAFREE is the primary SAGE baseline.Source: §5.1.2 · p. 15

5.2 Prompt Embedding Risk Classification

  • PE-MLP outperforms mE5 and mBERT across all risk dimensions and five LDMs while using over 10× fewer parameters.At operational thresholds, it also yields lower prompt-side false alarms on every model.Source: §5.2 · p. 15, Table 5 · p. 15

5.3 Embedding-Space Safety Enhancement

  • SAGE improves enhancement success by 8.5–24.9% and reduces benign disturbance by 1.1–5.1% versus SAFREE across five LDMs.On InternVL-U, success is 54.1% vs. 29.2%, while disturbance is 1.2% vs. 2.6%.Source: Table 6 · p. 16, §5.3 · p. 16

5.4 Latent-Space Safety Detection

  • At 50K, distillation pretraining raises Avg F1 by 0.32–1.00% across five LDMs, with larger gains at smaller pool sizes.For Z-Image-Turbo, 5K with pretraining reaches 90.18% Avg F1 versus 90.33% without distillation at 20K; higher Avg F1 does not guarantee better recall or benign disturbance at every operating point.Source: Figure 10 · p. 17, §5.4 · p. 17
  • At deployed intermediate steps, latent-detector Avg F1 is within 0.26–1.71% of image-level detection, and intercepted requests avoid 50–55.6% of denoising evaluations.The denoising-step fraction is a compute proxy; these figures do not establish reduced wall-clock latency or average serving cost.Source: Figure 11 · p. 17, §5.4 · pp. 17–18
  • 48–139× faster pretraining-data preparation uses a single VAE encoder pass, while fine-tuning data requires the full denoising pipeline.The reported ratio concerns sample preparation, not detector training or complete guardrail inference.Source: §5.4 · p. 18, Table 7 · p. 18

5.5 System-Level Comparison: Inner vs. Outer Guardrails

  • 57.5–73.5% lower benign disturbance accompanies safety rates matching or exceeding the outer guardrail, with approximately 3.7× fewer added parameters.At reported operating points, InGuard uses ∼100M versus 370.6M parameters; intercepted requests can stop after 44.4–50% of denoising steps.Source: §5.5 · pp. 18–19, Table 8 · p. 19

6 Conclusion and Future Work

InGuard combines prompt risk grading, embedding modification, and intermediate latent detection, with its base-model parameters untouched. Across five open-weight LDMs, it achieves high safety rates with less benign disturbance and fewer denoising steps, while the authors identify several scope and robustness directions for future work.

6 Conclusion and Future Work

  • Across five open-weight LDMs, InGuard achieves a 97.9–98.8% safety rate, matching or exceeding the outer guardrail while reducing benign disturbance by 57.5–73.5%.It uses approximately 3.7× fewer added parameters; interception at the selected intermediate step avoids 50–55.6% of denoising steps.Source: §6 · p. 19
  • InGuard uses PE-MLP to grade prompts, SAGE to modify risky-prompt embeddings, and a latent detector to check an intermediate one-step clean latent estimate.The base-model parameters remain untouched, and only guardrail classifiers are adapted to each backend.Source: §6 · p. 19
  • The label-gated design ties SAGE to prompt-classifier accuracy and a finite concept set, motivating work on adversarial rewriting and multi-character IP scenes.The evaluation covers porn, gore, and IP; hate and privacy remain unexplored, and future work could extend detection to unseen generators, video, and other iterative synthesis processes.Source: §6 · p. 19

7 Ethics Statement

The authors frame the work as reducing misuse risks from open-weight generators, not facilitating harmful or infringing content. The benchmark releases prompts and risk labels, but not source images or generated unsafe images.

7 Ethics Statement

  • The work studies safety mechanisms for T2I generation to reduce misuse risk, not to facilitate harmful or infringing content.The sensitive material includes NSFW content and controlled IP characters.Source: §7 · p. 19
  • The RevGen Safety Benchmark releases text prompts and risk labels, but withholds source images and generated unsafe images.It is built for research on safety evaluation.Source: §7 · p. 19
  • Controlled IP characters are used to evaluate policy compliance, and the authors discourage using the resources to generate, distribute, or promote unsafe or infringing content.Sensitive examples are included solely to illustrate the studied risks.Source: §7 · p. 20

8 Authors

The paper names five core contributors and acknowledges wider Alibaba AAIG team members for presentation, prototype-testing, and feedback contributions.

8 Authors

  • The listed core contributors are Zeyu Wang, Xiaodan LiB, Zhiwen Li, Yuefeng Chen, and Hui Xue.The paper identifies these individuals under “Core Contributors.”Source: §8 · p. 20
  • Wider Alibaba AAIG team members attended dry-run presentations, stress-tested early prototypes, and offered feedback.The authors acknowledge these contributions.Source: §8 · p. 20
  • The acknowledgement specifically credits candid feedback alongside presentation attendance and prototype stress-testing.These are the contributions the authors attribute to the wider team.Source: §8 · p. 20

A.1 Training Details

The appendix specifies training settings for the PE-MLP prompt classifier and the latent detector. The detector uses domain-adaptive pretraining followed by fine-tuning, with separate learning rates for its backbone and classification heads.

A.1 Training Details

  • PE-MLP is trained with AdamW at learning rate 1 × 10−3, weight decay 1 × 10−4, batch size 8, and cosine annealing.These are the prompt classifier's stated training settings.Source: §A.1 · p. 24
  • The latent detector's Phase A domain-adaptive pretraining runs 100k iterations with learning rate 1 × 10−4 and batch size 8.The single-layer stem is initialized with trunc normal.Source: §A.1 · p. 24
  • Latent-detector Phase B runs 100 epochs with batch size 16 and separate backbone and classification-head learning rates of 5 × 10−5 and 5 × 10−4.Both detector-training stages use weight decay 0.01, cosine annealing, and gradient clipping with max norm 1.0.Source: §A.1 · p. 24

A.2 Metric Definitions

The metrics distinguish final-output safety from disturbance to originally benign requests, with separate definitions for system-level and enhancement-only evaluation.

A.2 Metric Definitions

  • System safety is the fraction of test requests that are intercepted or yield a safe final image.Interception counts as safe.Source: §A.2 · p. 24
  • Benign disturbance combines prompt or latent/image interception with semantic alteration among requests whose prompt and original image are safe.For InGuard, these events are combined by logical OR without double counting; missing VLM judgments are estimated from successfully judged samples under a comparability assumption.Source: §A.2 · p. 24
  • For Table 6, SAGE and SAFREE use the same flagged-pool denominators, while enhancement success counts unsafe-to-safe conversion and disturbance counts semantic alteration without system interception.The denominators are U ∩ P for enhancement success and B ∩ P for benign disturbance.Source: §A.2 · p. 24

A.3 Tiered Guardrail Routing Logic

Tiered routing uses PE-MLP risk scores to select interception, enhancement, or generation with latent verification. In the deployed configuration, IP risks are enhanced and every request reaching generation is checked before completion.

A.3 Tiered Guardrail Routing Logic

  • The Tiered scheme routes prompts into unsafe, risky, or benign tiers according to PE-MLP risk scores.The deployed configuration sends IP-only risks to enhancement and gives every request that reaches generation latent verification before generation completes.Source: §A.3 · p. 24

A.4 Annotation Label Schemes

The appendix specifies the benchmark’s annotation criteria and reports risk-hit patterns, gating metrics, detector comparisons, and configuration details. It also describes how distillation and routing are evaluated, without supplying those full result values in these passages.

A.4 Annotation Label Schemes

  • Prompt-side labels grade porn and gore on six levels and IP on eight, with controlled-IP positives at levels 1–5 and associated or other IP outside policy.Image-side labels instead use three levels for porn and gore and seven for IP, with controlled characters coded 1–5 and associated IP coded 6.Source: §A.4 · p. 25
  • Manual audit found annotation accuracy consistently above 98%, while evaluation treats only explicit porn, gore, and controlled-IP labels as unsafe.Prompt-side and image-side labels use different scales; borderline porn/gore and associated IP count as safe in binary image evaluation.Source: §A.4 · pp. 24–26

A.5 Benchmark Risk Hit Ratio

  • Across five LDMs, gore risk-hit ratios range from 71.72% to 90.97%, while porn ratios range from 30.50% to 76.92%.Non-risk rows range from 0.03% to 3.20%; these are generated-content hit rates, not independently measured annotator false-positive rates.Source: §A.5 · p. 26, Table 9 · p. 26

A.6 Gating Effectiveness Details

  • Table 10 reports per-category and overall recall plus image-side false-alarm rates at operational thresholds, complementing Table 5’s accuracy and prompt-side false-alarm summary.The image-side false-alarm rate measures disagreement with image safety, not final system-level disturbance.Source: §A.6 · p. 26, Table 10 · p. 26

A.7 Backbone Architecture Comparison

  • On Z-Image-Turbo, ConvNeXt outperforms ViT across pool sizes; at 50K without distillation, Avg F1 is 91.4–92.1% versus 85.0–85.4%.Distilled ConvNeXt-Base is selected as a recall-oriented compromise, with 92.87% Avg F1 and 88.6M parameters.Source: §A.7 · p. 27, Table 11 · p. 27

A.8 Latent Detection Pretraining Effects

  • Table 12 compares image, latent-without-pretraining, and latent-with-distillation configurations across five LDMs, four pool sizes, and three metrics.Its delta rows are percentage-point changes; improvements need not occur simultaneously for Avg F1, risk recall, and disturbance rate.Source: §A.8 · p. 27, Table 12 · p. 28

A.9 System-Level Comparison Threshold Parameters

  • The inner configuration directly blocks prompts above per-LDM porn or gore thresholds, enhances remaining prompts meeting uniform porn, gore, or controlled-IP thresholds, and checks generation latents.The outer and inner configurations use different classifiers and routing; their numerically equal text and enhancement thresholds do not select identical samples.Source: §A.9 · pp. 27–28, Table 13 · p. 28
Loading 2609.27620v1…