PaperlayerView exact revision on arXiv

Source-linked AI summary

Latent Guard: a Safety Framework for Text-to-image Generation

Runtao Liu, Ashkan Khakzar, Jindong Gu, Qifeng Chen, Philip Torr, Fabio Pizzati

arXiv:2404.08031v2cs.CVcs.AIcs.LG

TL;DR

Existing T2I safeguards based on blacklists can be circumvented, while harmful-content classification can require large datasets and offer limited flexibility. Latent Guard uses generated safe and unsafe prompts, a learned latent representation, and contrastive training to detect blacklisted concepts; the paper reports robust detection across scenarios and generalization across datasets and concepts.

  • Problem

    Blacklists can be circumvented, while harmful-content classification requires large datasets and offers limited flexibility for T2I safety.

    Source: §1 · p. 2

  • Method

    Latent Guard uses task-specific generated data and contrastive learning to map blacklist concepts and prompts into a latent space on top of pretrained text encoders.

    Source: §1 · p. 2, §3.1 · p. 4, §3.2 · p. 5

  • Results

    Experiments demonstrate robust unsafe-prompt detection in many scenarios and good generalization across different datasets and concepts.

    Source: §5 · p. 14

  • Takeaways & Limitations

    The blacklist of concepts to check can be modified at test time without retraining.

    Source: §1 · p. 2

    Performance depends on the concepts selected for checking, and LLM-generated research data may differ from deployment inputs, potentially requiring additional data curation.

    Source: Limitations · p. 25

Abstract

from arXiv · show

With the ability to generate high-quality images, text-to-image (T2I) models can be exploited for creating inappropriate content. To prevent misuse, existing safety measures are either based on text blacklists, which can be easily circumvented, or harmful content classification, requiring large datasets for training and offering low flexibility. Hence, we propose Latent Guard, a framework designed to improve safety measures in text-to-image generation. Inspired by blacklist-based approaches, Latent Guard learns a latent space on top of the T2I model's text encoder, where it is possible to check the presence of harmful concepts in the input text embeddings. Our proposed framework is composed of a data generation pipeline specific to the task using large language models, ad-hoc architectural components, and a contrastive learning strategy to benefit from the generated data. The effectiveness of our method is verified on three datasets and against four baselines. Code and data will be shared at https://latentguard.github.io/.

1 Introduction

Text-to-image models can produce unsafe content, while existing blacklist and classification safeguards have important limitations. Latent Guard detects blacklisted concepts in text embeddings as a safety mechanism for T2I generators.

1 Introduction

  • T2I models can produce unsafe content such as deepfakes, propaganda, or offensive images, motivating safety mechanisms that block their creation.The paper identifies realistic image generation from textual descriptions as the source of these risks.Source: §1 · p. 2
  • Blacklists can be bypassed through rephrasing, while LLM-based filtering can be costly and vulnerable to encoder-targeting optimization.Latent Guard instead detects blacklisted concepts in latent text representations, which the authors describe as resistant to rephrasing and encoder-targeting optimization.Source: §1 · p. 2
  • Latent Guard introduces latent-space safety analysis for T2I generators, test-time adaptation, and evaluation across different scenarios.The paper presents it as a system for identifying content in latent text embeddings that can be adapted at test time.Source: §1 · p. 2

2 Related Work

Related work covers the development of text-to-image generation, methods for finding unsafe prompts, and approaches to safer image generation. Latent Guard is positioned alongside these approaches as a prompt-level safety mechanism.

Text-to-image generation

  • Diffusion models scaled text-to-image generation, while Latent Diffusion Models reduce computational requirements by performing diffusion in an autoencoder latent space.The discussed approaches use pretrained text encoders to understand input prompts.Source: §2 · p. 3

Seeking Unsafe Prompts

  • Unsafe-prompt research includes handcrafted prompts, adversarial prompt-space search, and optimization using textual-encoder representations [3] [17] [22] [27] [39] [42].The paper states that Latent Guard aims to defend against attacks that optimize prompts using textual-encoder representations.Source: §2 · p. 3

Towards Safe Image Generation

  • Safety approaches include blacklists, LLM preprocessing, image classifiers, diffusion-process manipulation, inpainting, and concept unlearning [45] [26].The authors contrast Latent Guard's deployment without further training with methods requiring expensive finetuning, and describe it as complementary to other safety strategies.Source: §2 · p. 3

3 The Latent Guard Framework

Latent Guard reframes prompt safety as detecting whether blacklisted concepts occur in input text, rather than classifying prompts as safe or unsafe. It generates paired training data, maps prompts and concepts into a learned latent space, trains their representations contrastively, and blocks prompts based on concept-to-prompt distances.

3 The Latent Guard Framework

  • The framework formalizes prompt safety as detecting blacklisted concepts, allowing the checked concept list to be changed at test time without retraining.This addresses the limitation of direct safe/unsafe classification, which the paper says cannot add new concepts without retraining.Source: §3 · pp. 3–4

3.1 Training Data Generation

  • LLMs generate unsafe prompts centered on sampled blacklist concepts and corresponding safe prompts that preserve the surrounding content while removing the unsafe concept.The unsafe prompts form U, and the paired safe prompts form S for contrastive training.Source: §3.1 · pp. 4–5

3.2 Embedding Mapping

  • A trainable Embedding Mapping Layer uses pretrained text-encoder features and cross-attention to emphasize prompt tokens related to a blacklisted concept.The resulting prompt embedding is conditional on the concept, while the concept embedding does not depend on the input prompt.Source: §3.2 · pp. 5–6

3.3 Contrastive Training Strategy

  • Contrastive training pulls a concept embedding toward its unsafe-prompt embedding and uses other unsafe prompts and paired safe prompts as negatives.The loss updates the Embedding Mapping Layer; safe prompts help distinguish the unsafe concept from surrounding sentence content.Source: §3.3 · pp. 6–7

3.4 Inference

  • At inference, Latent Guard compares each blacklisted concept with its conditional prompt embedding and blocks the prompt when a concept is detected by the distance rule.Concept embeddings can be precomputed, and inference adds Embedding Mapping Layer processing to standard T2I text encoding.Source: §3.4 · p. 7

4 Experiments

Latent Guard is evaluated on CoPro and public unsafe-prompt datasets, with tests covering explicit, synonym, and adversarial prompts. It ranks first or second across CoPro setups and outperforms baselines on unseen datasets, while analysis examines computational cost, latent features, and design choices.

Dataset details

  • CoPro contains 723 harmful concepts and 226,104 safe/unsafe prompts, with 578 ID concepts and 145 OOD concepts.Its evaluation includes Explicit, Synonym, and Adversarial sets for ID and OOD concepts; unsafe prompts are LLM-generated, with synonyms and adversarial texts substituted for concepts. Mixtral 8x7B generates data, and the CLIP Transformer is used as Etext.Source: Dataset details · pp. 8–9, Figure 5 · p. 8, Implementation details · p. 9

Baselines and metrics

  • The evaluation compares Text Blacklist, CLIPScore, BERTScore, and an LLM classifier, reporting accuracy and threshold-independent AUC where applicable.A single γ is tuned on CoPro validation data for CLIPScore, BERTScore, and Latent Guard; AUC is also reported for these threshold-based methods.Source: Baselines and metrics · p. 9

Quantitative evaluation

  • Latent Guard ranks first or second across all CoPro setups, training only on Explicit ID data, and ranks first or second independently of γ.Text Blacklist, CLIPScore, and BERTScore drop on Synonym and Adversarial sets; the LLM performs well across sets but uses 7 × 10^9 parameters and 0.383 seconds per prompt.Source: Table 1 · p. 10, Quantitative evaluation · p. 10

Generalization capabilities

  • Latent Guard outperforms all baselines in accuracy and AUC on Unsafe Diffusion and I2P++, and prompt filtering yields safer generated images than the baselines.On I2P++, Latent Guard's AUC is 0.749 versus 0.299 for CLIPScore.Source: Table 2 · p. 11, Generalization capabilities · p. 11

Qualitative evaluation

Feature space visualization

  • The t-SNE plots show mixed safe and unsafe CLIP embeddings but a clear separation in Latent Guard embeddings, although safe/unsafe separation is not explicitly enforced by its contrastive loss.The authors hypothesize that contrastive constraints separate embeddings of similar inputs and may produce higher-level safe/unsafe distinctions.Source: Figure 7 · p. 12, Feature space visualization · pp. 12–13

Computational requirements

  • For the tested configurations, Latent Guard adds only a few MB of GPU memory and at most around 35 ms of processing time.For the ID setup with batch size 578, one prompt requires 13 MB and around 1 ms; Stable Diffusion inference takes seconds.Source: Computational requirements · p. 12, Figure 6 · p. 12

Ablation studies

  • A 16-head, 128-dimensional mapping layer is selected; cross-attention and safe prompts improve performance, concept-centered detection generalizes better than a binary classifier, and reducing Ccheck consistently lowers performance.For fine-grained classification at d = 128, increasing heads from 1 to 16 raises performance from 0.874 to 0.931; removing safe prompts particularly harms Synonym and Adversarial results. The authors attribute the setup's generalization advantage to increased concept emphasis, and report that Ccheck can be changed at test time without retraining.Source: Ablation studies · pp. 13–14, Table 4 · p. 14, Table 5 · p. 14

5 Conclusion

Latent Guard detects unsafe prompts across multiple scenarios and datasets without finetuning the T2I model.

5 Conclusion

  • Latent Guard detects unsafe prompts robustly across multiple scenarios and generalizes across datasets and concepts without requiring T2I-model finetuning.The method addresses identification of blacklisted concepts in input prompts using a task-specific dataset called CoPro.Source: §5 · p. 14

Supplementary Material

The supplementary material details Latent Guard’s data-generation procedures and provides additional implementation information, results, and deployment guidance.

Supplementary Material

  • The supplement adds implementation and application details, additional results and ablations, and practical deployment recommendations and limitations.It organizes these materials into Sections A, B, and C, respectively.Source: Supplementary Material · p. 18

A Additional Information

Safe/Unsafe prompts generation

  • Unsafe prompts are generated as short captions containing the exact concept, while safe prompts rephrase the unsafe input to remove harmful content.Unsafe prompts are limited to 20 words, and multiple examples are supplied through in-context learning.Source: Safe/Unsafe prompts generation · pp. 19–20

Synonyms prompts

  • Usyn prompts are created by asking an LLM for a synonym of the unsafe concept that retains a harmful connotation.Source: Synonyms prompts · p. 20

Adversarial prompts

  • Uadv is generated with a GCG-based method that optimizes the concept alone, using 20 adversarial tokens for 500 iterations and filtering likely synonyms.The optimization uses CLIP’s tokenizer vocabulary and filters the 1,000 closest dictionary words by BERTScore before each optimization loop.Source: Adversarial prompts · pp. 20–21

A.2 Concept Generation

  • CID and COOD are generated with Mixtral 8x7B prompting, with in-context examples and category diversity used to improve output quality and breadth.Source: §A.2 · p. 21

B.1 Visualization of ROC Curves

The supplementary experiments report improved ROC performance over baselines, characterize harmful outputs in CoPro, and compare Latent Guard with concept removal.

B.1 Visualization of ROC Curves

  • Latent Guard’s ROC curves significantly outperform CLIPScore and BERTScore across CoPro’s Explicit, Synonym, and Adversarial scenarios on both ID and OOD cases.The reported comparison concerns false-positive and false-negative rates across thresholds.Source: §B.1 · pp. 21–22, Figure 8 · p. 22

B.2 CoPro images harmfulness

  • CoPro yields slightly fewer unsafe outputs than competing datasets by Q16+NudeNet classification, but more unique harmful samples and consistent unsafe-image counts across its splits.The authors note that Q16 and NudeNet may be negatively affected by distribution shift on synthetic data.Source: §B.2 · pp. 22–23, Table 7 · p. 23

B.3 Comparison with concept removal baselines

  • 0.365/0.312/0.066 on I2P and 0.471/0.321/0.029 on UnsafeDiffusion are the Inappropriate Probability values for No Safety Measure/[6]/Ours, respectively.Latent Guard is reported as competitive with concept removal while avoiding diffusion-model finetuning, preserving T2I quality, and allowing test-time blacklist extension.Source: §B.3 · p. 23

Impact of N during training

Additional experiments examine how training-concept count, available blacklist concepts, encoder choice, and adversarial attack affect Latent Guard’s performance.

Impact of N during training

  • When training concepts N were reduced from 300 to 150 to 75 by subsampling the original 578-concept ID set, performance consistently decreased, yet N = 75 remained competitive.The authors attribute the decrease to the smaller variance of training data.Source: Impact of N during training · p. 23

Impact of Ccheck on CoPro

Different text encoder

  • Latent Guard has comparable performance with OpenCLIP ViT-H and CLIP ViT-L, with the authors attributing a slight accuracy loss to OpenCLIP’s training dataset.The result is presented as evidence that the method transfers across text encoders with minimal adaptation.Source: Different text encoder · p. 24, Table 8 · p. 24

Alternative adversarial attacks

B.5 Qualitative Results

Additional qualitative results are reported as coherent with the main paper; the deployment recommendations combine Latent Guard with other safety checks and note a risk from open-source release.

B.5 Qualitative Results

  • Additional qualitative results for CoPro test prompts include generated images and detection results for Latent Guard and baselines, and are reported as coherent with the main paper.Source: §B.5 · p. 25

C Deployment Recommendations

Limitations

Latent Guard's effectiveness depends on the concepts selected at test time, the fit of LLM-generated data to deployment inputs, and engineering for the target text encoder.

Limitations

  • Results depend heavily on test-time concepts; users must customize unsafe concept lists for application requirements because no list can include every possible concept.Source: Limitations · p. 25
  • LLM-generated research data may shift from real deployment inputs, so additional data curation following the deployment distribution may be needed for better generalization.Source: Limitations · p. 25
  • Training Latent Guard on the text encoders used in T2I generation may require additional engineering; the authors recommend a multilayer security system to complement its limitations.Source: Limitations · p. 25

Related papers

Loading 2404.08031v2…