PaperlayerView exact revision on arXiv

Source-linked AI summary

Safe-CLIP: Removing NSFW Concepts from Vision-and-Language Models

Samuele Poppi, Tobia Poppi, Federico Cocchi, Marcella Cornia, Lorenzo Baraldi, Rita Cucchiara

arXiv:2311.16254v3cs.CVcs.AIcs.CLcs.MM

TL;DR

Unsafe and biased behavior in vision-and-language models can arise from inappropriate content in web-scale training data, and unsafe inputs can trigger inappropriate retrieval or generation. Safe-CLIP fine-tunes CLIP using synthetic safe and unsafe vision-language data to redirect unsafe content while preserving embedding structure. Experiments report improved safety across retrieval and visual and textual generation, and the resulting model is presented for use with downstream generative models.

  • Problem

    Unsafe textual or image inputs can reach inappropriate regions of CLIP-like embedding spaces, leading to inappropriate retrieval or toxic generated content.

    Source: §1 · p. 2

  • Method

    Safe-CLIP fine-tunes CLIP on synthetic safe and unsafe images and captions with losses that redirect unsafe content while preserving embedding-space structure.

    Source: §3.2 · p. 7, §5 · p. 14

  • Results

    Experiments report significantly improved safety in retrieval and visual and textual generation.

    Source: §1 · p. 3, §5 · p. 14

  • Takeaways & Limitations

    The resulting Safe-CLIP can be used for cross-modal retrieval and with image- and text-generative models while retaining safe semantics of input prompts.

    Source: §1 · p. 3

    The stated objective is to preserve as much as possible the original embedding structure near safe regions for straightforward connection to downstream models without further adaptation.

    Source: §3 · p. 5

Abstract

from arXiv · show

Large-scale vision-and-language models, such as CLIP, are typically trained on web-scale data, which can introduce inappropriate content and lead to the development of unsafe and biased behavior. This, in turn, hampers their applicability in sensitive and trustworthy contexts and could raise significant concerns in their adoption. Our research introduces a novel approach to enhancing the safety of vision-and-language models by diminishing their sensitivity to NSFW (not safe for work) inputs. In particular, our methodology seeks to sever "toxic" linguistic and visual concepts, unlearning the linkage between unsafe linguistic or visual items and unsafe regions of the embedding space. We show how this can be done by fine-tuning a CLIP model on synthetic data obtained from a large language model trained to convert between safe and unsafe sentences, and a text-to-image generator. We conduct extensive experiments on the resulting embedding space for cross-modal retrieval, text-to-image, and image-to-text generation, where we show that our model can be remarkably employed with pre-trained generative models. Our source code and trained models are available at: https://github.com/aimagelab/safe-clip.

1 Introduction

Unsafe text and images can reach unsafe regions of CLIP-like embedding spaces, leading to inappropriate retrieval or generation. Safe-CLIP fine-tunes CLIP to reduce sensitivity to such inputs while retaining their safe semantics across retrieval and generation tasks.

1 Introduction

  • Unsafe prompts and images can reach unsafe embedding regions, resulting in inappropriate retrieval or toxic image descriptions.The introduction frames this as a risk for CLIP-like vision-and-language models.Source: §1 · p. 2
  • Safe-CLIP fine-tunes CLIP with losses that redirect inappropriate content while preserving the embedding-space structure.Training data pairs safe and unsafe vision-language items; a toxic language model generates unsafe text from safe, visually grounded descriptions, and an NSFW-capable diffusion model generates images.Source: §1 · pp. 2–3
  • The fine-tuned model is presented for cross-modal retrieval, text-to-image generation, and image-to-text generation.Examples include retrieving an appropriate image with similar semantics, generating appropriate images with Stable Diffusion, and producing descriptions without inappropriate content with LLaVA.Source: §1 · p. 3
  • Experiments report significantly improved safety in retrieval and visual and textual generation.The approach also significantly reduces generation of NSFW content.Source: §1 · p. 3

2 Related Work

Prior work addresses concept removal in generative models and NSFW detection, while this paper targets NSFW concepts in contrastive CLIP-like embedding spaces. It also relates its approach to fine-tuning language models to produce unsafe prompts from safe counterparts.

2 Related Work

  • Earlier generative-model methods steer away from a fixed set of NSFW concepts or fine-tune to erase a visual concept using its name.Other concept-removal work considers styles, concepts, or objects in text-to-image models [56].Source: §2 · pp. 3–4
  • This work instead removes NSFW concepts from contrastive CLIP-like models for retrieval and visual or textual generation.Compositional patterns reported in CLIP suggest a distinctive path from safe to NSFW zones.Source: §2 · p. 4
  • The paper uses NSFW language and image detectors, including DistilBERT [43], Q16, and NudeNet, amid prior work on detection.It also fine-tunes Llama 2 [50] to produce unsafe prompts from pre-existing safe counterparts, connecting to work on parameter-efficient and low-data LLM fine-tuning [11] [21] [54].Source: §2 · p. 4

3 Proposed Approach

Safe-CLIP fine-tunes both CLIP encoders to ignore inappropriate input while preserving the embedding structure around safe content. It trains on paired safe and unsafe text-image examples generated for the ViSU dataset.

3.1 Building the ViSU dataset

  • Safe and unsafe text-image pairs are constructed so unsafe items retain similar semantics to their safe counterparts, supporting fine-tuning toward the desired encoder behavior.The dataset is generated automatically because such paired data is not available.Source: §3.1 · p. 5
  • Llama 2-Chat is fine-tuned on 100 manually curated safe-unsafe pairs, then preference-optimized using automatic rankings based on NSFW degree and semantic similarity.The resulting generator maintains semantic relatedness and can produce varied unsafe content beyond its training concepts.Source: §3.1 · pp. 5–7
  • ViSU contains 165k quadruplets of safe and unsafe sentences and images, generated from COCO Captions and balanced across twenty NSFW categories.Unsafe sentences are generated from safe, visually relevant sentences, then used to generate corresponding unsafe images.Source: §3.1 · pp. 6–7

3.2 Making CLIP Safe

  • Safe-CLIP fine-tunes both CLIP encoders with redirection losses for unsafe inputs and structure-preservation losses for safe regions.The method combines four losses, including a contrastive term between safe visual and textual embeddings.Source: §3.2 · pp. 7, 9
  • The redirection losses use symmetric InfoNCE to align unsafe text with corresponding safe images and unsafe images with corresponding safe text, distinguishing matching from non-matching pairs.Structure-preservation losses retain the original embedding organization for safe inputs.Source: §3.2 · p. 8

4 Experiments

Experiments evaluate ViSU and test Safe-CLIP on retrieval and generation, including real NSFW images. Results report improved safety across these settings while preserving safe semantics and, in retrieval, good safe-only performance.

4.1 Experimental Setting

  • The ViSU training set contains 165k safe-and-unsafe quadruplets; validation and testing use 5k samples, and text-to-image experiments also use 4,703 I2P prompts.The CLIP backbone is ViT-L/14, selected to match the encoders used by Stable Diffusion v1.4 and LLaVA.Source: §4.1 · p. 9

4.2 Evaluating the ViSU Dataset

  • Llama 2 fine-tuning raises generated NSFW sentences from 9.3% with the original model to 79.1% after SFT and DPO, while ViSU exceeds I2P’s NSFW-sentence rate.GPT-3.5 rates 79.1% of ViSU sentences as NSFW, compared with 13.9% for I2P.Source: §4.2 · p. 10, Table 1 · p. 10

4.3 Evaluating the Safe-CLIP Embedding Space

  • Safe-CLIP improves unsafe-query retrieval R@1 by 6.0 points for text-to-image and 14.6 points for image-to-text versus original CLIP, while preserving good safe-only performance.For real NSFW images from all three sources, it reduces retrieved NSFW items by more than 45 points for visuals and 30 points for text.Source: §4.3 · pp. 11–12, Table 2 · p. 11, Table 3 · p. 12

4.4 Safe-CLIP for Text-to-Image Generation

  • Safe-CLIP reduces NSFW image-generation probabilities by 13.5 points on I2P prompts and 22.6 points on ViSU prompts with Stable Diffusion v1.4.It also benefits existing NSFW-removal strategies; qualitative samples preserve scene semantics while preventing inappropriate content.Source: §4.4 · pp. 13–14, Table 4 · p. 13, Figure 4 · p. 13

4.5 Safe-CLIP for Image-to-Text Generation

  • Safe-CLIP’s visual encoder significantly reduces inappropriate text generation compared with original LLaVA when describing real NSFW images.The experiment uses LLaVA based on Llama 2-13B-Chat and reports NSFW-text percentages and toxicity measures.Source: §4.5 · p. 14, Table 5 · p. 14

5 Conclusion

Safe-CLIP fine-tunes CLIP-like models using synthetic safe and unsafe image-caption data and losses that redirect unsafe content while preserving embedding-space structure. Experiments demonstrate its applicability to cross-modal retrieval and image-to-text and text-to-image generation.

5 Conclusion

  • Safe-CLIP fine-tunes CLIP-like models on automatically collected safe and unsafe images and captions, redirecting unsafe content while preserving embedding-space structure.The ViSU visual data is withheld; verified researchers can request the textual portion and receive generation seeds and instructions.Source: §5 · p. 14

Supplementary Material

The supplementary material provides further dataset analyses, experiments with additional CLIP variants, comparisons, generated-output quality analyses, qualitative results, and discussion of ethical implications and limitations.

Supplementary Material

  • Supplementary materials add ViSU analyses, experiments with CLIP ViT-L/14@336 and OpenCLIP ViT-H/14, comparisons, output-quality analyses, qualitative results, and discussion of ethics and limitations.Source: Supplementary Material · p. 18

A ViSU Dataset Analysis

This analysis examines ViSU’s category alignment, text distributions, and the quality of its generated safe–unsafe data. Its user study supports the unsafe content and semantic coherence of the generated examples.

I2P-ViSU category mapping

Word distributions

  • ViSU’s unsafe sentences contain varied, COCO-like captions and frequent inappropriate concepts, unlike I2P prompts’ typical Stable Diffusion words.The compared word clouds are drawn from the test sets.Source: A ViSU Dataset Analysis · p. 19, Figure 5 · p. 19

Analysis on generated NSFW images

  • NewRealityXL generates more inappropriate images than Stable Diffusion v1.4 while producing images more consistent with their prompts.The comparison uses unsafe ViSU test sentences and NudeNet, Q16, and CLIP image-text similarity measures.Source: A ViSU Dataset Analysis · p. 19, Table 7 · p. 19

Dataset examples

  • After SFT, Llama 2-Chat generated toxic captions from 100 curated pairs; adding DPO improved their semantic alignment with the original safe captions.The original model often returned the safe text or standard refusal responses.Source: A ViSU Dataset Analysis · p. 20

User study

  • In a 1,000-quadruplet study with 20 participants, generated data was largely NSFW and semantically preserved; human preferences matched the DPO ranking strategy 76.4% of the time.The study assessed unsafe content and semantic coherence in generated examples and sentence pairs.Source: A ViSU Dataset Analysis · p. 20, Table 8 · p. 20

Dataset examples

  • Sample ViSU quadruplets pair safe image–text items with corresponding unsafe versions while preserving semantic content.Source: A ViSU Dataset Analysis · p. 20

B Additional Implementation Details

The implementation uses low-rank adaptation for CLIP and Llama 2 fine-tuning. The learned CLIP checkpoint can incorporate the adapted weights without retaining the original checkpoint.

Low-rank adaptation in CLIP fine-tuning

  • LoRA saves memory and speeds CLIP fine-tuning; its learned matrices can be collapsed into the pretrained weights, so the final checkpoint need not store them separately.A recipient of a sanitized model would not thereby have easy access to the original weights.Source: B Additional Implementation Details · p. 20

Llama 2 fine-tuning details

  • ViSU generation fine-tunes Llama 2-Chat 7B with SFT followed by DPO, using β = 0.1, batch size 16, learning rate 5 × 10^-7, and LoRA ranks 64 and 8, respectively.Both phases use α = 16.Source: B Additional Implementation Details · p. 21

C Additional Experimental Results

Additional experiments report safer retrieval and generation across CLIP-based backbones and downstream models, while also examining quality preservation and an exception under strong diffusion guidance. The results include comparisons with ESD and standard multimodal-LLM benchmarks.

Comparison with ESD

  • For Stable Diffusion v1.4, Safe-CLIP yields lower unsafe-image probabilities than both the original model and ESD, including for the sexual category.The comparison averages classifier probabilities over I2P and ViSU samples and five random seeds.Source: C Additional Experimental Results · p. 21, Table 9 · p. 21

Retrieval results with CLIP ViT-L/14@336 and OpenCLIP ViT-H/14

Text-to-image generation results with SD v2.0

  • Safe-CLIP lowers unsafe-image generation probability with Stable Diffusion v2.0 in almost all tested settings, except SLD-Strong on I2P.The authors attribute the exception to difficulty combining SLD-Strong guidance with the fine-tuned embedding space; SLD can degrade realism more than Safe-CLIP.Source: C Additional Experimental Results · p. 23, Table 12 · p. 23, Table 14 · p. 24

Additional qualitative results

Image-to-text generation results with LLaVA 1.5 and 1.6

Evaluating generation quality preservation

D Discussion and Limitations

The discussion frames Safe-CLIP as a post-training approach for mitigating inappropriate concepts across retrieval and generation. The authors note ethical and scope constraints, including possible failures and the need for broader training data.

Ethical implications

  • Safe-CLIP targets shared embedding spaces used in retrieval and image-to-text generation, and offers an alternative to diffusion-side suppression that cannot be disabled by changing prediction-time source code.The authors describe this as providing better safety guarantees than the alternatives discussed.Source: D Discussion and Limitations · p. 25
  • The approach depends on models acquiring inappropriate concepts during pretraining, so the authors propose post-training mitigation rather than removing unsafe content entirely from training data.Source: D Discussion and Limitations · p. 25
  • The toxic-content collection relies on a fine-tuned LLM, and its representations may reflect Llama 2 and ViSU’s societal dispositions, potentially limiting sentiment diversity.The paper says the LLM used to generate inappropriate content will not be released.Source: D Discussion and Limitations · p. 25

Addressing the legality of the dataset

Limitations

Related papers

Loading 2311.16254v3…