PaperlayerView exact revision on arXiv

Source-linked AI summary

SAFREE: Training-Free and Adaptive Guard for Safe Text-to-Image And Video Generation

Jaehong Yoon, Shoubin Yu, Vaidehi Patil, Huaxiu Yao, Mohit Bansal

arXiv:2410.12761v2cs.CVcs.AIcs.LG

TL;DR

Unsafe-content risks have grown in image and video generation, while existing safeguards face limits involving training, adaptability, and quality. SAFREE filters toxic concepts through text embeddings without changing model weights; the authors report state-of-the-art training-free T2I results and extension across models and tasks.

  • Problem

    Prior safeguards either modify model weights through unlearning or editing, or, among training-free filters, may miss implicit triggers and degrade quality under hard filtering.

    Source: §1 · p. 2

  • Method

    SAFREE identifies toxic-subspace-related tokens through masked-embedding proximity and jointly filters text embeddings and visual latent spaces without weight updates.

    Source: §1 · p. 3, §3.1 · p. 5

  • Results

    47% lower ASR than I2P, alongside state-of-the-art performance across five T2I benchmarks.

    Source: §1 · p. 3, §4.3 · p. 8

  • Takeaways & Limitations

    The authors present SAFREE as a safeguard for more reliable and responsible visual content creation across T2I and T2V.

    Source: §1 · p. 3

    Implicit toxic prompts written in a chain-of-thought style can still jailbreak SAFREE and produce unsafe content.

    Source: E LIMITATION & BROADER IMPACT · p. 19

Abstract

from arXiv · show

Recent advances in diffusion models have significantly enhanced their ability to generate high-quality images and videos, but they have also increased the risk of producing unsafe content. Existing unlearning/editing-based methods for safe generation remove harmful concepts from models but face several challenges: (1) They cannot instantly remove harmful concepts without training. (2) Their safe generation capabilities depend on collected training data. (3) They alter model weights, risking degradation in quality for content unrelated to toxic concepts. To address these, we propose SAFREE, a novel, training-free approach for safe T2I and T2V, that does not alter the model's weights. Specifically, we detect a subspace corresponding to a set of toxic concepts in the text embedding space and steer prompt embeddings away from this subspace, thereby filtering out harmful content while preserving intended semantics. To balance the trade-off between filtering toxicity and preserving safe concepts, SAFREE incorporates a novel self-validating filtering mechanism that dynamically adjusts the denoising steps when applying the filtered embeddings. Additionally, we incorporate adaptive re-attention mechanisms within the diffusion latent space to selectively diminish the influence of features related to toxic concepts at the pixel level. In the end, SAFREE ensures coherent safety checking, preserving the fidelity, quality, and safety of the output. SAFREE achieves SOTA performance in suppressing unsafe content in T2I generation compared to training-free baselines and effectively filters targeted concepts while maintaining high-quality images. It also shows competitive results against training-based methods. We extend SAFREE to various T2I backbones and T2V tasks, showcasing its flexibility and generalization. SAFREE provides a robust and adaptable safeguard for ensuring safe visual generation.

1 INTRODUCTION

SAFREE addresses safety risks in visual generation with a training-free filter that does not alter model weights. It combines text-embedding and latent-space filtering, and the authors report state-of-the-art performance among training-free methods across five T2I benchmarks.

1 INTRODUCTION

  • Existing unlearning and editing methods require training or alter model weights, while training-free filters may miss implicit triggers and hard filtering can degrade quality.The paper attributes the latter risk to distribution shifts in prompts subjected to hard filtering.Source: §1 · p. 2
  • SAFREE identifies tokens associated with a toxic subspace and projects them orthogonally while keeping them in the original input embedding space.This design aims to preserve safe prompt content while filtering harmful concepts.Source: §1 · pp. 2–3
  • Self-validating filtering adjusts denoising steps as needed, while latent re-attention reduces the influence of toxic features at the regional pixel level.The filtering is intended to preserve safe concepts and desirable outputs while suppressing unsafe content.Source: §1 · p. 3
  • SAFREE achieves state-of-the-art performance on five T2I benchmarks and extends to diverse image backbones and T2V models.The authors also report greater efficiency, lower resource use, and better inference-time adaptability than other training-free safeguards.Source: §1 · p. 3
  • SAFREE jointly filters textual embeddings and visual latents, achieving competitive results against training-based methods while maintaining high visual quality.The contribution statement describes the approach as a training-free safeguard for both T2I and T2V generation.Source: §1 · p. 3

2 RELATED WORK

Prior work examines vulnerabilities in T2I models through adversarial prompts and red-teaming, while safety methods range from training-based removal to training-free filtering and editing. SAFREE is framed as an adaptive, weight-preserving alternative for image and video generation.

2.1 T2I ATTACKS

  • Adversarial prompt optimization and red-teaming frameworks expose vulnerabilities in T2I models, including through cross-modality jailbreaks.The cited work includes Ring-A-Bell and automated adversarial-prompt generation frameworks.Source: §2.1 · p. 3

2.2 SAFE T2I GENERATION

  • Training-based safety methods use removal, negative guidance, adversarial training, representation filtering, or fine-tuning, but can require extensive tuning and degrade image quality.The related-work discussion also identifies a lack of inference-time adaptation as a limitation.Source: §2.2 · pp. 3–4
  • SAFREE detects tokens near a toxic subspace, projects them away while retaining them in the input space, and adapts filtering strength to regulate latent re-attention.The framework caption describes applicability across image and video diffusion backbones.Source: Figure 2 · p. 4
  • Training-free methods include weight editing and non-weight prompt or guidance changes, but prior approaches are described as lacking robustness and test-time adaptation.SAFREE is presented as dynamically adapting filtering without weight edits and extending to other architectures and video tasks.Source: §2.2 · p. 4

3 SAFREE: TRAINING-FREE AND ADAPTIVE GUARD FOR SAFE TEXT-TO-IMAGE AND VIDEO GENERATION

SAFREE jointly filters toxic concepts in text embeddings and visual latent space without modifying model weights. Its pipeline detects and projects risky tokens, adapts filtering across denoising steps, and extends to advanced image and video models.

3.1 ADAPTIVE TOKEN SELECTION BASED ON TOXIC CONCEPT SUBSPACE PROXIMITY

  • SAFREE detects trigger tokens by comparing each token-masked pooled prompt embedding’s residual distance from the toxic concept subspace with the other-token average.Tokens whose distance exceeds the average by the sensitivity threshold are selected; experiments use α = 0.01.Source: §3.1 · p. 5

3.2 SAFE GENERATION VIA CONCEPT ORTHOGONAL TOKEN PROJECTION

  • Detected tokens are projected into the input space orthogonal to the toxic concept subspace, while all unselected tokens retain their original embeddings.This avoids deleting or replacing words that could disrupt prompt coherence and safe content.Source: §3.2 · pp. 5–6

3.3 ADAPTIVE CONTROL OF SAFEGUARD STRENGTHS WITH SELF-VALIDATING FILTERING

  • A self-validating mechanism adjusts how many denoising steps use filtered embeddings according to the cosine distance between the original and projected prompts.It applies stronger filtering to prompts deemed undesirable while approximating the original model’s generation for safe content.Source: §3.3 · p. 6

3.4 ADAPTIVE LATENT RE-ATTENTION IN FOURIER DOMAIN

  • During denoising, adaptive Fourier-domain re-attention attenuates selected low-frequency features of filtered latent representations to reduce oversmoothing while suppressing target-concept influence.After round(t′), the filtering equation leaves the original feature unchanged.Source: §3.4 · pp. 6–7

3.5 SAFREE FOR ADVANCED T2I MODELS AND TEXT-TO-VIDEO GENERATION

  • SAFREE extends in a zero-shot, training-free manner to SDXL, SD-v3, ZeroScopeT2V, and CogVideoX, across UNet and DiT backbones and T2I and T2V tasks.The paper reports qualitative and quantitative results across these backbones and tasks.Source: §3.5 · p. 7

4 EXPERIMENTAL RESULTS

SAFREE is evaluated across adversarial-prompt safeguards, artist-style removal, efficiency, and extensions to newer image and video models. The results report lower attack success rates than training-free baselines, preservation of untargeted styles, and generalization across backbones and applications.

4.1 EXPERIMENTAL SETUP

  • SAFREE was evaluated on SD-v1.4 with adversarial prompts from five red-teaming methods, on two artist-style removal datasets, and on ZeroScopeT2V and CogVideoX using UNet and Diffusion Transformer backbones.For quantitative evaluation, SafeSora used 600 toxic prompts across 12 concepts to construct a benchmark of 296 examples across 5 categories.Source: §4.1 · p. 7

4.2 BASELINES AND EVALUATION METRICS

  • Comparisons include training-free and training-based safeguards, with ASR, FID, CLIP score, TIFA, LPIPS, artist-identification accuracy, and binary T2V safety assessment.FID, CLIP score, and TIFA use 1,000 COCO-30k samples; T2V assessment uses 16 sampled frames per video.Source: §4.2 · p. 7

4.3 EVALUATING THE EFFECTIVENESS OF SAFREE

  • SAFREE achieves 47%, 13%, and 34% lower ASR than the best-performing counterparts I2P, MMA-diffusion, and UnlearnDiff, respectively, across all attack types.Source: §4.3 · p. 8

4.5 EFFICIENCY OF SAFREE

  • SAFREE requires no model editing and maintains competitive generation speeds, with the authors reporting the best overall performance in safeguarding, generation quality, and flexibility.The efficiency comparison uses one A6000, 100 steps, and a nudity-removal setting.Source: Table 3 · p. 8, §4.5 · pp. 9–10

4.3 EVALUATING THE EFFECTIVENESS OF SAFREE

  • Against training-based methods, SAFREE provides comparable safeguarding with high-quality COCO-30k images; examples remove Van Gogh style while retaining other artists’ styles.The figure also presents concept-removal comparisons and examples using different T2I and T2V models.Source: §4.3 · p. 8, Figure 3 · p. 9
  • Ablations report that orthogonal token projection preserves image quality better than null-token replacement, while self-validating filtering improves quality but can slightly reduce filtering capability.Combining token selection, self-validating filtering, and latent re-attention balances filtering with image quality for prompts unrelated to toxic concepts.Source: Table 2 · p. 8, §4.3 · p. 9

4.4 EVALUATING SAFREE ON ARTIST CONCEPT REMOVAL TASKS

  • SAFREE achieves higher LPIPSe and LPIPSu than baselines, and artist-identification evaluation indicates targeted artist concepts are removed more effectively.LPIPSe measures average LPIPS for erased target styles and LPIPSu for unerased styles; the table covers famous and modern artists.Source: §4.4 · p. 9, Table 4 · p. 10

4.6 GENERALIZATION AND EXTENSIBILITY OF SAFREE

  • SAFREE reduces unsafe outputs by 48% on SD-XL and 47% on SD-V3, and reduces unsafe concepts on ZeroScopeT2V and CogVideoX-5B.The authors describe these results as generalization across architectures and applications, with safe yet faithful outputs in the image and video examples.Source: Table 5 · p. 10, Table 6 · p. 10, §4.6 · p. 10

5 CONCLUSION

SAFREE is presented as a training-free approach to safe text-to-image and text-to-video generation that filters toxic concepts while preserving benign-request quality. The authors propose it as a strong baseline for further work on safer generative models.

5 CONCLUSION

  • SAFREE addresses unsafe image and video generation with a training-free method that does not alter model weights.It identifies a target-concept subspace and selectively removes critical prompt tokens that direct embeddings toward it.Source: §5 · p. 10
  • The method is reported to prevent unsafe content generation while preserving the quality of benign textual requests.Source: §5 · p. 10
  • The authors position SAFREE as a strong training-free baseline for research on safer text-to-image and text-to-video generation.Source: §5 · p. 10

ETHICS STATEMENT

The ethics statement identifies unsafe visual content as a concern in text-to-image and text-to-video generation. It links safety mechanisms with responsible deployment across creative, social, and educational contexts.

ETHICS STATEMENT

  • Unsafe or toxic visual content, including explicit or violent material, is an ethical concern for text-to-image and text-to-video models.Source: ETHICS STATEMENT · p. 11
  • The statement describes unlearning and filtering mechanisms as ways safe generation models address harmful associations and inappropriate content.Source: ETHICS STATEMENT · p. 11
  • Responsible use is framed as important across creative, social, and educational deployment contexts.Source: ETHICS STATEMENT · p. 11

REPRODUCIBILITY STATEMENT

The paper states that it discloses information needed to reproduce its main experimental results and provides code, hyperparameters, and model details. The appendix covers experimental details, additional analyses and visualizations, discussion, limitations, broader impact, and licensing.

REPRODUCIBILITY STATEMENT

  • The authors state that they disclose the information needed to reproduce the main experimental results and include code in the supplementary material.Hyperparameter settings and model details are reported in the appendix.Source: REPRODUCIBILITY STATEMENT · p. 11

APPENDIX

  • The appendix lists implementation and evaluation details, analyses of undesirable prompts and toxic-subspace distance, and further visualizations and generation results.Source: APPENDIX · p. 17
  • The appendix also includes discussion of image attribute control, limitations and broader impact, and license information.Source: APPENDIX · p. 17

A.1 EXPERIMENTAL SETUP

Experiments use SD-v1.4 for text-to-image evaluation on prompts from five red-teaming benchmarks, with separate artist-style removal datasets. Text-to-video evaluation uses two backbones and a selected SafeSora benchmark subset assessed with GPT-4o.

A.1 EXPERIMENTAL SETUP

  • Text-to-image safety is evaluated with SD-v1.4 on inappropriate or adversarial prompts from I2P, P4D, Ring-a-bell, MMA-Diffusion, and UnlearnDiff.Source: §A.1 · p. 17
  • Artist-style removal is tested on two datasets, each covering five artists whose styles are confirmed to be imitable by Stable Diffusion.The datasets include famous artists and modern artists.Source: §A.1 · p. 17
  • Text-to-video evaluation applies SAFREE to ZeroScopeT2V and CogVideoX using 296 examples selected from five categories of the 12-concept SafeSora set.GPT-4o performs binary safety checks on 16 sampled frames per video.Source: §A.1 · p. 17

A.2 BASELINES AND EVALUATION METRICS

The section specifies the comparison baselines and evaluation measures, then examines whether prompt toxicity tracks distance from the nudity concept subspace.

A.2 BASELINES AND EVALUATION METRICS

  • Baselines include training-free SLD/UCE/RECE variants and training-based ESD, SA, CA, MACE, and SDID.Source: §A.2 · p. 17
  • Safeguard capability is measured by ASR on adversarial nudity prompts; generation quality uses FID, CLIP score, and TIFA on COCO-30k.FID and TIFA are evaluated on 1,000 randomly selected samples.Source: §A.2 · p. 17
  • Artist-style removal uses LPIPS for perceptual difference and GPT-4o multiple-choice identification of the generated image’s artist among five candidates.Source: §A.2 · pp. 17–18

B ANALYSIS: CORRELATION BETWEEN Undesirable Prompts AND DISTANCE TO Toxic Concept Subspace.

C MORE QUALITATIVE VISUALIZATION

The qualitative appendix reports removal of targeted styles and unsafe concepts while retaining desirable content, and discusses how SAFREE differs from text-embedding attribute-control methods.

C MORE QUALITATIVE VISUALIZATION

  • Across the artist-style examples, SAFREE removes Van Gogh while retaining prompted subjects and other artist styles; CA and SLD fail to preserve some desirable styles.The appendix reports a similar conclusion for nudity removal examples.Source: C MORE QUALITATIVE VISUALIZATION · p. 18
  • Examples on SDXL and SD-v3, ZeroScopeT2V, and CogVideoX show filtering of user-defined unsafe concepts while preserving desirable content across T2I and T2V settings.The reported T2V examples include animal abuse, pornography, violence, and terrorism.Source: C MORE QUALITATIVE VISUALIZATION · p. 18

D EXTRA DISCUSSION WITH IMAGE ATTRIBUTE CONTROL WORKS

E LIMITATION & BROADER IMPACT

The paper identifies implicit, chain-of-thought toxic prompts as a remaining jailbreak risk and notes broader impacts associated with biases inherited from training data. It also lists community-standard licenses for datasets and models.

E LIMITATION & BROADER IMPACT

  • Implicit toxic prompts, especially chain-of-thought prompts, can still jailbreak SAFREE and produce unsafe or inappropriate content.The paper describes perfect safeguarding as an open problem for future study.Source: E LIMITATION & BROADER IMPACT · p. 19
  • Photorealistic image and video generation may inherit training-data biases, with societal stereotypes, biased interpretations of actions, and privacy concerns among the stated broader impacts.Source: E LIMITATION & BROADER IMPACT · p. 19

F LICENSE INFORMATION

  • The paper says it will release code and lists standard community licenses for its datasets and models, including CC-BY-NC-4.0 for ZeroScopeT2V.Source: F LICENSE INFORMATION · p. 19

Related papers

Loading 2410.12761v2…