PaperlayerView exact revision on arXiv

Source-linked AI summary

Discovering Universal Semantic Triggers for Text-to-Image Synthesis

Shengfang Zhai, Weilong Wang, Jiajun Li, Yinpeng Dong, Hang Su, Qingni Shen

arXiv:2402.07562v1cs.CRcs.AI

TL;DR

Text-to-image models’ robustness and ethical risks remain incompletely explored, while hidden-vocabulary discovery has relied on empirical construction. The paper proposes SGS to find Universal Semantic Triggers and SemSR to evaluate semantic shifts, and reports vulnerability across mainstream models and online services.

  • Problem

    The robustness and potential ethical issues of text-to-image models are not fully explored, and existing hidden-vocabulary research relies on empirical construction and lacks efficient automation.

    Source: §1 · p. 1

  • Method

    The paper introduces SGS to discover target-conditioned Universal Semantic Triggers and SemSR to quantitatively evaluate their semantic shifts.

    Source: §1 · p. 2, §5.1 · p. 4

  • Results

    Experiments show wide vulnerability to the triggers across mainstream text-to-image models and online services.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors aim to increase awareness of this vulnerability and help users audit models before deployment.

    Source: §1 · p. 2, §7 · p. 6

    The threat model assumes adversaries have white-box access only to the text encoder and lack knowledge of other model components and the test dataset.

    Source: §3.2 · p. 3

Abstract

from arXiv · show

Recently text-to-image models have gained widespread attention in the community due to their controllable and high-quality generation ability. However, the robustness of such models and their potential ethical issues have not been fully explored. In this paper, we introduce Universal Semantic Trigger, a meaningless token sequence that can be added at any location within the input text yet can induce generated images towards a preset semantic target.To thoroughly investigate it, we propose Semantic Gradient-based Search (SGS) framework. SGS automatically discovers the potential universal semantic triggers based on the given semantic targets. Furthermore, we design evaluation metrics to comprehensively evaluate semantic shift of images caused by these triggers. And our empirical analyses reveal that the mainstream open-source text-to-image models are vulnerable to our triggers, which could pose significant ethical threats. Our work contributes to a further understanding of text-to-image synthesis and helps users to automatically auditing their models before deployment.

1. Introduction

The paper addresses gaps in text-to-image safety and robustness research, especially the limited automation of hidden-vocabulary discovery and evaluation of semantic shifts. It introduces Universal Semantic Triggers, SGS, and SemSR, and reports vulnerability across mainstream models and online services.

Motivation

  • Existing safeguards cannot eliminate all threats: filtering harmless concepts can compromise generation, while adversarial attacks can bypass text filters.The paper therefore calls for studying model safety and robustness across varied inputs.Source: §1 · p. 1

Research gap

  • Prior hidden-vocabulary research relied on empirical construction and lacked efficient automation, motivating automated discovery of meaningless sequences that target specific semantics.The paper also identifies quantitative evaluation of semantic shifts as an open question because attack success rates may not capture them fully.Source: §1 · pp. 1–2

Approach

  • SGS searches for triggers based on target semantics, while SemSR measures their semantic triggering effect using semantic-space similarity.The paper also analyzes trigger characteristics and considers harmful, sensitive, and harmless target semantics.Source: §1 · p. 2

Findings

  • Experiments on mainstream text-to-image models and online services show wide vulnerability to the proposed triggers.The authors frame this finding as a reason to raise user awareness and support automated audits before deployment.Source: §1 · p. 2

Ethical considerations

  • The authors report that triggers may be harmful, benign, or threatening in inappropriate contexts, and state that potentially harmful outputs were blurred.They also report informing user-study evaluators about possible harmful content and responsibly disclosing findings to online service providers.Source: §1 · p. 2

2. Background

Text-to-image systems use textual descriptions to guide image synthesis. The background describes text and image encoders, with CLIP serving as a cross-modal embedding model used in text-to-image systems.

2.1. Text-to-Image Synthesis

  • Text-to-image models encode a prompt and use it to condition an image-generation component that synthesizes an image.For Stable Diffusion, generation starts from latent noise, which becomes a text-conditioned latent image embedding before decoding.Source: §2.1 · p. 2
  • The paper focuses primarily on CLIP text encoders because they are widely used, while stating that SGS can be applied to other text encoders with minimal changes.Source: §2.1 · p. 2

2.2. CLIP

  • CLIP maps text and images into a cross-modal embedding space, and its text embeddings can guide text-to-image synthesis.The passage reports that CLIP was trained on 400M text-image pairs using contrastive loss.Source: §2.2 · p. 2

3. Problem Formulation

The formulation treats a Universal Semantic Trigger as a fixed-length, seemingly meaningless token sequence inserted into a text input to shift generated images toward target semantics. The threat model specifies a misuse scenario and an adversary with limited white-box access to the text encoder.

3.1. Triggers Formulation

  • Universal Semantic Triggers are fixed-length, seemingly meaningless token sequences that can be inserted into an input text.The formulation represents the input as X and the trigger as T_k.Source: §3.1 · p. 2
  • The insertion operator places the trigger at a location m ranging from 0 to n in the input text.The formulation distinguishes images generated from the original and trigger-inserted texts and relates them to target semantics.Source: §3.1 · p. 2
  • Table 1 identifies text encoders used by mainstream text-to-image models and notes that their open-source availability makes model structures and parameters easy to obtain.Source: Table 1 · p. 3

3.2. Threat model

  • The paper treats triggers as an attack for exploring potential risks, while noting that they are not inherently harmful.Source: §3.2 · p. 3
  • The scenario assumes malicious users insert triggers into local models or online applications to bypass text filters or other security mechanisms and generate potentially harmful images.The described downstream risk is that such images may spread online and cause negative societal impacts.Source: §3.2 · p. 3
  • The adversary ideally seeks token sequences that appear meaningless to humans while encoding semantics they preset.Source: §3.2 · p. 3
  • The threat model assumes white-box access only to the text encoder, with other model components and the test dataset unknown to the adversary.The authors also report black-box online-service testing that demonstrates a certain degree of transferability.Source: §3.2 · p. 3

4. Design

Section 4 presents SGS as an automated framework for discovering triggers from target semantics, using target-text construction and gradient-based optimization in the text encoder's embedding space.

4. Design

  • SGS constructs target-semantic text, then searches a filtered vocabulary with a text-encoder loss to produce semantically inconspicuous triggers.Source: §4 · p. 3

4.1. Target Semantics

  • Targets cover harmful, sensitive, and harmless semantics; target sentences use insertion, substitution, or prefix/suffix modifications, with seven targets overall.Source: §4.1 · p. 3, Table 2 · p. 3

4.2. Loss Function

  • The loss brings triggered-text embeddings close to embeddings of explicit target-semantic sentences using cosine distance.Source: §4.2 · p. 3

4.3. Gradient-based Search

  • Greedy gradient search excludes tokens explicitly associated with the target, initializes 3–5-token triggers randomly, and iteratively replaces tokens along gradient-descent directions.Source: §4.3 · p. 4
  • Multiple epochs address local optima by selecting the trigger with lowest accumulated loss; beam search and projected gradient descent did not significantly improve results, while the current method converged faster.Source: §4.3 · p. 4

5. Evaluation

The evaluation introduces SemSR to quantify image-semantic shifts, then tests trigger effectiveness, insertion-position transfer, SaaS behavior, and combined triggers. Results include semantic shifts versus random triggers, transfer across insertion locations, and examples of risks in online services and with trigger ensembles.

5.1. SemSR Metric

  • SemSR quantifies semantic shift using CLIP image–text similarity, comparing triggered images with benign outputs and normalizing by an upper-limit offset.The upper limit is represented by images generated from explicit semantic sentences.Source: §5.1 · p. 4

5.2. Experimental Setup

  • Experiments use Stable Diffusion v1.4 and CLIP ViT-L, with 1,000 CC 12M training texts and 1,000 MS-COCO test texts; triggers are searched at the prefix.Trigger lengths are 2, 3, or 4 tokens; training uses batch size 32 for 20 epochs.Source: §5.2 · pp. 4–5

5.3. Main Results

  • Universal triggers shift image semantics compared with random-token triggers, and some approach explicit semantic text; style targets are easier than specific figures.Longer triggers tend to be more effective, and length-4 triggers receive user-study scores similar to explicit semantic text.Source: §5.3 · p. 5, Table 3 · p. 5, Table 4 · p. 5

5.4. Insertion Position Transferability

  • Triggers remain effective when tested at positions different from their generation position; prefix and middle insertions tend to have greater impact.The authors suggest that proximity to the object whose semantics are altered may improve results.Source: §5.4 · pp. 5–6, Figure 2 · p. 5

5.5. Effectiveness on SaaS Scenarios

  • On Midjourney, v5 and v6 show partial resistance to certain triggers, but the authors report inadequate defenses overall against diverse trigger words.The tested triggers include two harmful semantic types and one sensitive type; examples cover three Midjourney versions.Source: §5.5 · p. 6, Figure 3 · p. 6

5.6. Ensemble Triggers

  • Preliminary ensemble tests combine two semantic triggers at a sentence’s beginning and end, with examples that force models to generate malicious images.The authors characterize the input-agnostic, stackable triggers as more like a semantic hidden vocabulary than conventional adversarial attacks.Source: §5.6 · p. 6, Figure 4 · p. 6

6. Related Work

Prior work studies hidden vocabularies and adversarial attacks on text-to-image models, while this paper contrasts its automated approach with methods relying on manual analysis or reinforcement learning.

6. Related Work

  • Hidden-vocabulary methods include manually analyzed prompting approaches and special characters used to guide semantic styles [22], [23],.Source: §6 · p. 6
  • Prior work includes black-box and white-box attacks by Gao et al. [19] and Zhuang et al. that degrade generated-image quality, while Rando et al. [11] red-team Stable Diffusion’s safety filter.Source: §6 · p. 6
  • Compared with reinforcement-learning-based safety-filter auditing [21], the authors describe their method as simpler and more efficient for user self-auditing before deployment.Source: §6 · p. 6

7. Conclusion

The paper introduces Universal Semantic Triggers and SGS for automated discovery, with evaluation metrics for trigger effectiveness. Experiments indicate the issue spans text-to-image models and online SaaS services, and the work aims to support pre-deployment auditing.

7. Conclusion

  • The paper proposes Universal Semantic Triggers and SGS to search for them automatically.Source: §7 · p. 6
  • The authors devise metrics to evaluate the effectiveness of Universal Semantic Triggers.Source: §7 · p. 6
  • Experiments show the issue is widespread across various text-to-image models and online SaaS services, with the stated aim of helping users audit models before deployment.Source: §7 · p. 6

Appendix A. Experiments on Other Text-to-image Models

The appendix tests SGS beyond SD v1.4 on models with different structures and parameters, including CLIP-based SD variants and a BERT-encoded latent diffusion model; Figure 5 presents examples of trigger universality.

Appendix A. Experiments on Other Text-to-image Models

  • Examples in Figure 5 demonstrate the method's universality on other text-to-image models.Source: Figure 5 · p. 9, Appendix A · p. 9
  • The additional tested models are SDXL v1.0 and SD v2.1, both using CLIP as the text encoder, like SD v1.4.Source: Appendix A · p. 9
  • Testing also includes a latent diffusion model that uses BERT as its text encoder.Source: Appendix A · p. 9

Related papers

Loading 2402.07562v1…