Source-linked AI summary
Adversarial Attacks on Image Generation With Made-Up Words
Raphaël Millière
TL;DR
Nonce strings may elicit specific imagery, but their associations and reliability are not well established. The paper introduces macaronic and evocative prompting to generate targeted images with made-up words; preliminary evidence suggests some attacks may work across models, while keyword blacklists are insufficient.
Problem
Gibberish prompts have been reported to elicit specific imagery, but their reliability and cross-model behavior remain unclear, leaving the relationship between tokenization and image generation insufficiently investigated.
Source: §1 · p. 2, §4 · p. 12Method
Results
Preliminary evidence suggests some text-based adversarial attacks may work somewhat reliably across models, and macaronic prompting is more viable than evocative prompting for malicious attacks.
Source: §4 · p. 11, §5 · p. 12Takeaways & Limitations
The findings highlight the insufficiency of keyword-based blacklists for content filtration in text-guided image generation models.
Source: §4 · p. 11Further research is needed to determine why models differ in their responses to macaronic and evocative prompting; evocative prompting is mostly limited to vague associations.
Source: §4 · pp. 11–12
Abstract
from arXiv · showhide
Text-guided image generation models can be prompted to generate images using nonce words adversarially designed to robustly evoke specific visual concepts. Two approaches for such generation are introduced: macaronic prompting, which involves designing cryptic hybrid words by concatenating subword units from different languages; and evocative prompting, which involves designing nonce words whose broad morphological features are similar enough to that of existing words to trigger robust visual associations. The two methods can also be combined to generate images associated with more specific visual concepts. The implications of these techniques for the circumvention of existing approaches to content moderation, and particularly the generation of offensive or harmful images, are discussed.
1 Introduction
Text-guided image generation raises misuse concerns, while nonce-string attacks remain poorly understood. The paper introduces macaronic and evocative prompting as ways to elicit visual concepts with seemingly nonsensical strings.
1 Introduction
- Adversarial attacks on text-guided image generation had not been thoroughly investigated, despite concerns about misuse and the possibility that nonce prompts could evade blacklisted-word filters.Earlier work suggested gibberish strings could elicit specific imagery, but did not establish a reliable way to find such strings.Source: §1 · p. 2
- The robustness of gibberish-string associations is unclear, and apparent bird prompts may partly reflect shared BPE subword tokens with Latin bird-family names.The “hidden vocabulary” account may be misleading, motivating investigation of tokenization and nonce-string attacks.Source: §1 · p. 2
- Macaronic prompting combines multilingual subword units, while evocative prompting uses invented words with morphological features that trigger associations with real concepts.The authors report macaronic prompting as more reliable than the earlier approach and say it can work across different models rather than being tailored to one.Source: §1 · pp. 2–3
- These methods raise content-moderation concerns because seemingly innocuous nonce strings may elicit targeted imagery and potentially avoid content filters.The broader motivation includes harmful or offensive generated content.Source: §1 · pp. 1, 3
2 Macaronic prompting
Macaronic prompting combines multilingual subword units into nonce strings that appear nonsensical to people but can evoke visual concepts in image-generation models. The prompts often produce consistent images and can transfer across models, though transfer failures occur; nonce strings can also be composed in sentences for more specific scenes.
2 Macaronic prompting
- Macaronic prompting combines multilingual subword units associated with similar visual concepts into nonce strings that obscure the intended concept from human readers.The example uccoisegeljaros combines chunks from German, Italian, French, and Spanish words for birds.Source: §2 · p. 3
- Lexical hybridization need not respect BPE subword boundaries: voiscellpajaraux and oisvogajaro also generate bird images.The authors describe carving and recomposing arbitrary chunks as effective.Source: §2 · p. 4
- Non-cherry-picked DALL-E 2 samples from additional hybridized prompts show high consistency despite the prompts' cryptic appearance to humans.Table 1 presents additional examples across visual domains.Source: §2 · p. 4, Table 1 · p. 4
- Some macaronic prompts work across DALL-E 2 and DALL-E mini despite differences in architecture, training data, and tokenization.The authors suggest macaronic prompting may apply to models trained on multilingual data, but note that transfer is not universal.Source: §2 · p. 4
- Transfer can fail: farpapmaripterling yields butterflies with DALL-E 2 but does not produce its intended result with DALL-E mini.Other examples include prompts that generate houses instead of birds or lizards with DALL-E mini.Source: §2 · pp. 4, 6
- Macaronic strings can be placed in English sentences to generate more specific scenes, while the intended concepts remain relatively obscured.English syntax acts as a scaffold; the authors note that such prompts may still avoid blacklist-based content filters when censored concepts are encoded.Source: §2 · p. 6
- Single-language portmanteau prompts also work, but typically reveal more about the intended visual concepts than multilingual hybridization.The paper gives creepooky, happeerful, and lovssionate as examples of invented English portmanteau words.Source: §2 · pp. 6–7
3 Evocative prompting
Evocative prompting uses nonce words whose broad morphological features resemble existing words or domain-specific forms, rather than requiring meaningful word chunks. Examples span biological and medicine-like names, language-linked places, and combinations with real word chunks that target more specific visual features.
3 Evocative prompting
- Evocative prompts rely on broad morphological similarity to words, word categories, or language-specific features, rather than meaningful parts of existing words.The text suggests associations may be mediated by statistically salient letter combinations within a domain.Source: §3 · p. 8
- Pseudolatinate forms can evoke organisms, while nonce strings resembling medicine names produce medicine-like prompting examples.Figures 8 and 9 show examples across models and with prompts including vacyloraxin and walbotricypofen.Source: §3 · p. 8, Figure 8 · p. 8, Figure 9 · p. 9
- Geography-like nonce names evoke location-associated imagery, including German or Austrian villages, old Italian towns, and historic French towns.DALL-E mini instead associates Beaussoncour with 17th-century French portraits, while retaining an association with French culture.Source: §3 · pp. 8–9, Figure 10 · p. 9
- Models may interpret hybrid-looking prompts through local word chunks or overall morphology, and their responses can differ with training data, size, and architecture.The paper notes that some French-looking prompts can be understood in either a macaronic or evocative fashion across models.Source: §3 · p. 9
- Combining real English word chunks with pseudolatinate forms can target more specific animal features, such as scary, cute, aquatic, or flying.Examples include scariosus ferocianensis for creepy crawlies and watosus swimensis for aquatic animals.Source: §3 · p. 10, Figure 11 · p. 10
4 Discussion
The discussion presents macaronic and evocative prompting as targeted adversarial methods that may circumvent text-side filters, while emphasizing differences in reliability and interpretability. It also outlines potential defenses and unresolved questions about how prompts behave across models.
4 Discussion
- Macaronic prompts combine word chunks from different languages into human-cryptic nonce words that can consistently evoke specific imagery, with preliminary experiments suggesting broad targeting potential.The authors warn that such prompts could bypass blacklisted-prompt filters and elicit harmful or sensitive images.Source: §4 · pp. 10–11
- Evocative prompting lacks a principled way to craft reliable, specific associations, and its proof-of-concept domains are biology, medicine, and language-linked geography or culture.Combining it with real word chunks can improve specificity, but the resulting prompts are easier for people and blacklists to interpret.Source: §4 · p. 11
- The authors judge macaronic prompting more viable than evocative prompting for malicious attacks and argue keyword blacklists are insufficient.Possible complements include filtering generated images or retrieving related linguistic concepts; filtering nonce words wholesale could impede proper names and typo resilience.Source: §4 · p. 11
- These prompts are not completely inscrutable to humans, and the tested macaronic hybrids use only English, French, German, Italian, and Spanish.The authors suggest adding lesser-known languages could make strings more cryptic, while noting model responses are difficult to predict and explain.Source: §4 · pp. 11–12
5 Conclusion
The conclusion reports preliminary evidence that some text-based adversarial prompts work somewhat reliably across models, while model properties may influence vulnerability. It calls for further research on response differences and mitigation.
5 Conclusion
- Preliminary evidence suggests some macaronic and evocative attacks work somewhat reliably across models, although size, architecture, tokenization, and training data may influence vulnerability.The authors call for further study of model response factors and ways to mitigate malicious image generation.Source: §5 · p. 12