PaperlayerView exact revision on arXiv

Source-linked AI summary

Discovering the Hidden Vocabulary of DALLE-2

Giannis Daras, Alexandros G. Dimakis

arXiv:2206.00169v1cs.LGcs.CLcs.CRcs.CV

TL;DR

The paper asks whether DALLE-2’s apparently nonsensical generated text contains a hidden vocabulary, a question relevant to understanding and securing generative models. Using black-box API queries, it extracts gibberish from generated images and feeds it back to the model. The authors find visual correspondences and occasional compositional or caption-aligned behavior, but with important inconsistency and security implications.

  • Problem

    DALLE-2 often produces gibberish when asked to generate text, raising whether that output reveals a hidden vocabulary and why such behavior matters for model security and interpretability.

    Source: §1 · p. 2, §4 · p. 5

  • Method

    The authors use query-only prompting to extract gibberish text from generated images and test those strings as new DALLE-2 prompts.

    Source: §2 · p. 2

  • Results

    Some gibberish terms correspond to visual concepts, including birds and sometimes bugs or pests, and can occasionally compose into related images.

    Source: §1 · p. 2, §3 · p. 3

  • Takeaways & Limitations

    DALLE-2’s hidden vocabulary creates interpretability and security challenges because absurd prompts can sometimes generate consistent, caption-related images or bypass filters.

    Source: §3 · p. 4, §4 · p. 5

    The discovered vocabulary is inconsistent: some prompts produce unrelated images, robust prompts are difficult to find, and success can depend on context.

    Source: §2 · p. 2, §4 · p. 5

Abstract

from arXiv · show

We discover that DALLE-2 seems to have a hidden vocabulary that can be used to generate images with absurd prompts. For example, it seems that \texttt{Apoploe vesrreaitais} means birds and \texttt{Contarra ccetnxniams luryca tanniounons} (sometimes) means bugs or pests. We find that these prompts are often consistent in isolation but also sometimes in combinations. We present our black-box method to discover words that seem random but have some correspondence to visual concepts. This creates important security and interpretability challenges.

1 Introduction

DALLE-2’s gibberish text can reveal an internally developed hidden vocabulary. Some discovered terms correspond to visual concepts and can generate natural images with absurd prompts.

  • DALLE-2’s generated gibberish is not always random, but can reveal a hidden vocabulary developed internally by the model.Source: §1 · p. 2
  • Apoploe vesrreaitais frequently produces airplanes when supplied as a prompt, illustrating correspondence between gibberish text and visual concepts.Source: §1 · p. 2
  • Apoploe vesrreaitais seems to mean birds, while Contarra ccetnxniams luryca tanniounons sometimes means bugs or pests.Source: §1 · p. 2
  • These learned terms can generate natural images with absurd prompts and can sometimes be composed, such as birds eating bugs.Source: §1 · p. 2

2 Discovering the DALLE-2 Vocabulary

The authors present a query-only method for discovering DALLE-2 vocabulary by extracting gibberish text from generated images and prompting the model with it. The method can reveal words correlated with visual concepts, but it is inconsistent and requires experimentation.

  • The method uses only API query access to discover DALLE-2 vocabulary through prompts designed to elicit written text in generated images.Source: §2 · p. 2
  • Prompts about books, conversations, or multilingual text usually produce images containing text that appears gibberish to humans.Source: §2 · p. 2
  • The authors parse generated text and feed it back to DALLE-2, finding correlations such as Vicootes with vegetables and Apoploe vesrreaitais with birds.Source: §2 · p. 2
  • The method does not always work because generated text can produce random images when prompted back to the model.Source: §2 · p. 2
  • Experimentation with word selection and multiple generated texts can usually identify random-looking words correlated with visual concepts in some contexts.Source: §2 · p. 2

3 A Preliminary Study of the Discovered Vocabulary

The preliminary study examines whether discovered vocabulary is compositional, style-flexible, and consistent with the captions that generated it. The results show these properties only inconsistently, though gibberish can sometimes preserve caption-related meaning.

Compositionality

  • Apoploe vesrreaitais and Contarra ccetnxniams luryca tanniounons sometimes compose into images of birds eating bugs.Source: §3 · p. 3

Method illustration

  • The method begins by querying DALLE-2 about a target concept and analyzing the gibberish text generated in the resulting images.Source: Figure 2 · p. 3

Style Transfer

  • The discovered concept associated with Apoploe vesrreaitais can remain something that flies across different visual styles, although it sometimes appears as insects rather than birds.Source: §3 · p. 3, Figure 3 · p. 4

Text’s consistency with the caption and the generated image

  • Gibberish text sometimes has meaning aligned with the text conditioning and generated image that originally produced it.Source: §3 · p. 4
  • For the whale-and-food example, feeding “Wa ch zod ahaakes rea.” back to the model produces seafood-related images.Source: Figure 4 · p. 4, §3 · p. 4

4 Security and Interpretability Challenges

The discovered vocabulary raises security and interpretability concerns because gibberish prompts can sometimes produce consistent images while remaining difficult to predict. Such prompts may bypass filters and expose weaknesses in generative-model behavior.

  • The origins and consistency of many gibberish words remain unclear, and the authors did not identify examples that were misspellings of ordinary words in other languages.Source: §4 · p. 5
  • The method’s robustness is uneven: the bugs-and-pests prompt works in about half of generated images, whereas Apoploe vesrreaitais is more robust and combinable.Source: §4 · p. 5
  • Finding robust prompts is challenging and requires substantial experimentation with generation methods, selected text, and consistency testing.Source: §4 · p. 5
  • Even rare, unexpected behaviors from gibberish prompts challenge confidence in large generative models and motivate more foundational research.Source: §4 · p. 5
  • Gibberish prompts may serve as backdoor adversarial attacks that circumvent text filters designed to block policy-violating prompts.Source: §4 · p. 5
Loading 2206.00169v1…