PaperlayerView exact revision on arXiv

Source-linked AI summary

Mass-Producing Failures of Multimodal Systems with Language Models

Shengbang Tong, Erik Jones, Jacob Steinhardt

arXiv:2306.12105v2cs.LGcs.CLcs.SE

TL;DR

Evaluators struggle to anticipate all failures of deployed multimodal systems. MULTIMON mines erroneous agreement, uses language models to describe and generate systematic failures, and finds failures that transfer across multimodal systems. Its generated pairs produce errors on 80.0% of five text-to-image systems’ pairs, compared with 20.5% for a baseline.

  • Problem

    Evaluators struggle to anticipate and test all possible failures of text-based multimodal systems before deployment.

    Source: §1 · p. 1

  • Method

    MULTIMON scrapes semantically different inputs with similar CLIP embeddings, uses language models to describe systematic failure patterns, and generates new instances.

    Source: §1 · pp. 1–2

  • Results

    Failures transfer across text-to-image, text-to-video, and text-to-3D systems; generated pairs cause errors on 80.0% of five text-to-image systems’ pairs, versus 20.5% for a baseline.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors present the pipeline as a simple, general foundation for broader automatic evaluation that explores potential system failures.

    Source: §1 · p. 2

    The approach is especially suited to multimodal systems with embeddings; applying it to language models or classifiers would require efficient approximations of erroneous agreement.

    Source: §8 · p. 13

Abstract

from arXiv · show

Deployed multimodal systems can fail in ways that evaluators did not anticipate. In order to find these failures before deployment, we introduce MultiMon, a system that automatically identifies systematic failures -- generalizable, natural-language descriptions of patterns of model failures. To uncover systematic failures, MultiMon scrapes a corpus for examples of erroneous agreement: inputs that produce the same output, but should not. It then prompts a language model (e.g., GPT-4) to find systematic patterns of failure and describe them in natural language. We use MultiMon to find 14 systematic failures (e.g., "ignores quantifiers") of the CLIP text-encoder, each comprising hundreds of distinct inputs (e.g., "a shelf with a few/many books"). Because CLIP is the backbone for most state-of-the-art multimodal systems, these inputs produce failures in Midjourney 5.1, DALL-E, VideoFusion, and others. MultiMon can also steer towards failures relevant to specific use cases, such as self-driving cars. We see MultiMon as a step towards evaluation that autonomously explores the long tail of potential system failures. Code for MULTIMON is available at https://github.com/tsb0601/MultiMon.

1 Introduction

Evaluators struggle to anticipate all deployment failures, motivating systematic, human-compatible evaluations that explore possible behaviors and describe steerable patterns. MULTIMON mines erroneous agreement, uses language models to describe and generate failures, and finds failures that transfer across multimodal systems.

1 Introduction

  • The evaluation goal is to explore long-tail behaviors without prespecifying what to look for or depending on human labor, while describing patterns and allowing steering toward designers’ goals.Source: §1 · p. 1
  • MULTIMON uses large language models to categorize scraped individual failures into natural-language patterns, then generates novel instances that can be steered toward specific goals.It scrapes a corpus for inputs with similar outputs but different semantics, using CLIP embeddings to test output similarity.Source: §1 · pp. 1–2
  • MULTIMON uncovers 14 systematic CLIP text-encoder failures and generates over one thousand individual failures, with 12 descriptions succeeding at least half the time and 7 at least 75% of the time.Failures include negation, spatial and numerical differences, role ambiguity, and quantifiers.Source: §1 · p. 2
  • Across five text-to-image systems, MULTIMON-generated pairs produce errors on 80.0% of pairs, versus 20.5% for a baseline system.The failures also transfer to text-to-video and text-to-3D systems.Source: §1 · p. 2

2 Related Work

Related work studies embedding-model failures, systematic-failure discovery, automated generation of individual failures, and language models’ ability to infer patterns from examples. MULTIMON combines corpus-based failure discovery with language-model categorization and generation.

Embedding-model failures

Automated individual failures

  • Song et al. [2020] adversarially constructs input pairs that embeddings should distinguish but encode similarly, potentially replacing MULTIMON’s corpus-scraping step.Source: §2 · p. 3

Systematic failures

  • Classifier-slice methods use ground-truth labels and test predefined descriptions, whereas MULTIMON does neither and instead generates candidate systematic failures [Eyuboglu et al., 2022, Jain et al., 2022, Gao et al., 2022, Wiles et al., 2022, Metzen et al., 2023, Zhang et al., 2023].Source: §2 · p. 3
  • Human-conjectured failures and holistic evaluations cover generative-system behaviors, while Jones and Steinhardt [2022] and Perez et al. [2022b] use cognitive biases or language models to test conjectured code-model failures.Nushi et al. [2018] develop a system to help humans identify systematic failures and test it on a captioning system.Source: §2 · p. 3

Automated individual failures

  • Other individual-failure methods use specifications, model fine-tuning, or discrete optimization, while corpus scraping has also been used to find text preceding toxic outputs [Perez et al., 2022a; Jones et al., 2023; Wen et al., 2023; Gehman et al., 2020].Source: §2 · p. 3

Language models drawing conclusions from instances

  • Large autoregressive language models have been used to infer distribution differences, explain prediction patterns, and interpret neurons; Zhong et al. [2023] is closest to MULTIMON’s pattern-categorization step.Source: §2 · p. 3

3 The MULTIMON Pipeline

MULTIMON scrapes semantically different inputs with similar CLIP embeddings, asks a language model to describe recurring failure patterns, and generates new candidate failures. Its pipeline can be steered by selecting corpus examples or conditioning generation, and is evaluated for quantity, quality, and relevance.

3 The MULTIMON Pipeline

  • The pipeline moves from corpus sentences to individual failures, then language-model categories, then generated failures that can produce incorrect images.Source: §3 · p. 3, Figure 2 · p. 4

3.1 Constructing MULTIMON

  • MULTIMON defines systematic failures as generalizable natural-language patterns, categorizes scraped examples with a language model, and generates new candidate pairs from those descriptions.For scraping, it selects pairs with high CLIP cosine similarity but below-threshold DistilRoBERTa similarity, avoiding expensive decoder runs.Source: §3.1 · pp. 4–5

3.2 Steering MULTIMON

  • Steering filters corpus pairs through a classifier for the target domain before categorization, or prompts generation for domain-relevant examples using existing failure descriptions.The two approaches serve different needs: corpus-based steering suits domains represented in the corpus, while generation steering can target out-of-distribution domains.Source: §3.2 · p. 5

3.3 Evaluating MULTIMON

  • MULTIMON evaluates quantity by counting systematic failures, quality by the success rate of generated pairs above a CLIP-similarity threshold, and relevance by the fraction classified as domain-relevant.Source: §3.3 · pp. 5–6

4 Automatically Finding Failures of CLIP

MULTIMON identifies CLIP text-encoder failures and produces instances that transfer to downstream systems; its findings vary with the language model and corpus. Steering yields self-driving-related failures and also generates relevant examples for a domain absent from the original corpora.

4.1 Identifying systematic failures of CLIP with MULTIMON

  • Across two corpora, GPT-4 identified 14 systematic failures, Claude found 11, and GPT-3.5 found 8.Source: §4.1 · p. 6
  • The corpus changes which failures are found, and MULTIMON autonomously recovers known failures while uncovering additional ones.Only COCO uncovers temporal differences, whereas only SNLI uncovers numerical differences.Source: §4.1 · p. 7
  • For systematic failures found by all three models, average success rates were 80.2% for GPT-4, 83.3% for Claude, and 69.5% for GPT-3.5.Source: §4.1 · p. 7
  • Without scraped examples, language models identify only 2 of the 14 failures, with an average success rate of 29.3%.Source: §4.1 · p. 7
  • When GPT-4 is replaced as the generator, Claude produces similar average success rates with variability across failures, while GPT-3.5 performs worse.Source: §4.1 · p. 7

4.2 Steering MULTIMON towards specific applications

  • Self-driving steering produces five systematic failures, four with success rates above 95%, and all five have relevance rates above 90%.Four of the five have a 100% relevance rate.Source: §4.2 · p. 8
  • For self-driving individual failures, 74.6% of generated pairs succeed and 95.0% are relevant; steering also yields Pokémon Go examples with 66.9% average success and 82.5% relevance.Source: §4.2 · p. 8

5 Failures of CLIP lead to Failures Downstream

MULTIMON’s CLIP failures also appear in downstream image, video, and 3D systems. Manual evaluation and qualitative examples show failures across tested systems, including failures steered toward self-driving scenarios.

5.1 Manually evaluating generated images

  • 80% of MULTIMON-generated pairs were downstream failures, compared with 20% of baseline pairs, in a manual evaluation of four text-to-image systems.A pair counted as a downstream failure if at least one generated image was judged incorrect; the evaluation used 100 pairs from each system.Source: §5.1 · pp. 9–10

5.2 Qualitative examples on state-of-the-art multimodal models

  • MULTIMON failures appeared across tested text-to-image systems, including Stable Diffusion XL, Stable Diffusion 2.1, Midjourney 5.1, and DALL-E.The authors report that these results show the same inputs produce failures across the tested systems, which rely on CLIP embeddings.Source: §5.2 · p. 10
  • Examples also show failures in Shap-E’s numerical quantity handling and VideoFusion’s representation of action states, including a moving turbine prompted to be at rest.The authors note that the turbine prompts could be visually identical in a static scene but are semantically distinct in video.Source: §5.2 · p. 10
  • Steering MULTIMON toward self-driving surfaced incorrect car positions and colors, a stop/yield sign mix-up, and a video of a car running a red light.The paper argues these examples would have been challenging to uncover without steering toward the target subdomain.Source: Figure 6 · p. 10, §5.2 · p. 11

6 Extending MULTIMON Beyond CLIP

The authors test whether CLIP-derived failures transfer to systems with other embedding models, and run MULTIMON directly with T5 to find additional failures. They report transfer to DeepFloyd and DALL-E 3, as well as systematic failures specific to T5.

Transferring failures

  • 70.8% of tested inputs produced downstream failures on DeepFloyd, and 69.3% did so on DALL-E 3, suggesting that some failures transfer across embedding models.The transfer test manually assessed 27 inputs from the initial June 2023 arXiv version of the paper.Source: §6 · p. 11

Finding failures directly

  • Running MULTIMON with T5 found eight systematic failures, with an average success rate of 77.3%.The pipeline used GPT-4 for categorization and generation after substituting T5 for CLIP at each step.Source: §6 · p. 11
  • T5-specific failures included pronoun ambiguity and temporal differences, which produced downstream failures in T5-based systems but not CLIP-based systems.Figure 8 contrasts outputs from T5-based DeepFloyd with those from CLIP-based Stable Diffusion XL.Source: §6 · p. 12, Figure 8 · p. 12

7 Evaluating Safety Filters with MULTIMON

The authors combine systematic failures to probe a text-to-image safety filter, finding prompts that pass the filter but produce gory outputs. They frame such evaluations as useful for finding vulnerabilities, while noting the risk that disclosed failures can be exploited.

7 Evaluating Safety Filters with MULTIMON

  • Combining negation and temporal-difference failures yielded examples that bypassed Midjourney’s safety filter with less than 10 minutes of human labor.The prompts were written to be semantically safe and unfiltered, yet produce gory outputs.Source: §7 · p. 12
  • The study presents systematic failures as a way for human evaluators to find vulnerabilities they might otherwise miss, even in hardened systems.The tested filter aims to block some inputs automatically to prevent visually shocking or disturbing content.Source: §7 · p. 12
  • The authors identify a disclosure risk: evaluation results expose deployed-model failures that adversaries could exploit.For this study, they evaluated a closed-source system that could be updated and sent Midjourney their results before publication.Source: §7 · p. 12

8 Discussion

The discussion presents MULTIMON as a way to find failures that generalize across text-guided multimodal systems, while identifying pipeline improvements and broader evaluation challenges. The supplied appendix passages also describe its pseudocode, prompts, corpus counts, and examples of categorized failure types.

8 Discussion

  • MULTIMON scrapes failures through erroneous agreement, then uses language models to categorize and generate new failures that generalize across text-to-image, text-to-video, and text-to-3D systems.Source: §8 · p. 13
  • The pipeline can miss failures its scraping stage does not catch, and efficient erroneous-agreement methods for systems such as language models or classifiers remain future work.The authors also note that language-model improvements can plug into the existing pipeline.Source: §8 · p. 13
  • Reused components such as CLIP may carry homogeneous errors across models, while evaluating complex outputs can be time-consuming and difficult for people to anticipate.The authors argue that scalable evaluation should improve with existing scaling trends and avoid bottlenecks from human ingenuity.Source: §8 · p. 13

A.1 Pseudo code of scraping and categorizing for MULTIMON

  • The scraping pseudocode pairs corpus inputs whose semantic-encoder cosine similarity meets a threshold, optionally filters for a steering direction, then categorizes the pairs.Source: §A.1 · p. 18

B Prompts Used in MULTIMON

  • The categorization prompt asks a language model to generalize failure types from example pairs, while the generation prompt requests additional pairs matching a selected failure mode.The generation procedure can continue in the same session to produce more instances.Source: §B.1 · p. 18, §B.2 · p. 19

C.1 The number of erroneous agreements in each corpus

  • The authors report scraping 33,922 erroneous-agreement pairs from SNLI and 2,131,440 from MS-COCO, using 157,351 and 616,767 examples, respectively.They note that possible pairs scale quadratically with corpus size.Source: §C.1 · p. 19

C.2 Description of systematic failures

  • Using MS-COCO and SNLI, GPT-4 categorized systematic failures including attribute differences (such as color or size), numerical differences, action-state differences, negation, and spatial relations.Source: §C.2 · pp. 19–20

C.3 Ablation study on using different corpus and LLM

The ablations compare language-model and corpus choices for identifying systematic failures, and compare the quality of descriptions produced by different language models. Claude’s results are comparable to GPT-4’s in one comparison, while GPT-3.5 generally performs worse.

C.3 Ablation study on using different corpus and LLM

  • Claude produces quality comparable to GPT-4 despite identifying fewer systematic failures, while GPT-3.5 lags behind.The comparison reports mean CLIP similarity, standard deviation, and success rate for each language-model–corpus pair.Source: §C.3 · p. 21, Table 2 · p. 22
  • Most systematic failures in the generated-instance analysis produce high-quality instances, across both plotted success-rate groups.The groups separate systematic failures with success rates below and above 80%.Source: §C.3 · p. 22, Figure 9 · p. 22

C.4 Ablation study on description using LLM

  • GPT-4 and Claude produce equally good descriptions for the five shared systematic failures, while GPT-3.5’s descriptions are slightly worse.GPT-4 generates the individual failures used to compare the descriptions.Source: §C.4 · p. 22, Table 3 · p. 23

C.5 Ablation study on using different LLM as generator

The ablations compare generators and corpus use, then examine steering toward particular domains. Steering generation toward self-driving cars yields high success and relevance rates, while steering can also target Pokemon Go and manual evaluation compares MULTIMON examples with a baseline.

C.5 Ablation study on using different LLM as generator

  • GPT-4 and Claude are good generators of instances from GPT-4-categorized failures, whereas GPT-3.5 is less competent.The study uses the first seven systematic failures categorized by GPT-4.Source: §C.5 · p. 23, Table 4 · p. 23
  • Better generators may make MULTIMON’s measured success rate an underestimate because they may follow systematic-failure descriptions more faithfully.The authors suggest more reliable generation can produce more pairs that contain failures.Source: §C.5 · p. 23

C.6 Ablation study on no corpus

  • Without corpus examples, GPT-4’s two identified failure types—homonyms and subtle differences—yield an average success rate of 29.3.The authors take this as evidence that corpus data is important when generating systematic failures.Source: §C.6 · p. 23, Table 5 · p. 23

C.7 Steering MULTIMON

  • Steering scraping toward self-driving cars identifies five relevant failure patterns, including negation, temporal, quantitative, spatial, and object-attribute distinctions.The examples include differences such as a car stopping versus not stopping and a pedestrian versus a cyclist crossing.Source: §C.7 · p. 24
  • 74.56% average success rate and 95.01% average relevance rate are obtained by steering generation toward self-driving cars.The systematic failures were identified using normal corpus data; the generation prompt was then modified to put examples in a self-driving context.Source: §C.7 · p. 24, Table 7 · p. 25
  • Steering also produces failures involving concepts beyond the original corpus, with Pokemon Go given as an example.The reported measures include success and relevance rates, but the supplied passage gives no values.Source: §C.7 · p. 25, Table 8 · p. 25

D.1 Additional manual study details

  • The manual study compares 100 MULTIMON pairs with 100 baseline pairs across four text-to-image models, asking annotators about prompt attribution and visual identity.Two authors labeled all 400 images, and their labels were added together.Source: §D.1 · pp. 25–26

D.2 Additional manual evaluation results

Manual evaluation finds frequent downstream failures among pairs with high CLIP similarity, while most generated prompts remain suitable for evaluation. Additional examples span text-to-image, text-to-3D, and varied individual failure types.

D.2 Additional manual evaluation results

  • Over 65% of pairs around 0.88 CLIP similarity were downstream failures, and annotator mistakes exceeded 60% for pairs above 0.88.The manual evaluation also reports that around 9% of generated prompts were visually identical, with the majority suitable as failure examples.Source: §D.2 · pp. 26–27, Figure 12 · p. 27
  • The 0.88 similarity threshold is model-dependent; non-identical CLIP embeddings could, in principle, still yield separate images downstream.The authors select the threshold because failures rise abruptly around that value.Source: §D.2 · p. 27

D.3 Additional results on text-to-image models

  • Additional examples show MULTIMON-generated failures applied to MidJourney 5.1, DALL-E, Stable Diffusion XL, and Stable Diffusion 2.1.The appendix also provides further generated text-to-image examples.Source: §D.3 · p. 27, Figure 13 · p. 28

D.4 Additional results on text-to-3D models

  • The appendix provides further examples of MULTIMON-generated failures applied to text-to-3D models.Figure captions identify additional text-to-3D examples, including errors in Shap-from inputs.Source: §D.4 · p. 27, Figure 14 · p. 28

D.5 Additional results on the individual failures generated by MULTIMON

  • Generated individual failures include contrasts in quantity, agent roles, negation, and timing.Examples include two versus four children, a man versus a woman teaching, a landscape with versus without a river, and a child opening versus about to open a present.Source: §D.5 · p. 29

E Automatically Finding Failures of New Embedding Models

E.1 Using MULTIMON to find failures of new embeddings directly

Running MULTIMON with T5 finds systematic embedding failures, including some not found in CLIP. The generated inputs also produce failures in DeepFloyd when it uses the T5 encoder.

E.1 Using MULTIMON to find failures of new embeddings directly

  • The authors repeat the MULTIMON pipeline with T5 in the scraping step while keeping categorization and generation the same.Source: §E.1 · p. 30
  • MULTIMON finds eight systematic failures of the T5 encoder using GPT-4 as categorizer and generator.Examples include temporal differences, negation, homonyms, and comparative or superlative distinctions.Source: §E.1 · p. 30
  • Other T5 failures include difficulty distinguishing real from hypothetical scenarios and interpreting implicit versus explicit meanings.Source: §E.1 · p. 30
  • T5 failures also include pronoun ambiguity, which can change the number of people represented, and reversed semantic roles, which change who initiates an action.Source: §E.1 · p. 30
  • 77.3% is the average success rate for T5 systematic failures; pronoun ambiguity and temporal-difference failures are unique to T5 and absent in CLIP.Source: §E.1 · p. 31
  • T5-generated inputs lead to failures in DeepFloyd; T5-specific failures cause issues there with the T5 encoder, but not in models based on CLIP.Figure 15 shows examples of T5-generated inputs producing failures in T5-based DeepFloyd.Source: §E.1 · p. 31, Figure 15 · p. 31

E.2 Prompts used in transfer experiments

  • The transfer experiments use prompts found with CLIP and presented in the figures, including examples about counts, negation, and spatial relations.Source: §E.2 · p. 31

F Results on Using MULTIMON to test contemporary safety filters

The appendix illustrates using MULTIMON’s categorized failures to probe MidJourney 5.1’s safety filter. The examples suggest this can help evaluators find failures before deployment.

F Results on Using MULTIMON to test contemporary safety filters

Loading 2306.12105v2…