PaperlayerView exact revision on arXiv

Source-linked AI summary

VLTest: Finding Joint Image-Text Failures in Vision-Language Models

Bin Duan, Jintao Lin, Guowei Yang

arXiv:2610.30794v1cs.SEcs.AI

TL;DR

Mutating either input can change a VLM’s expected answer, and testing joint failures requires matched single-modality controls. VLTest generates and filters image and text variants, then tests them alone and in combination with oracle confirmation. Across three tasks and eight VLMs, its generators find 1.8 to 2.5 times as many confirmed failures as SINVAD and TextFooler under the same budget.

  • Problem

    For VLMs, an output change after mutation may be a correct response to a changed expected answer, and joint failures need matched single-modality controls for attribution.

    Source: §1 · p. 1

  • Method

    VLTest generates controlled image and text variants, filters variants likely to change expected answers, tests variants alone before combinations, and uses a VLM judge to confirm failures.

    Source: §1 · p. 2, §4.4 · p. 5, §4.6 · p. 6

  • Results

    Under the same budget, VLTest’s generators find 1.8 to 2.5 times as many confirmed failures as SINVAD and TextFooler.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Four-cell confirmation rejects 22.5% of joint candidates, including candidates rejected by the joint input and by single-modality controls.

    Source: §1 · p. 2

    The evaluation covers three tasks, eight models, and one tokenizer family; results may differ for other tokenizers and generators, high-resolution or text-heavy images, and multiturn systems.

    Source: §8 · p. 14

Abstract

from arXiv · show

Vision-language models (VLMs) are increasingly embedded in software systems. Testing them raises two problems. Mutating either input can change the expected answer, so an output change is not by itself a failure. Some failures appear only when both inputs are mutated; attributing them to the combination requires testing each mutation alone. We present VLTest, a black-box VLM testing approach. It generates image variants by editing VQGAN latent codes and text variants with a rewrite-expand-prune pipeline, filters variants likely to change the expected answer, and tests each variant alone before testing combinations, so every joint failure has four matched cells. A VLM judge confirms a failure only after checking that the input keeps the expected answer. On three tasks with eight VLMs, including two commercial API models, the oracle confirms 81.7% of the candidates, with 88.8% to 94.6% precision against human labels, and four-cell confirmation rejects 22.5% of joint candidates. Under the same budget, VLTest's generators find 1.8 to 2.5 times as many confirmed failures as SINVAD and TextFooler, and its joint tests find 91 to 197 attributed joint failures per 1,000 joint calls, 1.3 times as many as the same pairing applied to the baselines' variants.

1 Introduction

VLTest addresses the challenge that image or text mutations can change the correct answer, and that joint failures require matched single-modality controls for attribution. It combines controlled generators, answer-preservation filters, two-phase testing, and an oracle; across three tasks and eight VLMs, it finds more confirmed failures than the named baselines under the same budget.

1 Introduction

  • Mutating either input can change the expected answer, so an output change alone does not establish failure; joint failures also require checking each mutation alone.The four cells are the seed, image-only input, text-only input, and joint input, with a common reconstructed reference.Source: §1 · p. 1
  • VLTest generates visual variants by editing VQGAN latent codes and text variants through rewrite–expand–prune, then filters variants likely to change the expected answer.It tests variants individually before combinations, builds all four cells on a reconstructed seed image, and uses single-modality results to prioritize pairings.Source: §1 · pp. 1–2
  • A VLM judge checks whether each input preserves the seed’s expected answer before judging the model output, and a joint failure must pass those checks across all four cells.Human labels on reported failures assess the oracle’s precision.Source: §1 · p. 2
  • Under the same budget, VLTest’s generators find 1.8 to 2.5 times as many confirmed failures as SINVAD and TextFooler, while joint tests find 91 to 197 attributed joint failures per 1,000 joint calls.The joint-test yield is 1.3 times that of stable-first pairing on the baselines’ variants and 2.8 times that of applicable MetaRA relations.Source: §1 · p. 2
  • The evaluation covers image–text matching, visual reasoning, and visual question answering with eight VLMs, including two commercial API models.The oracle confirms 81.7% of VLTest (SF) candidates and has 88.8% to 94.6% precision against human labels on failures reported by VLTest (SF).Source: §1 · p. 2

2 Background and Related Work

The paper studies three VLM task types and positions VLTest among testing, metamorphic, adversarial, and text-perturbation approaches. Its distinctive focus is generated image–text inputs with oracle-confirmed expected answers and matched controls for joint failures.

2.1 Background

  • VLMs take an image and text and produce a decision or natural-language answer; the paper considers binary image–text matching and visual reasoning, plus VQA with short free-form answers.The described architectures range from single-stream fusion encoders to generative models coupling vision encoders with language models.Source: §2.1 · p. 2

2.2 Related Work

  • Unlike fixed or curated VLM benchmarks, VLTest generates new image–text inputs from seeds and confirms the expected answer for each reported failure.Related seed-centric generators, latent-space methods, and metamorphic testing provide precedents, but the cited test-generation work targets unimodal classifiers.Source: §2.2 · p. 2
  • VLTest checks the expected-answer-preservation precondition for each reported failure and pairs joint failures with matched, oracle-confirmed single-modality controls.The paper compares applicable MetaRA relations; adversarial attacks instead maximize attack success under a perturbation bound, and TextFooler supplies a text-side baseline.Source: §2.2 · p. 3
  • VLTest treats image and text variants as generated levels of two factors, selecting combinations to test and using four-cell evidence to attribute failures.This differs from classical combinatorial interaction testing, which uses small predefined factor domains and covering arrays.Source: §2.2 · p. 3

3 Problem Formulation

The formulation treats a VLM input as an image–text pair whose expected answer must be established rather than inferred from the mutation. It distinguishes confirmed failures from attributed joint failures and uses four matched cells to assess joint effects.

3.1 Inputs and Expected Answers

  • A VLM maps an image–text pair to an output, while task-specific normalization makes outputs comparable as binary labels or canonical short answers.A visual-reasoning seed also contains a second image, which remains fixed in derived inputs.Source: §3.1 · p. 3
  • The seed’s dataset answers are reference answers, while a generated input’s expected answer is the set of answers correct for that input and may differ from the reference set.For VQA, reference answers are those receiving full credit under the official metric, given by at least four of ten annotators.Source: §3.1 · p. 3

3.2 Test Oracle

  • VLTest uses an invariance relation: if an input preserves the seed’s expected answer, the model should produce an answer in that expected set.Because the relation’s precondition depends on the image–text relation, an oracle establishes expected-answer preservation.Source: §3.2 · p. 3
  • A candidate is an input whose output is incompatible with the seed’s reference answers; it is confirmed as a failure only when the oracle verifies answer preservation and an incorrect model output.A candidate can instead reflect a changed expected answer, an invalid input, or a correct answer outside the reference set.Source: §3.2 · p. 3
  • An attributed joint failure additionally requires both single-modality cells to be valid, preserve the seed’s expected answer, and receive correct model outputs.The joint cell is checked separately because two answer-preserving single mutations do not guarantee that their combination preserves the answer.Source: §3.2 · p. 3

3.3 Reference Input

  • All cells use a reconstructed seed image so that reconstruction artifacts are shared and each cell differs from the reference only by its mutations.A seed is admitted only after the oracle confirms the reconstruction preserves its expected answer and the model output remains compatible.Source: §3.3 · p. 3

3.4 Interaction and Additive Failures

  • An attributed joint failure may result from additive effects that together cross a decision boundary; with model scores, the paper distinguishes these from interaction failures.An interaction failure occurs when the additive prediction still prefers a reference answer; the interaction contrast applies to each attributed joint failure, while VLTest itself uses normalized outputs.Source: §3.4 · pp. 3–4
  • Table 1 compares VLTest with representative approaches; its final column distinguishes approaches that do not test joint inputs from those that test them without matched single-modality controls.The caption labels image and text as I and T, respectively.Source: Table 1 · p. 4

4 Approach

VLTest generates image and text variants, filters those likely to change the expected answer, and tests single-modality inputs before combinations. Its oracle confirms failures only when the expected answers are preserved and the model outputs are wrong.

4 Approach

  • VLTest combines controlled variant generators, relation filters, two-phase testing, and an automated oracle to confirm failures and attribute joint failures.Every joint test uses a seed, image-only input, text-only input, and joint input built on a common reconstructed reference.Source: §4 · p. 4

4.1 Visual Variant Generation

  • VQGAN edits replace frequent latent-grid indices with nearby codebook entries, producing decoded visual variants in a fixed, deduplicated order.Frequent indices represent dominant visual prototypes, so replacing them can create observable edits with few variants.Source: §4.1 · p. 4

4.2 Textual Variant Generation

  • Text variants are rewritten, expanded with same-part-of-speech token substitutions, then ranked by Sentence-BERT distance and deduplicated.The pipeline interleaves retained candidates by within-group rank, so a larger pool appends to a smaller one.Source: §4.2 · pp. 4–5

4.3 Relation Filters

  • Bidirectional entailment filters text, embedding similarity filters images, and task-specific alignment rules filter applicable image–text pairs.For false visual-reasoning statements and VQA questions, only the text and image filters apply; the oracle checks the joint relation.Source: §4.3 · p. 5

4.4 Two-Phase Testing

  • Two-phase testing runs each filtered single-modality variant before combinations, giving every joint candidate complete four-cell evidence.The pair filter is reapplied to each combination, and combinations that fail this check are skipped without a model call.Source: §4.4 · p. 5

4.5 Pairing Strategies

  • Round-robin pairing covers every pair once, while stable-first prioritizes pairs whose single-modality inputs both have compatible outputs.Stable-first orders these pairs in round-robin order before the remaining pairs.Source: §4.5 · p. 6

4.6 Oracle

  • The oracle first checks whether each input preserves the seed’s expected answer, then checks output correctness; attributed joint failures require preserved answers in all cells.Both single-modality outputs must be correct and the joint output incorrect; abstentions do not confirm failures.Source: §4.6 · p. 6

5 Evaluation Design

The evaluation tests VLTest’s validity, testing utility, joint-failure findings, and sensitivity across tasks and models. It uses matched-budget baselines, defined failure metrics, and fixed implementation settings.

5.1 Research Questions

  • Four research questions assess oracle validity, testing utility and cost, attributed joint failures, and sensitivity across design choices and VLMs.The study evaluates oracle precision and filter validity, compares methods under the same budget, and categorizes joint failures.Source: §5.1 · p. 6

5.2 Datasets and Models

  • The evaluation covers ITM on MSCOCO, VR on NLVR2, and VQA on VQA v2, with two fine-tuned models per task and 100 admitted seeds per task–model pair.Seeds come from validation splits; NLVR2 inputs contain two images and a statement, and VLTest mutates the first image.Source: §5.2 · pp. 6–7

5.3 Baselines

  • Baselines include SINVAD-VQ and TextFooler for single-modality testing and random, stable-first, and MetaRA-c methods for joint testing.All methods share the same budget; all except RND and MetaRA-c apply relation filters.Source: §5.3 · p. 7

5.4 Metrics

  • The evaluation reports confirmed failures and attributed joint failures per 1,000 budgeted calls, alongside confirmation rate, coverage, and interaction share.It also measures filter precision and recall against the oracle’s answer check and evaluates reconstruction effects.Source: §5.4 · p. 7

5.5 Settings and Implementation

  • Primary experiments use 1,000 calls per single-modality or joint phase, with joint methods first making 500 single-modality calls per modality.The VQGAN uses a 32 × 32 latent grid and 8,192 codebook entries; the text generator keeps 4,000 variants.Source: §5.5 · pp. 7–8

6.1 RQ1: Validity

VLTest’s oracle confirms most candidates as failures, and human labels support its precision. The evaluation also measures answer-preservation filters and the effects of reconstruction.

6.1.1 Candidate Outcomes.

  • 81.7% of VLTest (SF) candidates were confirmed failures, and its precision against human labels ranged from 88.8% to 94.6%.The oracle confirmed 121,235 of 148,314 candidates; the precision range covers confirmed and attributed joint failures.Source: §6.1.1 · p. 8, §6.1.2 · p. 8
  • Joint inputs had the lowest confirmation rates (75.0% to 83.0%), while 14.8% of candidates had changed expected answers.Image-only confirmation rates were 86.3% to 91.6%, and text-only rates were 77.0% to 86.8%; changed-answer candidates are not failures under the oracle.Source: §6.1.1 · p. 8

6.1.2 Oracle Precision.

  • The answer check agreed with annotators on 95.6% of reconstructed seeds, and annotators agreed with 85.0% to 95.0% of oracle rejections across strata.For VLTest (SF), human-label precision ranged from 88.8% to 94.6% for confirmed and attributed joint failures.Source: §6.1.2 · pp. 8–9

6.1.3 Filters and Reconstruction.

  • Ftxt, Fimg, and Fpair had answer-preservation precision of 93.1%, 95.4%, and 91.7%, and recall of 83.6%, 91.2%, and 86.0%, respectively.Reconstruction changed expected answers for 2% to 6% of sampled seeds; model outputs differed on 4% to 9% of admitted seeds.Source: §6.1.3 · p. 9

6.2 RQ2: Testing Utility and Cost

VLTest's generators find more confirmed single-modality failures than the baselines, while its joint-testing variants improve attributed-failure yield through filtering and stable-first ordering.

6.2.1 Single-Modality Failures.

  • VLTest-V finds 1.8 to 2.1 times as many confirmed failures as SINVAD-VQ, and VLTest-T finds 2.3 to 2.5 times as many as TextFooler.VLTest's generators also cover more seeds: 75.7% and 71.0%, versus 63.4% and 56.7%.Source: §6.2.1 · p. 9

6.2.2 Attributed Joint Failures.

  • VLTest (SF) finds 91.4 to 196.8 attributed joint failures per 1,000 joint calls, averaging one per 7.6 calls and finding at least one on 98.2% of seeds.All joint methods except MetaRA-c use the full joint budget.Source: §6.2.2 · p. 9
  • Applying relation filters raises attributed joint-failure yield 1.6 to 1.9 times over random pairing, while stable-first ordering raises AJF@1k by 14% to 27% for VLTest (RR) versus VLTest (SF).Across six task–model pairs, AJF@1k rises from 121.9 to 142.3.Source: §6.2.2 · p. 9
  • Stable-first ordering spends joint calls only on pairs that can yield an attributed joint failure, without making stable pairs more likely to fail or adding cost.Moving calls onto stable pairs yields average AJF@1k gains of 16% and 17%, close to the approximately 17% expected if stable pairs were equally likely to yield failures.Source: §6.2.2 · p. 9
  • Round-robin ordering changes AJF@1k by only 1% to 5% within a priority class, but covers more distinct variants and seeds than random ordering.VLTest (SF) finds 1.26 to 1.33 times as many attributed joint failures as EXT across six task–model pairs.Source: §6.2.2 · p. 9

6.3 RQ3: Joint Failures

Four-cell evidence filters joint candidates to failures attributable to the combined mutations, while margin analysis and sampled labels characterize the resulting failures.

6.3.1 Four-Cell Evidence.

  • Of VLTest (SF)'s 110,152 joint candidates, 85,390 (77.5%) are attributed joint failures and four-cell confirmation rejects 24,762 (22.5%).The joint-input check rejects 19.7% of candidates, and the single-modality controls reject a further 2.8%.Source: §6.3.1 · p. 9, Table 7 · p. 11

6.3.2 Interaction and Additive Failures.

  • Among score-classifiable attributed joint failures, 16.5% are interaction failures and 83.5% are additive; interaction shares range from 13.8% on PaliGemma to 22.6% on ViLT for VR.The analysis excludes 622 PaliGemma failures with η(x,t) > 0; Δint is positive on every pair.Source: §6.3.2 · p. 10, Table 8 · p. 11

6.3.3 Failure Types.

  • Attribute or entity binding and spatial or relational composition account for 98 of 180 sampled failures, while quantifier, count, and scope errors concentrate on VR.Quantifier, count, and scope errors account for 26 of 34 such errors on VR.Source: §6.3.3 · p. 10
  • In two examples, both single-modality inputs pass although the joint input fails, including a VQA answer changing from “brown” to “yellow.”The examples cover image–text matching and visual question answering.Source: §6.3.3 · pp. 10–11

6.4 RQ4: Sensitivity and Generalization

VLTest's design choices affect candidate confirmation and attributed joint-failure yield; its stable-first variant also exceeds two comparison methods on all twelve general-purpose-model task–model pairs.

6.4.1 Design Choices.

  • The default setting reaches an 81.6% confirmation rate, 55.1 CF@1k, and 145.9 AJF@1k; removing filters lowers AJF@1k by 25% to 108.7.Removing any text stage lowers confirmation to 73.9%–79.6% and AJF@1k by 9%–13%.Source: §6.4.1 · p. 11

6.4.2 General-Purpose Models.

  • The general-purpose evaluation includes Qwen3-VL at two scales and two commercial API models, with 37,650 single-modality calls and 37,650 joint calls for API failure discovery.API models were evaluated with 50 sampled seeds per task, a budget of 50, and n=25.Source: §6.4.2 · p. 11
  • VLTest (SF) finds 111.8 to 249.8 attributed joint failures per 1,000 joint calls and exceeds SR by 5.6 to 24.1 and CRND by 21.6 to 41.3 on all twelve task–model pairs.On the API models, the smallest difference, 5.6 for GPT-4o mini on ITM, corresponds to about 12 attributed joint failures.Source: §6.4.2 · p. 11

7 Discussion

Output changes can overstate failures because mutations may change the expected answer; four-cell evidence checks whether a joint failure is attributable to the combination. The discussion also shows how confirmed failure patterns can inform regression testing, while warning against treating observed rates as task or model rankings.

7.1 Output Changes Overstate Failures

  • 20% to 33% relative overstatement occurs for joint inputs when output changes are counted as failures; filtering raises confirmation from 71.8% to 81.6%.The overstatement is 9% to 16% for image-only inputs and 15% to 30% for text-only inputs; on VQA, 6.4% of output changes stay within the reference answers.Source: §7.1 · pp. 11–12

7.2 What Four-Cell Evidence Adds

  • 22.5% of joint candidates are rejected by four-cell confirmation, while single-modality controls reject a further 2.8% and establish attribution to the combination.Among score-classifiable attributed joint failures, 16.5% are interactions between modalities; in the rest, additive effects already cross the decision boundary, and the positive median Δint indicates partial cancellation is usual.Source: §7.2 · pp. 12–13

7.3 Implications

  • Attributed joint failures can serve as regression tests, but their rates are not task or model rankings because oracle rejection and admitted seeds differ.More than half of the sampled failures concern attribute binding or spatial composition, while quantifier and count errors cluster on VR.Source: §7.3 · p. 13

8 Threats to Validity

The validity discussion identifies limits in oracle measurement and study scope. Results depend on judge verdicts and reconstructed seeds, and the evaluation covers a bounded set of tasks, models, and input conditions.

Internal Validity

  • The VLM judge can make mistakes, and missed failures are not measured for other methods; reference answers may bias its answer-preservation checks.Seeds whose reconstruction changes the expected answer are excluded, and admitting seeds per model favors inputs that the model handles correctly.Source: §8 · p. 13

External Validity

  • The study covers three tasks, eight models, and one tokenizer family, using 256×256 VQGAN reconstructions that may limit transfer to other settings.Results may differ for other tokenizers and generators, high-resolution or text-heavy images, multiturn systems, and API model versions beyond those studied.Source: §8 · p. 14

9 Conclusion

VLTest tests VLM failures while checking expected-answer validity and using four-cell evidence to attribute joint failures. Its generators outperform two baselines under the same budget, and its joint tests yield attributed failures across three tasks.

Conclusion

  • 1.8 to 2.5 times as many confirmed failures as SINVAD and TextFooler are found by VLTest generators under the same budget.VLTest finds 91 to 197 attributed joint failures per 1,000 joint calls, 1.3 times as many as the same pairing applied to the baselines' variants.Source: §9 · p. 14
  • VLTest combines controlled visual and textual generators, relation filters, two-phase testing, feedback-directed pairing, and an automated oracle.The oracle confirms a failure only after checking the expected answer, while four-cell evidence attributes joint failures to image-text combinations.Source: §9 · p. 14
  • 81.7% of VLTest (SF) candidates are confirmed, and its default configuration has 88.8% to 94.6% agreement with human labels on reported failures.Across the compared methods, human-label agreement ranges from 80.0% to 94.6%.Source: §9 · p. 14
  • 22.5% of joint candidates are rejected by four-cell confirmation; 19.7% are rejected through the joint input and 2.8% through single-modality controls.Under an additive margin model, 16.5% of score-classifiable attributed joint failures are modality interactions; the additive prediction already fails in the rest, although each single-modality input passes.Source: §9 · p. 14
  • Relation filters account for most of the difference between configurations, and VLTest (SF) finds more attributed joint failures than CRND and SR on every task for four general-purpose models.The four models include two commercial API models.Source: §9 · p. 14

Related papers

Loading 2610.30794v1…