PaperlayerView exact revision on arXiv

Source-linked AI summary

Diagnosing the Sources of Compositional Failure in Vision-Language Models: A Controlled Analysis

Mona Gandhi, Cenk Merih Olcay, Kuan-Chieh Lo, Santiago Castro, Christopher W. Myers, Srinivasan Parthasarathy

arXiv:2609.31456v1cs.CV

TL;DR

Existing benchmarks leave the contribution of joint reasoning to VLM compositional failures unquantified. COMPASS compares composed and decomposed captions and measures skill-specific load. It finds a measurable but partial integration cost, alongside degradation tied primarily to each skill’s own primitive count.

  • Problem

    Existing benchmarks evaluate composed captions without isolating how much degradation comes from joint reasoning versus other factors.

    Source: §1 · p. 2

  • Method

    COMPASS uses scene-graph captions and matched composed/decomposed comparisons to measure integration costs, plus skill-targeted analyses of primitive-count load.

    Source: §3 · p. 3, §5 · p. 5

  • Results

    Across model families, integration incurs a consistent but partial cost, while each skill degrades primarily with its own primitive count and cross-load effects are predominantly positive.

    Source: §1 · p. 2, §6.3 · pp. 8–9

  • Takeaways & Limitations

    Compositional degradation reflects separable skill-load and joint-reasoning factors, so caption complexity is not adequately captured as a single scalar.

    Source: §6.3 · p. 9

    COMPASS uses synthetically generated captions from Visual Genome scene graphs, which may not reflect natural descriptive language and inherit scene-graph annotation biases.

    Source: §7 · p. 9

Abstract

from arXiv · show

Vision-language models (VLMs) often struggle with compositional reasoning tasks, but the reasons for this underperformance remain unclear. A common hypothesis is that models struggle to integrate multiple components, leading to training interventions to improve compositional binding. However, this assumption has never been directly quantified. Existing benchmarks evaluate captions only in their composed form, making it impossible to separate the cost of joint reasoning from the cost of recognizing individual components under increasing load. We introduce COMPASS (COMPositional Analysis of SkillS), a controlled evaluation framework designed to isolate and measure the distinct factors underlying compositional failure. By comparing performance on composed captions with their decomposed counterparts , we directly quantify the cost of compositional integration across 87K image-caption pairs. Across multiple VLMs, this gap is real but partial, accounting for only part of the observed degradation. This motivates a finer-grained investigation into what additional factors govern model behavior. We analyze performance at the level of individual skills: object detection, attribute binding, and relation reasoning, using skill-targeted perturbations across 274K image-caption pairs. We find a consistent skill-specific pattern: each skill degrades primarily with the count of its own primitive type (self-load), while cross-load effects are predominantly positive, suggesting that primitives of different types provide useful grounding context. This pattern holds across standard contrastive encoders, explicitly trained compositional reasoning models, and non-contrastive architectures. These findings show that compositional degradation reflects multiple separable factors that cannot be reduced to joint reasoning alone.

1 Introduction

VLMs must understand how objects relate to attributes and one another, yet the sources of their compositional failures remain unclear. COMPASS separates the cost of joint reasoning from skill-specific load and finds that integration is only part of the degradation.

  • Compositional understanding—the ability to understand and produce novel combinations of known concepts—remains a persistent challenge for VLMs.It matters in settings including retrieval, referring-expression grounding, and visual question answering.Source: §1 · p. 1
  • Existing benchmarks evaluate only composed captions, leaving the contribution of joint reasoning unquantified despite its being a common explanation for degradation.Prior interventions have often focused on improving compositional binding.Source: §1 · p. 2
  • COMPASS compares composed captions with matched decomposed counterparts and measures skill load by varying primitive types and counts.The analyses cover 87K image-caption pairs for compositional integration and 274K pairs for skill load.Source: §1 · p. 2
  • Across model families, compositional integration incurs a consistent cost but does not fully explain the observed degradation.Performance also varies systematically with primitive counts and individual skills.Source: §1 · p. 2

2 Related Work

Prior benchmarks document degradation with caption complexity and examine particular primitive types, but often leave joint reasoning or individual primitive contributions unresolved. COMPASS frames these as separable sources of failure and provides controlled analyses of both.

  • CREPE establishes monotonic performance degradation with entity count, while SugarCREPE uses LLM-generated negatives to address benchmark hackability.The supplied discussion says existing benchmarks still do not directly quantify how much decline stems from joint reasoning.Source: §2 · p. 2
  • VL-CheckList reports model-specific variability across primitive types, and SugarCREPE finds relation negatives harder than attribute and object negatives.Complexity-varying benchmarks can still conflate the contributions of different primitive types by treating complexity as one axis.Source: §2 · p. 2
  • Training interventions target compositional reasoning, but reported gains are described as modest and inconsistent, motivating attribution of gains to specific skills or joint reasoning.The cited approaches include hard-negative fine-tuning, grounding supervision, and structured concept training.Source: §2 · p. 3

3 COMPASS: A Testbed for Analysing Compositional Failures

COMPASS uses scene-graph captions with controlled object, attribute, and relation structure to diagnose compositional failure. Its retrieval-based analyses isolate joint reasoning and skill-specific load using hard negatives.

3 COMPASS: A Testbed for Analysing Compositional Failures

  • COMPASS separates compositional integration gap, measured by comparing composed and decomposed captions under matched perturbations, from skill load across primitive types.The testbed defines complexity as the total number of primitives and supports analyses of both joint reasoning and individual skills.Source: §3 · p. 3

3.1 Structured Caption Construction

  • Captions are built from Visual Genome scene graphs, sampled as coherent subgraphs through random walks of up to ten steps.The same image-specific primitives are reused across evaluations; the testbed uses 5K image–scene graph pairs.Source: §3.1 · p. 3
  • L1 contains objects, L2 combines objects with attributes or relations, and L3 combines objects, attributes, and relations; complexity varies by primitive count.The study focuses on L2 and L3, with count ranges set by structural level.Source: §3.1 · pp. 3–4, Figure 1 · p. 4
  • Structured primitive sets are converted into natural-language captions with GPT-4o-mini, yielding 1.38M composed ground-truth captions.The captions share ground-truth primitives across evaluations and are assessed through retrieval with hard negatives.Source: §3.1 · p. 4

3.2 Retrieval-based Evaluation using Hard Negatives

  • Hard negatives replace one primitive with another of the same type while holding the remaining caption content fixed, isolating recognition of that component.The retrieval protocol uses an LLM-based substitution approach and a perplexity audit to check linguistic indistinguishability.Source: §3.2 · pp. 4–5

4 Experimental Setup

The experiments compare VLMs spanning contrastive, explicitly compositional, and non-contrastive architectures. A unified image-to-text retrieval protocol supports comparisons across models.

  • The model set includes standard contrastive baselines and variants, compositional models NegCLIP and CE-CLIP, and non-contrastive BLIP-L and Qwen3-VL-Embedding-8B.The selection is intended to test whether skill-load effects extend beyond contrastive architectures.Source: §4 · p. 5
  • All architectures are evaluated with a unified retrieval protocol that ranks candidate captions by image-text embedding similarity.This provides a consistent setting for comparing compositional integration and skill load.Source: §4 · p. 5
  • The experiments were run on a single NVIDIA A100 GPU.Source: §4 · p. 5

5 Compositional Integration Gap: Composed vs. Decomposed

The compositional integration gap compares retrieval of composed captions with retrieval of their independent primitives. Across models and structural levels, joint reasoning usually incurs a measurable cost, but exceptions and continued degradation with caption complexity show that this gap is not the whole story.

5.1 Data Setup

  • COMPASS pairs each composed caption with N single-primitive captions and matched perturbations, yielding 87K caption pairs across structural levels.The decomposed captions target one scene-graph primitive each; matched negatives apply the same semantic perturbation in both settings.Source: §5.1 · pp. 5–6, Figure 2 · p. 6

5.2 Metrics for Compositional Integration.

  • Composed recall requires beating all N hard negatives, while decomposed success requires every primitive to beat its paired negative; Δ is decomposed minus composed recall.This joint-success comparison measures the retrieval difference between the two settings.Source: §5.2 · p. 6

5.3 How Much Does Joint Reasoning Cost?

  • Across models and structural levels, Δ is predominantly positive, indicating better independent than joint performance and a measurable cost of joint reasoning.Table 2 aggregates the difference across all complexities.Source: §5.3 · p. 6, Table 2 · p. 7
  • PE-CLIP has the smallest integration gaps among standard and recent contrastive models, while SigLIPv2 has the largest; Qwen3 has the largest OAR gap overall.BLIP-L shows moderate gaps consistent with the contrastive baseline. NegCLIP has negative Δ only on OA, while CE-CLIP has negative Δ across all structural levels.Source: §5.3 · p. 6
  • NegCLIP and CE-CLIP achieve negative Δ values, meaning composed performance exceeds decomposed performance; the authors suggest compositional training helps leverage joint context.The authors characterize this as improving joint reasoning rather than merely shifting distributional preferences.Source: §5.3 · p. 7
  • Even when compositional training reverses the integration gap, CLIP, NegCLIP, and CE-CLIP still degrade monotonically with increasing caption complexity on CREPE’s productivity benchmark.The discussion motivates analyzing primitive counts independently to investigate degradation beyond joint reasoning.Source: §5.3 · p. 7

6 Skill Load: Skill Targeted Perturbations

COMPASS varies object, attribute, and relation counts to measure how each skill responds to its own and other primitive types. Across models and structural levels, each skill degrades most with its own primitive count, while cross-load effects are predominantly positive.

6 Skill Load: Skill Targeted Perturbations

  • Skill load measures how each skill’s R@1 changes with counts of its own and other primitive types, distinguishing self-load from cross-load.Self-load concerns the tested primitive type; cross-load concerns other types.Source: §6 · p. 7

6.1 Data Setup

  • Skill-targeted negatives replace only the tested primitive type, with four hard negatives generated per caption; the evaluation covers 143K object, 63K attribute, and 68K relation caption pairs.Other primitives remain fixed, isolating detection of the targeted type.Source: §6.1 · pp. 7–8

6.2 Metrics

  • R@1 is modeled against object, attribute, and relation counts using OLS with image-clustered standard errors; negative coefficients indicate degradation as a primitive count increases.Two-sided t-tests use p < 0.005 and p < 0.001 significance thresholds.Source: §6.2 · p. 8

6.3 What Drives Skill-Level Degradation?

  • Across models and structural levels, each skill degrades primarily with its own primitive count, while cross-load effects are predominantly positive and attribute self-load is strongest.Relation self-load is smaller but significant across most models; object count provides positive cross-load for relation detection. These patterns persist across standard contrastive, compositional, and non-contrastive models.Source: §6.3 · pp. 8–9, Table 3 · p. 8
  • The findings show that skill-specific load effects and grounding benefits complicate any account of degradation as a single joint-reasoning bottleneck.Even NegCLIP and CE-CLIP, which reverse the compositional integration gap, retain self-load degradation.Source: §6.3 · p. 9

7 Conclusion

The paper concludes that compositional degradation has multiple separable sources, including integration cost and skill-specific load. Its appendices detail negative generation and describe a perplexity-based audit and effect-size measure.

7 Conclusion

  • Compositional degradation reflects multiple separable factors: integration cost accounts for only part of it, and skills degrade primarily with their own primitive counts.Cross-load effects provide grounding benefits, and the conclusion suggests skill-specific interventions to preserve robustness as primitive load increases.Source: §7 · p. 9
  • COMPASS uses Visual Genome scene graphs and synthetic captions, which may not reflect natural descriptive language or avoid inherited annotation biases.Attribute and relation evaluation also retains objects as grounding context, while retrieval-based evaluation does not cover generative VLM settings.Source: §7 · p. 9

A Details About Hard Negative Generation

  • Caption prompts turn scene-graph primitives into fluent sentences without adding objects or relations; separate object, attribute, and relation prompts generate plausible non-identical alternatives.Generated alternatives are filtered for semantic distinctiveness, with thresholds of 0.9 for N ≥ 6 and 0.95 for N < 6.Source: A Details About Hard Negative Generation · p. 12, Figure 4 · p. 13
  • Skill-targeted negative generation uses weighted primitive sampling across four negatives, while composed and decomposed settings target each primitive once and discard instances lacking valid negatives.Skill-targeted generation makes up to 10 attempts; composed and decomposed generation makes up to three attempts per primitive.Source: A Details About Hard Negative Generation · p. 12

B.1 Overview

  • The perplexity audit compares ground-truth captions with hard negatives to test whether linguistic statistics alone could distinguish them.It measures token-level perplexity under GPT-2 across primitive types and structural levels.Source: §B.1 · p. 13

B.2 Evaluating on the Hardest Negative

  • For each ground-truth caption, the hardest negative is the skill-targeted candidate whose perplexity is closest to the ground truth, providing a worst-case check of exploitability.In the four-negative task, success via perplexity requires the ground truth to have lower perplexity than all four negatives.Source: §B.2 · pp. 13–14

B.3 Effect Size Metric

  • Rank-biserial correlation characterizes the distributional difference between ground-truth and negative perplexities without depending on retrieval format or chance-level definitions.The paper interprets r < 0.1 as negligible, 0.1 ≤ r < 0.3 as small, and r ≥ 0.3 as medium or large.Source: §B.3 · p. 14

B.4 Results: Skill-Targeted Negatives

Skill-targeted negatives show little linguistic separation from ground-truth captions under hardest-negative selection, with relation negatives somewhat more distinguishable than object and attribute negatives.

B.4 Results: Skill-Targeted Negatives

  • Object and attribute negatives have negligible effect sizes across structural levels (r = 0.062–0.099), while relation negatives are slightly larger but still small (r = 0.259–0.267).The analysis uses rank-biserial effect sizes on the hardest negative subset.Source: §B.4 · p. 14, Table 4 · p. 14
  • Under worst-case selection, ground-truth and negative captions are linguistically indistinguishable for object and attribute negatives.Source: §B.4 · p. 14

B.5 Results: Compositional Integration Negatives

Composed captions show negligible-to-small perplexity artifacts, supporting their use in the composed side of the integration-gap analysis. Decomposed-caption perplexity is not informative because the captions are short and word substitutions make corpus frequency dominate.

B.5 Results: Compositional Integration Negatives

  • Composed captions show negligible-to-small perplexity artifacts across structural levels (r = 0.060–0.152), so the composed evaluation setting is not systematically exploitable through linguistic statistics.These results validate the composed side of the compositional integration-gap analysis.Source: §B.5 · pp. 14–15, Table 5 · p. 14
  • Decomposed perplexity results are not informative for benchmark validity because captions typically contain 6–8 tokens, making perplexity dominated by the substituted word’s corpus frequency.The passage attributes these artifacts to caption length rather than the negative-generation pipeline.Source: §B.5 · p. 15

C Compositional Integration Gap Raw Results

Composed-versus-decomposed comparisons show an integration penalty for most models, but the direction and size vary by model and structural setting. CE-CLIP is a recurring negative-gap exception, while NegCLIP’s exception does not persist in the object-relation setting.

C Compositional Integration Gap Raw Results

  • Most models have positive integration-gap values in the object-attribute setting, indicating better accuracy on independent than joint evaluation, while NegCLIP and CE-CLIP are exceptions.CE-CLIP has a mean Δ of −20.27; positive Δ indicates an advantage for decomposed evaluation.Source: C Compositional Integration Gap Raw… · p. 15, Table 6 · p. 15
  • Object-relation results show considerable integration penalties, with the most severe drops in OpenCLIP and SigLIPv2; NegCLIP is positive across complexity levels, while CE-CLIP has a mean Δ of −4.96.The table reports the decomposed-minus-composed accuracy difference by complexity level for the object-relation setting.Source: C Compositional Integration Gap Raw… · p. 15, Table 7 · p. 16
  • Most object-attribute-relation settings retain considerable integration overhead despite lower Δ values than the simpler object-relation and object-attribute settings.Source: C Compositional Integration Gap Raw… · p. 15, Table 8 · p. 16
  • In object-attribute-relation settings, NegCLIP has the largest gap (mean Δ = 3.75), whereas CE-CLIP has a negative mean Δ of −5.66.The passage notes that hard-negative discrimination with attributes does not directly translate to whole-scene compositions.Source: C Compositional Integration Gap Raw… · p. 16, Table 8 · p. 16

D Compute Resources

Evaluations of most models took 2–3 hours per model, while Qwen3-VL-Embedding-8B took approximately 8 hours.

D Compute Resources

  • Most evaluated models completed evaluation within 2–3 hours per model, while Qwen3-VL-Embedding-8B required approximately 8 hours.The reported runs used a single NVIDIA A100 GPU.Source: D Compute Resources · p. 16

E Skill Load: Visualization

Across seven models, skill-load visualizations show self-load dominance: each negative type declines with its own primitive count, while cross-load patterns remain small or inconsistent.

E Skill Load: Visualization

  • Each panel plots R@1 against one primitive count, with separate lines for each controlled-count value; its global slope corresponds to the Table 3 OLS coefficient.The regression controls for all primitive counts.Source: E Skill Load: Visualization · p. 16
  • For object negatives, R@1 declines with object count; for attribute negatives, it declines with attribute count, while cross-load lines are flat or inconsistent across all seven models.Figures 5–7 visualize the self-load dominance reported in Table 3.Source: E Skill Load: Visualization · pp. 16–17, Figure 5 · p. 17, Figure 6 · p. 18

Related papers

Loading 2609.31456v1…