PaperlayerView exact revision on arXiv

Source-linked AI summary

Where MLLMs Fail and Why: Causal Task Decomposition for Capability Failure Diagnosis

Xia Hu, Brian Potetz, Chun-Ta Lu, Huanfen Yao, Leonidas Guibas, Zhicheng Wang, Howard Zhou, Pengfei Xing, Andrew Gallagher

arXiv:2609.38851v1cs.CLcs.AIcs.CVcs.LG

TL;DR

End-to-end accuracy cannot distinguish an intrinsic capability deficit from a cascading error caused by an upstream prerequisite. The paper uses controlled interventions on explicit prerequisite structures and capability and contribution metrics, then finds that correct prerequisites substantially lift cognitive-task performance while a single critical prerequisite captures most of the gain.

  • Problem

    End-to-end accuracy records failures on compositional tasks but cannot distinguish an intrinsic deficit in the target capability from a cascading error due to an upstream prerequisite.

    Source: §1 · p. 1

  • Method

    The framework intervenes on prerequisites in explicit task structures, using NC, IC, and RC for capability diagnosis and N-Score and S-Score for prerequisite contributions; CADET instantiates it across 10 tasks and 46 unit tasks.

    Source: §1 · p. 2

  • Results

    Correct prerequisites eliminate 54% of cognitive natural errors, while spatial and temporal tasks retain 72–75% of error.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Prerequisite contributions are concentrated in a few critical prerequisites, and the highest-S prerequisite alone captures 84% of full-prerequisite gain.

    Source: §1 · p. 2

    Unit-task states are identified only through interventions, because the LLM evaluation setting lacks a faithful observational readout.

    Source: §3.2 · p. 4

Abstract

from arXiv · show

End-to-end accuracy on compositional tasks records how often MLLMs fail, but cannot distinguish whether a failure reflects an intrinsic deficit in the targeted capability or a cascading error from an upstream prerequisite. We propose a causal decomposition framework that isolates these two failure modes through controlled interventions on the prerequisite dependencies of each task. Our capability metrics (NC, IC, RC) score each task under unassisted, correct, or incorrect prerequisites to diagnose where failures arise; contribution metrics (N-Score, S-Score), adapted from probabilities of causation, quantify each prerequisite's necessity and sufficiency to determine why. We instantiate the framework in CADET, a diagnostic benchmark of 10 composite tasks decomposed into 46 unit tasks with over 33,000 human-annotated questions spanning perception, spatial, temporal, and cognitive categories. Diagnosing frontier MLLMs with our framework uncovers systematic patterns that end-to-end accuracy obscures. Capability-wise, supplying correct prerequisites eliminates 54\% of errors on cognitive tasks, lifting them from weakest to above spatial and temporal. Prerequisite-wise, causal contributions are concentrated in a few critical prerequisites, and supplying the single most important one alone captures 84\% of the gain from supplying all prerequisites.

1. Introduction

End-to-end accuracy conflates intrinsic capability deficits with cascading prerequisite errors, so the introduction proposes controlled causal decomposition and the CADET benchmark. Evaluation of 12 frontier MLLMs reveals capability-specific patterns and concentrated prerequisite contributions.

1. Introduction

  • End-to-end accuracy cannot distinguish intrinsic deficits from cascading errors, and untreated subtask scores or reasoning traces show co-occurrence rather than each prerequisite’s counterfactual contribution.The introduction identifies the missing evaluation setting as one that controls every prerequisite in a shared dependency structure and measures each prerequisite’s contribution.Source: §1 · p. 1
  • In Fig. 1, supplying task-2 alone raises task-6 accuracy from 0.27 to 0.65, although task-5 has the lowest standalone accuracy.The example illustrates that the weakest standalone prerequisite need not be the one final performance turns on.Source: §1 · p. 1, Figure 1 · p. 2
  • NC, IC, and RC measure target-task performance under unassisted, correct, or incorrect prerequisites; N-Score and S-Score quantify each prerequisite’s causal necessity and sufficiency.The contribution metrics adapt probabilities of causation without assuming monotonicity and provide valid lower bounds on the probabilities of necessity and sufficiency.Source: §1 · p. 2
  • CADET uses explicit structural causal models and verified correct and plausible incorrect answers to support three-state interventions across 10 tasks, 46 unit tasks, and over 33,000 questions.Its four capability categories are perception, spatial, temporal, and cognitive; human annotators created the questions and annotations.Source: §1 · p. 2
  • Correct prerequisites eliminate 54% of cognitive natural errors, while spatial and temporal tasks retain 72–75%; the highest-S prerequisite alone captures 84% of full-prerequisite gain.The evaluation covers 12 frontier MLLMs and finds that the most sufficient prerequisite rarely coincides with the lowest-NC prerequisite.Source: §1 · p. 2

2. Related Work

Related work covers capability benchmarks, task decomposition and trace inspection, inference-time interventions, and causal attribution. The paper distinguishes its approach by intervening on task-defined prerequisite structures and measuring their contributions.

2. Related Work

  • Prior work benchmarks broad or specialized MLLM skills [48] [53] and decomposes tasks [31] [52] or inspects reasoning traces [21] [50].Some studies vary visual and textual inputs to test modality reliance [39].Source: §2 · p. 2
  • Subtask scores and traces observe unassisted behavior and co-occurrence; this work instead sets prerequisites to known correct or incorrect states.Stated chains of thought may not reflect the computation behind a final answer [5] [40].Source: §2 · pp. 2–3
  • Prior input-level interventions supply ground-truth intermediates [23] [36] or inject false premises [26], whereas this work intervenes on a task-defined causal graph shared across models.Other interventions modify model internals or single-run trajectories [25] [44] [2] [24] [35] [41].Source: §2 · p. 3
  • Prior attribution work formalizes necessity and sufficiency, but related LLM studies test model outputs or edit reasoning steps [14] [47]; this work attributes final outcomes to prerequisites.Source: §2 · p. 3

3. Causal Decomposition Framework

The framework represents task dependencies with structural causal models and uses controlled prerequisite interventions to distinguish intrinsic capability from upstream influence. It combines task-level capability measures with prerequisite-level causal contribution scores.

3.1. Task Decomposition via Structural Causal Models

  • A task decomposition SCM represents each unit-task outcome as a structural function of its parent tasks and an unmodeled noise variable.Root nodes have no parents, while the final node represents the overall task outcome.Source: §3.1 · p. 3

3.2. Causal Intervention Design

  • The ternary intervention sets a unit task to a correct answer, an incorrect answer, or its natural, unassisted state.The paper distinguishes an unassisted run from actively supplying a false answer; it identifies task states through interventions because no faithful observational readout is available.Source: §3.2 · pp. 3–4

3.3. Capability Metrics

  • NC, IC, and RC measure target-task performance with no prerequisites injected, correct prerequisites, and incorrect prerequisites, respectively.For non-root nodes, a large IC−NC gap indicates upstream cascading errors, while low IC with a small gap suggests an intrinsic deficit; comparing RC with NC tests whether false prerequisites mislead or guide reasoning.Source: §3.3 · p. 4

3.4. Contribution Metrics: N-Score and S-Score

  • N-Score and S-Score are observable lower bounds on the probabilities of necessity and sufficiency when monotonicity is not assumed.N-Score measures success harmed by degrading one prerequisite; S-Score measures failure recovered by upgrading one prerequisite. The framework evaluates four ordered state contrasts to capture correction, corruption, omission, and misinformation effects.Source: §3.4 · pp. 4–5

4. Benchmark

CADET instantiates the framework in multimodal tasks decomposed into explicit prerequisite structures and unit-task questions. Its design spans varied capabilities and reasoning structures, with newly created and quality-checked evaluation content.

Benchmark Overview

  • CADET comprises 10 multimodal tasks across over 1,100 instances, decomposed into 46 unit tasks and over 33,000 unit-task questions.Tasks cover perception, spatial, temporal, and cognitive capabilities, with correct and incorrect answers annotated to support interventions at every node.Source: §4 · p. 5

Task Design and SCM Structure

  • Task structures range from shallow aggregations to deep, multi-path cascades, and recurring iterative sub-problems are modeled as Markov chain unit tasks.A Markov chain unit task appears as one node in the task-level SCM and can be unfolded into a step-wise chain to study reasoning depth.Source: §4 · pp. 5–6

Data Construction and Quality Assurance

  • Evaluation content is newly created by raters, independently verified, and revised or removed when deficient, ambiguous, or unannotatable.Incorrect answers are randomly sampled from the valid answer space rather than manually crafted.Source: §4 · p. 6

5. Experiments and Findings

Experiments on 12 frontier MLLMs show that correct prerequisites reorder apparent capability rankings, while causal contribution scores identify sparse, high-leverage prerequisites. The analyses also find that incorrect prerequisites affect capability categories and reasoning-chain steps differently.

5.1. Experiment Setting

  • The evaluation covers 12 frontier MLLMs and reports NC, IC, and RC for unit tasks and N-Score and S-Score for overall tasks across four state contrasts.Interventions provide correct, incorrect, or no prerequisite information.Source: §5.1 · p. 6

5.2. Capability Diagnostics

  • Correct prerequisites raise cognitive accuracy from NC=0.41 to IC=0.72, eliminating 54% of cognitive failures and moving cognition above spatial and temporal.Spatial and temporal retain 75% and 72% of their errors, respectively.Source: §5.2 · p. 7
  • Across models, spatial and temporal tasks retain large intrinsic error shares, while models with similar natural accuracy can have different intrinsic and cascading error compositions.The top four models reach similar intrinsic ceilings, but their cascading losses differ by category.Source: §5.2 · p. 7
  • Incorrect prerequisites lower RC below NC for all models in perception and spatial tasks, but raise it in cognition for 10 of 12 models.Cognitive RC−NC gaps are positive; gaps vary substantially across models.Source: §5.2 · pp. 7–8

5.3. Causal Contribution Analysis

  • Across eight non-Markov tasks, prerequisite effects are uneven: each task has one or a few critical parents, while high-necessity, low-sufficiency bottlenecks are nearly absent.Critical parents can be core bottlenecks or compensable levers; other prerequisites tend to have small individual effects.Source: §5.3 · p. 8
  • SOTA models show low oracle vulnerability and often have multiple high-sufficiency prerequisites, whereas other models shift toward greater omission sensitivity and weaker single-prerequisite recovery.This split is more pronounced in Pattern 2 tasks, which average 5.25 prerequisites versus 3.0 in Pattern 1.Source: §5.3 · p. 8
  • Supplying the prerequisite with the highest S(1, ∅) recovers 84% of the all-prerequisite gain overall; in Pattern 2 it recovers 76%, compared with 36% for NC-based selection.S(1, ∅) does not track a prerequisite’s standalone NC, and alternative selection criteria do not match its overall recovery.Source: §5.3 · pp. 8–9

5.4. Markov Chain Unit Task Analysis

  • In Markov chains, capability and NS scores vary with reasoning depth, while position and action nodes respond differently to incorrect prerequisites.For cognitive action nodes, NS(0, ∅) rises from 0.12 at t=1 to 0.30 at t≥14; for spatial position nodes it stays at or below 0.06.Source: §5.4 · p. 9

6. Conclusion

The paper introduces causal decomposition and CADET to diagnose where and why MLLMs fail. Its analyses find that prerequisite effects are concentrated in a few critical tasks, and that a single selected prerequisite can recover most of the gain from providing all prerequisites.

6. Conclusion

  • The causal decomposition framework and CADET benchmark apply controlled ternary interventions over explicit prerequisite structures to diagnose where and why MLLMs fail.Capability metrics NC, IC, and RC separate cascading influence from intrinsic deficits.Source: §6 · p. 9
  • A few critical prerequisites drive each task, and supplying the most sufficient one recovers most of the gain from supplying all prerequisites, although it rarely matches the least accurate prerequisite.Source: §6 · p. 9
  • The authors suggest that these diagnostics can inform inference-time guidance, query rewriting, training, and multi-step system design.Source: §6 · p. 10

A. Details and Proofs of Causal Decomposition Framework

The appendix generalizes the contribution metrics, establishes lower-bound guarantees, and explains the controls used to interpret prerequisite effects. It also formalizes how those metrics simplify for single-parent Markov chains.

A. Details and Proofs of Causal Decomposition Framework

A.1. General Definition of N-Score, S-Score

  • For arbitrary states of the remaining prerequisites, N-Score measures the fraction of success lost when one prerequisite degrades, while S-Score measures the fraction of failure recovered when it improves.The main-text definitions are special cases using homogeneous states for the other prerequisites.Source: §A.1 · p. 14

A.2. Lower Bound Guarantee of N-Score and S-Score

  • Under exogeneity, general N-Score and S-Score provide lower bounds on probability of necessity and sufficiency for any fixed prerequisite context.The bounds use Fréchet inequalities because the paper does not assume monotonicity of LLM reasoning.Source: §A.2 · pp. 14–16

A.3. Theoretical Properties

  • Under monotonicity, the contribution scores equal probability of necessity and probability of sufficiency; the paper does not impose that assumption for LLM evaluation.Incorrect answers can act as reasoning scaffolds, so resilience can exceed natural capability.Source: §A.3.1 · p. 17

A.3.2. Remaining Prerequisite A−𝑖 Controlling

  • Because uncontrolled remaining prerequisites confound a parent’s effect with upstream model variation, the framework uses homogeneous background states and assigns N-Score and S-Score different states.This makes the scores distinct diagnostic dimensions in multi-parent tasks: N-Score measures vulnerability, while S-Score measures recovery.Source: §A.3.2 · pp. 17–18

A.4. Markov Chain Structure

  • Markov-chain tasks unfold into sequential unit tasks with a single direct parent, simplifying N-Score and S-Score to the same NS-Score, a lower bound on PNS.CADET includes three such tasks: Maze and Map Navigation with cycle length 1, and Multistep vision reasoning with cycle length 2.Source: §A.4 · pp. 18–19

B.1.1. Task Inventory and Statistics

CADET comprises decomposed visual tasks with explicit prerequisite structures and unit questions targeting distinct capabilities. The benchmark was designed and annotated through structured procedures, with ambiguous or unreliable instances removed.

B.1.1. Task Inventory and Statistics

  • The task inventory covers 10 overall evaluation tasks, with SCM depth defined as the longest directed path terminating at the overall outcome.Source: §B.1.1 · p. 19, Table 3 · p. 20

B.1.2. Capability Categories

  • Unit tasks are grouped by primary capability into perception, spatial, temporal, and cognitive categories, containing 15, 12, 10, and 9 tasks, respectively.Source: §B.1.2 · p. 20

B.1.3. Per-Task SCM and Unit Task

  • The task specifications list each unit task’s capability, answer format, question count, and parent set; CADET contains over 33,000 unit-task questions.Source: §B.1.3 · p. 20, Table 4 · p. 21

B.2. Benchmark Design Rationale

  • Task selection prioritizes unsaturated compositional tasks and real-world visual scenarios across single-image, multi-image, and video modalities.Prerequisite links represent information or capabilities required by downstream unit tasks, using the same task structure for all evaluated models.Source: §B.2 · p. 20

B.3.1. Image/Video Source and License

B.3.2. Data Construction and Quality Assurance

  • Construction first gathers overall questions and answers, then uses closed-form unit-task annotations and templates to create questions and correct and plausible incorrect answers.Two raters verify each instance; 323 of 1,467 candidates were dropped for unresolved ambiguity or unreliable unit-task annotation, leaving 1,144 instances.Source: §B.3.2 · pp. 22–23

C. Experiments Setup

The experiments evaluate 12 MLLMs across six model families using their default sampling settings and high (or default-active) thinking modes. Structured unit-task outputs are primarily checked by exact match, while three free-form tasks use a language-model judge.

C. Experiments Setup

  • 12 MLLMs from six model families are evaluated using default sampling settings and high or default-active thinking modes.The Gemini, GPT, Grok, and Gemma models use official APIs; the remaining open-weight models use OpenRouter.Source: C. Experiments Setup · p. 23
  • 43 of 46 unit tasks use structured outputs evaluated by deterministic parsing and exact match; three free-form tasks use a language-model judge.The judge is also a fallback for structured responses that deviate from the required format; exact match resolves 98.03% of structured predictions directly.Source: C. Experiments Setup · p. 23
  • Active interventions prepend prerequisite question–answer context, whereas the natural state omits that context and retains the target question and output-format instructions.Correct interventions may also render spatial ground-truth annotations on the image.Source: C. Experiments Setup · p. 23

D.1. More Capability Diagnosis Results (NC, IC, RC)

Root tasks score higher than downstream non-root tasks without assistance, while correct prerequisites substantially narrow the gap. Incorrect prerequisites lower accuracy in perception, spatial, and temporal tasks, but cognitive tasks show a modest average increase.

D.1. More Capability Diagnosis Results (NC, IC, RC)

  • Cognitive non-root accuracy rises from NC = 0.406 to IC = 0.721, while spatial and temporal IC remain below their root NC baselines.Spatial IC is 0.671 versus root NC = 0.627; temporal IC is 0.614 versus root NC = 0.714. Cognitive tasks have no root unit tasks.Source: Table 6 · p. 24, §D.1 · pp. 24–25, Figure 13 · p. 25
  • All 12 models have RC < NC in perception and spatial tasks; 10 of 12 show positive RC − NC shifts in cognitive tasks.In temporal tasks, 11 of 12 models decline; GPT-5.4n shows a +0.030 shift.Source: §D.1 · p. 25
  • 0.740 root NC versus 0.490 non-root NC across 12 models; supplying correct prerequisites raises non-root IC to 0.688.The root-to-non-root gap narrows from 0.250 to 0.052 under correct prerequisites.Source: §D.1 · p. 24, Figure 14 · p. 26
  • Incorrect prerequisites lower mean non-root accuracy overall from 0.490 to 0.435, with RC − NC = −0.144 in perception, −0.116 in spatial, and −0.046 in temporal tasks.Cognitive tasks instead shift from 0.406 to 0.445 (RC − NC = +0.040).Source: §D.1 · p. 25, Figure 15 · p. 26

D.2. More Causal Contribution Diagnosis Results (𝑁, 𝑆-Score)

Causal contribution scores distinguish the effects of omitting, corrupting, or correcting prerequisites, with patterns varying across state contrasts and model groups. The section also explains how CADET’s sequential unit tasks encode different dependencies and how their preceding trajectories are intervened on.

D.2. More Causal Contribution Diagnosis Results (𝑁, 𝑆-Score)

  • Scores are set to zero when their baseline denominator is below 0.10; this affects 3.0% (94/3,168) of scores and only two of the eight metrics.The thresholded cases are concentrated in N(0, ∅), whose baseline success can be near zero in the all-incorrect environment.Source: §D.2 · p. 26
  • In Pattern 2, SOTA models have higher single-prerequisite recovery scores than Others under (1, 0) and (∅, 0), while their N(1, 0) scores are lower.For S(1, 0), the reported ranges are 0.143–0.263 for SOTA and 0.029–0.105 for Others; for N(1, 0), they are 0.097–0.106 and 0.108–0.295, respectively.Source: §D.2 · p. 27
  • S(1, ∅) has mean pairwise inter-model Spearman correlation 0.72, compared with 0.41 for N(1, ∅).Agreement on S(1, ∅) averages 0.85 among the four SOTA models and 0.70 among Others.Source: §D.2 · p. 27
  • Rankings correlate at ρ = 0.64 between N(1, ∅) and N(1, 0), and at ρ = 0.53 between S(1, ∅) and S(1, 0).Positive S(0, ∅) values occur primarily for prerequisites feeding downstream cognitive or spatial-relational integration, including Temporal Sequencing (0.34) in Traffic Causation.Source: §D.2 · p. 28
  • For Music Note perception prerequisites, N(1, 0) exceeds N(1, ∅): corruption yields scores of 0.27–0.73, while omission yields 0.04–0.16.Under (1, 0), S is at most 0.02, compared with 0.51–0.59 under (1, ∅).Source: §D.2 · p. 26, Figure 17 · p. 29

D.3. Detail Explanation of Markov Chain Unit Task Results

  • The three Markov-chain unit tasks fall into goal-directed and process-cumulative regimes: the former depends on the remaining path, while the latter requires tracking the accumulated execution process.Maze and Map Navigation route-planning are goal-directed; Multistep Reasoning is process-cumulative.Source: §D.3 · p. 29
  • At step t, interventions provide the preceding trajectory; in Maze, correct-state prompts omit prior textual answers to avoid revealing the repeated exit answer.Incorrect-state prompts use randomly sampled wrong answers, while the natural state leaves the preceding trajectory unassisted.Source: §D.3 · p. 29

D.4.1. Ablation on Intervention Modality

Text-only correct-answer interventions approach the capability gains of interventions that also include visual annotations, while question-only injection improves accuracy less than answer provision. The visual-annotation contribution is especially pronounced for perception prerequisites.

D.4.1. Ablation on Intervention Modality

  • Text-only correct-answer intervention reaches overall accuracy 0.71 against IC = 0.74 across the two evaluated models.The condition is tested on Gemini 3.5 Flash and Gemma-4-31B across all 30 non-root unit tasks.Source: §D.4.1 · pp. 29–30
  • Text-only intervention matches IC at 0.66 in temporal tasks and differs by 0.02 in perception and spatial tasks.The largest category gap is cognitive: 0.71 versus 0.77, though text-only remains above NC = 0.44 and RC = 0.47.Source: §D.4.1 · pp. 29–30
  • Text-only S-Scores largely preserve the magnitude and ranking for spatial, temporal, and cognitive prerequisites, but fall below full-intervention scores for perception prerequisites.Across prerequisites, the corresponding N-Scores have Pearson r = 0.87.Source: §D.4.1 · p. 30

D.4.2. Ablation on Question-Only Injection

  • Question-only injection yields overall accuracy 0.58 versus NC = 0.53, while text-only answer injection reaches 0.71 and IC reaches 0.74.Mean S(q, ∅) is 0.05 across prerequisites, compared with 0.26 for S(1, ∅) on the same two models; question-only scores do not exceed 0.09 in any prerequisite category.Source: §D.4.2 · pp. 30–31

E. Discussion

The discussion clarifies that metric validity does not require exhaustive task decompositions, while emphasizing that results describe model responses to input-level interventions and prerequisite reliance under specified contrasts.

Scope and Completeness of Task Decompositions

  • Declared SCM parent sets are not exhaustive, but completeness is unnecessary for metric validity: outside factors remain natural, and guarantees apply to the specified prerequisite contexts.The lower-bound guarantees rely on assigning prerequisite regimes through injection, not on declared parent sets capturing every influence on the outcome.Source: E. Discussion · p. 31

Interpreting the Decomposition

  • IC−NC measures the gain from supplying declared prerequisites, while 1−IC measures error that persists despite correct prerequisites.Category-level findings compare resolved and persistent shares of natural error under the same inclusion criterion.Source: E. Discussion · p. 31

Interpreting Prerequisite Contributions

  • Small N- and S-Scores under (1, ∅) indicate low model reliance in that contrast, not that a declared prerequisite is spurious.Scores can be small when the model already resolves a prerequisite, other supplied prerequisites compensate for its omission, or unsupplied prerequisites limit the outcome.Source: E. Discussion · pp. 31–32

Input-level Interventions

  • Input-level interventions measure responses to prerequisite information in prompts, not internal representations; randomized incorrect answers test false-premise behavior rather than natural model errors.The same controlled states can be used across models, including closed-weight models accessed through APIs.Source: E. Discussion · p. 32

Evaluation Cost

  • 3 + 6k runs per instance cover all four contrasts, while evaluating only (1, ∅) requires 2 + 2k runs.The latter contrast is described as the most directly informative for diagnosing how much each prerequisite limits and can improve performance relative to the natural state.Source: E. Discussion · p. 33

Scope and Future Directions

  • The work establishes diagnostic evaluation; developing training or inference-time methods to address diagnosed bottlenecks remains future work.The framework is also proposed for extension to agentic workflows, post-training and reinforcement learning, and chain-of-thought analysis.Source: E. Discussion · p. 33
Loading 2609.38851v1…