PaperlayerView exact revision on arXiv

Source-linked AI summary

MM-CondChain: A Programmatically Verified Benchmark for Visually Grounded Deep Compositional Reasoning

Haozhan Shen, Shilin Yan, Hongwei Xue, Shuaiqi Lu, Xiaojun Tang, Guannan Zhang, Tiancheng Zhao, Jianwei Yin

arXiv:2603.12266v1cs.CV

TL;DR

MM-CondChain addresses the limited evaluation of MLLMs on visually grounded deep conditional reasoning, where chained visual checks determine branching or early termination. It introduces a VPIR-based agentic synthesis pipeline to construct mechanically verifiable benchmark chains across three visual domains. The strongest model achieves only 53.33 average Path F1, with performance declining on hard negatives and as chain depth or predicate complexity increases.

  • Problem

    Existing benchmarks emphasize shallow compositions, independent constraints, or single-layer contrastive changes rather than deeply chained visual conditionals.

  • Method

    A Planner incrementally constructs VPIR predicates grounded in structured visual facts, mechanically verifies them, and a Composer renders paired True-path and False-path instances.

  • Results

    53.33 average Path F1 is achieved by the strongest model, with sharp performance drops on False-path hard negatives and increasing depth or predicate complexity.

  • Takeaways & Limitations

    Deep visually grounded conditional reasoning remains a fundamental bottleneck for current MLLMs, and MM-CondChain supports fine-grained difficulty calibration through independently controllable depth and predicate complexity.

  • Takeaways & Limitations

    The benchmark assumes that each subject is uniquely localizable in the visual input so conditions can be grounded in visual evidence.

Abstract

from arXiv · show

Multimodal Large Language Models (MLLMs) are increasingly used to carry out visual workflows such as navigating GUIs, where the next step depends on verified visual compositional conditions (e.g., "if a permission dialog appears and the color of the interface is green, click Allow") and the process may branch or terminate early. Yet this capability remains under-evaluated: existing benchmarks focus on shallow-compositions or independent-constraints rather than deeply chained compositional conditionals. In this paper, we introduce MM-CondChain, a benchmark for visually grounded deep compositional reasoning. Each benchmark instance is organized as a multi-layer reasoning chain, where every layer contains a non-trivial compositional condition grounded in visual evidence and built from multiple objects, attributes, or relations. To answer correctly, an MLLM must perceive the image in detail, reason over multiple visual elements at each step, and follow the resulting execution path to the final outcome. To scalably construct such workflow-style data, we propose an agentic synthesis pipeline: a Planner orchestrates layer-by-layer generation of compositional conditions, while a Verifiable Programmatic Intermediate Representation (VPIR) ensures each layer's condition is mechanically verifiable. A Composer then assembles these verified layers into complete instructions. Using this pipeline, we construct benchmarks across three visual domains: natural images, data charts, and GUI trajectories. Experiments on a range of MLLMs show that even the strongest model attains only 53.33 Path F1, with sharp drops on hard negatives and as depth or predicate complexity grows, confirming that deep compositional reasoning remains a fundamental challenge.

1 Introduction

MM-CondChain introduces a benchmark for visually grounded deep compositional reasoning, addressing the limitations of shallow or independently constrained evaluations. Its VPIR-based synthesis pipeline constructs verifiable multi-layer chains across visual domains, while experiments show substantial difficulty for current MLLMs.

  • Prior benchmarks typically assess single-layer compositions, independent constraints, or single-layer contrastive changes rather than deep chained reasoning.
  • MM-CondChain requires multi-layer control flow in which each decision is gated by a visually verified compositional condition and may branch or terminate early.
  • The pipeline represents verified conditions alongside minimally perturbed counterfactuals, enabling chained hard negatives that can alter execution paths.
  • 53.33 average Path F1 is achieved by the strongest model, with sharper drops on False-path hard negatives and greater depth or predicate complexity.
  • VPIR decouples logical construction from language rendering and enables mechanical verification during scalable benchmark synthesis.
  • The benchmark spans natural images, data charts, and GUI trajectories and evaluates ten MLLMs on fine-grained compositional visual verification.

2 Related Work

Related work covers programmatically verifiable instruction following, compositional and logical visual reasoning, and complex multimodal instruction following. MM-CondChain is positioned around deep visual conditionals and construction-time verification.

  • Prior verifiable-instruction benchmarks use code mainly to check output-format compliance after generation.
  • Compositional and logical visual reasoning benchmarks target fine-grained evidence extraction, relations, spatial intelligence, and visual puzzles.
  • Complex visual instruction-following benchmarks evaluate layered multimodal directives and structured inference in cross-contextual environments.

3.1 Overview

The synthesis pipeline incrementally constructs verified multimodal reasoning chains before compiling them into paired evaluation instances. A Planner, Verifier, and Composer coordinate chain growth, validation, and path generation.

  • The pipeline first constructs executable Python-like VPIR predicates and renders their verified logic into natural language.
  • At each layer, it selects a visually grounded subject, extracts structured facts, generates an executable predicate, and verifies it before extending the chain.
  • The Planner decides whether to extend, terminate, or roll back while the Verifier performs quality control.
  • The Composer creates paired True-path and False-path instances, with the latter replacing one condition by a minimally perturbed counterfactual.

3.2 Layer-wise VPIR Synthesis: Facts, Strategy, and Programmatic Logic

MM-CondChain constructs each reasoning layer by selecting a relational strategy, extracting structured visual facts, and generating mechanically verifiable true and counterfactual predicates before language rendering.

  • Strategy and subject selection: Each layer selects a relational strategy that governs subject transitions, including Deepening within a subject and Transition to a related entity.The strategy taxonomy constrains candidate subject selection and inter-layer relations.
  • Structured fact extraction: The extractor jointly selects a uniquely grounded subject and produces typed key-value facts from the visual input and prior chain history.Facts may encode attributes such as color, spatial relation, count, or GUI state.
  • Programmatic logic: VPIR represents each layer with executable true-logic and counterfactual predicate programs over the extracted fact namespace.Predicates are evaluated in a sandboxed environment with whitelisted operators and deterministic boolean outputs.
  • Programmatic logic: The pipeline mechanically verifies predicate truth values before translating them into natural-language conditions, preserving minimal counterfactual changes for paired evaluation.Expression-level verification checks that the rendering is fluent, unambiguous, and faithful to VPIR semantics.

3.3 Dedicated Verifier

A dedicated Verifier provides centralized, two-stage quality control: it validates grounded facts and subjects first, then checks the natural-language realization against verified VPIR logic.

  • Verification structure: Each candidate layer is represented as a bundle containing grounded subjects, facts, true and counterfactual predicates, and their language renderings.The Verifier returns a structured verdict with pass status, reasons, and a fix hint.
  • Stage I: Fact and Subject Verification: Stage I validates visual grounding, non-repetition, relational compliance, and schema consistency before language rendering.The subject must be uniquely localizable, while facts must remain coherent and strategy-compliant.
  • Stage II: Language Realization Verification: Stage II checks semantic fidelity between rendered conditions and the verified VPIR predicates, including removal of residual code artifacts.This stage validates both true and counterfactual condition renderings.
  • Feedback-driven regeneration: Verification failures trigger stage-aware regeneration: grounded materials are rebuilt after Stage I failures, while only language is re-rendered after Stage II failures.Verified predicates and facts are retained when the failure is limited to language realization.

3.4 Planner: Verification-Aware Chain Control

The verification-aware Planner controls chain growth by choosing whether to extend, finish, or roll back, combining depth rules with MLLM-driven decisions and verifier feedback.

  • Planner decisions: At each layer, the Planner maps chain history to an action and relational strategy that govern the next synthesis step.The action decision is written as (a_t, r_t) = π(H_t−1).
  • Planner decisions: The Planner can EXTEND with a new layer, FINISH and compose the chain, or ROLLBACK to a verified prefix.ROLLBACK discards the most recent non-seed layer.
  • Hybrid depth control: The policy combines hard-coded depth control with an MLLM-based choice based on chain coherence and remaining synthesis potential.The planner operates within a target depth interval [d_min, d_max].
  • Verifier coupling: Repeated verification failures cause rollback, allowing synthesis to resume from the last verified prefix instead of remaining in an unrecoverable state.Examples include persistent subject repetition and unsatisfiable relational constraints.

3.5 Composition: Paired-Path Instruction Compilation

The Composer turns verified control-flow skeletons into evaluation-ready paired instances by attaching exit questions, preventing subject leakage, and contrasting true and minimally perturbed false paths.

  • Skeleton and questions: A finalized skeleton contains grounded subjects and true or counterfactual conditions for each layer, with questions attached to every possible exit point.The terminal layer uses q_fin, while intermediate exits use auxiliary questions.
  • Subject de-leakage: The Composer uses subject de-leakage to remove condition-revealing attributes while preserving uniquely identifying visual descriptions.Alternative descriptors such as spatial location may replace leaked attributes.
  • Paired-path instantiation: Paired-path instantiation creates a True-path where all conditions hold and a False-path where one condition is replaced at a sampled divergence layer.The false path terminates early and uses the auxiliary question for that exit.
  • Paired-path instantiation: The resulting near-isomorphic hard negatives differ by one subtly perturbed condition, requiring fine-grained reasoning rather than superficial pattern matching.The paired instances share overall structure and nearly identical wording.

3.6 Domain-Specific Instantiation

MM-CondChain uses a unified VPIR framework across natural images, charts, and GUI trajectories, with domain-specific preprocessing isolated to input adaptation and fact extraction.

  • 3.6 Domain-Specific Instantiation: The VPIR synthesis pipeline is domain-agnostic, while domain-specific adaptations are confined to preprocessing and fact extraction.Each modality enters the unified engine through its own input normalization.
  • 3.6 Domain-Specific Instantiation: Natural images require no preprocessing because the MLLM directly extracts open-schema visual attributes from raw images.Extracted attributes include color and spatial relations.
  • 3.6 Domain-Specific Instantiation: Charts undergo deterministic CSV alignment and LLM-based value extraction to repair annotation inconsistencies before engine invocation.These steps produce clean meta_json from mismatched lengths, null placeholders, and missing entries.
  • 3.6 Domain-Specific Instantiation: GUI trajectories require completeness checks, structured action parsing, and multi-image sequence input to represent actions, targets, locations, and states.The pipeline verifies screenshot counts against annotation lengths before processing trajectories.
  • 3.6 Domain-Specific Instantiation: VPIR predicate generation, two-stage verification, and Planner backtracking remain unchanged across visual modalities.Domain-specific code is isolated to input adapters, fact builders, and strategy registries.

4 Evaluation

The evaluation spans three domains and examines model performance, visual-fact usage, logical-pattern diversity, and controlled difficulty. Results show that current MLLMs struggle with balanced path following, deeper chains, and more complex predicates.

  • 4 Evaluation: 398 Natural images, 200 Chart images, and 377 GUI trajectories form the three-domain benchmark evaluation set.The GUI domain contains 3,421 screenshots, averaging 9.07 frames per trajectory.
  • 4 Evaluation: VPIR selectively reuses subsets of extracted attributes rather than mirroring full fact distributions, making structured composition central to benchmark difficulty.Natural instances emphasize object attributes and spatial relations, Charts numerical and structural statistics, and GUI action, state, and trajectory metadata.
  • 4 Evaluation: 128 unique VPIR templates are needed for 80% coverage, while the top 20 cover only 50.07%, indicating broad structural diversity.Dominant templates can combine multiple predicates, nested logical operators, executable program form, and natural-language renderings.
  • 4.2 Main Results: Gemini-3-Pro achieves the best overall result with 53.33 average Path F1, followed by GPT-5-0807 at 50.34.Path F1 balances True-path and False-path accuracy across the three domains.
  • 4.2 Main Results: Models often score substantially higher on True-path than False-path instances, favoring continuation when conditions may be violated.For example, GPT-4o-1120 scores 83.92 versus 12.81 on Natural, while Qwen3.5-9B scores 91.69 versus 13.10.
  • 4.2 Main Results: GUI is the most challenging domain overall, with a best F1 of 40.19 versus 55.91 on Natural and 66.04 on Chart.GUI requires multi-frame trajectory, action, and interface-state reasoning, whereas many Chart conditions reduce to grounded numerical comparisons.
  • 4.3 Ablations: From depth 2 to depth 6, Path F1 drops by approximately 29–33% in relative terms across tested models.Increasing predicate complexity separately causes relative degradation ranging from 27.7% to 36.0% at fixed depth.
  • 4.3 Ablations: Chain depth and intra-layer predicate complexity are independently controllable difficulty axes that enable fine-grained diagnosis of model failures.Increasing either dimension consistently degrades performance across tested models.

5 Conclusion

MM-CondChain benchmarks visually grounded deep conditional reasoning through multi-layer, visually gated control flow. Its VPIR-centered synthesis provides deterministic ground truth and hard negatives, while experiments show persistent model bottlenecks as depth or predicate complexity increases.

  • 5 Conclusion: MM-CondChain evaluates multi-layer control flow in which every decision is gated by a visually verifiable condition.The benchmark targets deep conditional reasoning rather than shallow compositions or independent constraints.
  • 5 Conclusion: VPIR decouples logic formation from language rendering and supports scalable construction with deterministic ground truth and near-isomorphic hard negatives.The synthesis pipeline produces verified benchmark instances across three visual domains.
  • 5 Conclusion: Experiments across three visual domains show that state-of-the-art MLLMs struggle as chain depth or predicate complexity increases.The paper positions MM-CondChain as a resource for diagnosing model weaknesses and supporting future multimodal reasoning research.
Loading 2603.12266v1…