PaperlayerView exact revision on arXiv

Source-linked AI summary

Unveiling the Compositional Ability Gap in Vision-Language Reasoning Model

Tianle Li, Jihai Zhang, Yongming Rao, Yu Cheng

arXiv:2505.19406v1cs.AI

TL;DR

The paper addresses whether VLMs can compose independently learned abilities across modalities and tasks under out-of-distribution conditions. Using ComPABench and comparisons of SFT, RL, and SFT-initialized RL, it finds that RL generalizes better compositionally, while RL-Ground’s caption-before-reasoning and progress rewards yield strong gains. The study concludes that visual-to-text alignment and grounded supervision are important for multimodal compositional reasoning.

  • Problem

    Whether VLMs can compose independently learned skills across modalities and reasoning domains under out-of-distribution conditions remains underexplored.

  • Method

    The paper introduces ComPABench, paired pure-text and multimodal diagnostic tasks, and controlled comparisons of SFT, RL, and SFT-initialized RL.

  • Results

    RL-trained models outperform SFT compositionally, but multimodal composition remains difficult; RL-Ground improves it by combining caption-before-reasoning with progress rewards.

  • Takeaways & Limitations

    Visual-to-text alignment and accurate grounding of intermediate reasoning are supported ingredients for improving compositional generalization in VLMs.

  • Takeaways & Limitations

    The benchmark uses controlled synthetic geometric and spatial tasks, so its coverage of real-world multimodal complexity and natural-image domain shift remains limited.

Abstract

from arXiv · show

While large language models (LLMs) demonstrate strong reasoning capabilities utilizing reinforcement learning (RL) with verifiable reward, whether large vision-language models (VLMs) can directly inherit such capabilities through similar post-training strategies remains underexplored. In this work, we conduct a systematic compositional probing study to evaluate whether current VLMs trained with RL or other post-training strategies can compose capabilities across modalities or tasks under out-of-distribution conditions. We design a suite of diagnostic tasks that train models on unimodal tasks or isolated reasoning skills, and evaluate them on multimodal, compositional variants requiring skill integration. Through comparisons between supervised fine-tuning (SFT) and RL-trained models, we identify three key findings: (1) RL-trained models consistently outperform SFT on compositional generalization, demonstrating better integration of learned skills; (2) although VLMs achieve strong performance on individual tasks, they struggle to generalize compositionally under cross-modal and cross-task scenario, revealing a significant gap in current training strategies; (3) enforcing models to explicitly describe visual content before reasoning (e.g., caption-before-thinking), along with rewarding progressive vision-to-text grounding, yields notable gains. It highlights two essential ingredients for improving compositionality in VLMs: visual-to-text alignment and accurate visual grounding. Our findings shed light on the current limitations of RL-based reasoning VLM training and provide actionable insights toward building models that reason compositionally across modalities and tasks.

1 Introduction

The paper probes whether VLMs can compose independently learned abilities across modalities, tasks, and distribution shifts. It introduces a diagnostic benchmark and finds that RL improves compositional generalization, while visual-to-text alignment and grounded intermediate rewards further improve multimodal reasoning.

  • The paper targets an open gap: gains from RL on isolated vision-language tasks do not establish whether VLMs can integrate independently acquired skills in complex scenarios.
  • The study asks whether textual reasoning can transfer to visual inputs, visual skills can combine across domains, and these abilities can generalize to OOD objectives.
  • RL-trained models consistently outperform SFT in compositional settings, especially for cross-task generalization, despite VLMs remaining limited under multimodal composition.
  • ComPABench systematically evaluates VLM compositional generalization across modalities, reasoning tasks, and distribution shifts.
  • RL-Ground reduces the compositional gap by captioning visual content before reasoning and rewarding accurate grounding during intermediate reasoning steps.

2 Related Work

Related work motivates extending RL-based reasoning from language models to VLMs while highlighting uncertainty about the generalization benefits of SFT and RL. Prior approaches include structured reasoning, search, and outcome-based reward training, but directly transferring LLM methods to VLMs has proven ineffective.

  • Recent VLM research adapts reasoning-oriented LLM methods, including manually structured paths, tree search, and outcome-based reward RL.
  • Directly transplanting DeepSeek-R1’s training methodology to VLMs has proven ineffective, motivating specialized multimodal post-training strategies.
  • Comparative studies report that SFT often memorizes training patterns, whereas RL can encourage more transferable principles and stronger generalization.

3 Preliminaries

The paper formalizes SFT, RL with verifiable rewards using GRPO, and SFT-initialized RL as its three training paradigms for probing compositional VLM reasoning. These objectives and hybrid initialization establish the experimental pipeline for evaluating multimodal, compositional, and generalization-intensive settings.

  • Training paradigms: The study compares three paradigms: supervised fine-tuning, reinforcement learning with verifiable reward, and RL initialized from an SFT-trained checkpoint.These paradigms define the backbone of the experimental pipeline.
  • Supervised Fine-Tuning: SFT minimizes the negative log-likelihood of target output sequences paired with text-only or multimodal inputs.Outputs contain both a reasoning trace in <think> and a final response in <answer> blocks.
  • Reinforcement Learning with GRPO: GRPO optimizes token-level generation using per-sample reward-to-advantage estimates while regularizing the policy toward a reference policy.The reward may combine answer correctness and format adherence, while the KL term controls divergence from the reference distribution.
  • Reinforcement Learning with GRPO: In GRPO, the advantage is estimated at each token step and the KL coefficient β controls the strength of regularization.The inner term scales policy gradients using a reward signal modulated across token positions.
  • SFT-initialized RL: SFT-initialized RL inherits both policy parameters and the KL reference policy from the SFT checkpoint.This hybrid strategy is introduced to leverage prior task-distribution alignment and accelerate convergence.

4 Experiments

Experiments with ComPABench show that VLMs struggle to compose reasoning across modalities, tasks, and OOD objectives, while RL and RL-Ground improve compositional generalization relative to SFT.

  • 4.1 Benchmark: ComPABench evaluates compositional ability through cross-modal, cross-task, and OOD experiments using controlled task variants.The benchmark pairs pure-text and multimodal tasks and includes individual, compositional, and OOD evaluations.
  • RQ1: Cross-Modality Composition: Pure-text reasoning does not automatically transfer to images: RL improves multimodal shape-area accuracy from 20.8% to 28.0% for 7B, but performance remains far below pure-text results.Pure-text initialization helps further, increasing 3B grid-position accuracy from 49.6% to 64.4%.
  • RQ2: Cross-Task Composition: SFT severely impairs compositional reasoning despite near-perfect isolated skills, reducing pure-text compositional accuracy to 0.6% (3B) and 2.2% (7B).SFT-initialized RL remains extremely low at 2.6% (3B) and 1.0% (7B), suggesting that the hybrid setup can preserve flawed compositional strategies.
  • RQ2: Cross-Task Composition: Multimodal RL reaches 17.4% (3B) and 31.2% (7B), improving over multimodal SFT but remaining far below pure-text RL levels.This gap highlights the difficulty of cross-task compositional reasoning with visual inputs.
  • RQ2: Cross-Task Composition: RL-Ground achieves 28.6% (3B) and 52.8% (7B), surpassing the other evaluated post-training strategies on multimodal compositional reasoning.The method combines caption-before-thinking with fine-grained progress rewards for intermediate vision-grounded reasoning.
  • RQ3: OOD Generalization: RL-Ground consistently achieves high accuracy across OOD tasks, including 83.2% (3B) and 88.6% (7B) on the farthest-shape task and 38% (3B) and 52.8% (7B) on OOD composition.It shows the strongest and most stable generalization across individual and compositional OOD settings.

5 Conclusion

The paper introduces ComPABench to evaluate VLM compositionality across cross-modal, cross-task, and OOD settings. It finds that RL integrates independently learned skills better than SFT, while RL-Ground improves compositional reasoning through structured prompting and grounded supervision.

  • Contributions: ComPABench evaluates VLM compositional ability across cross-modal, cross-task, and OOD settings.The benchmark supports comparisons of SFT, RL, and SFT-initialized RL.
  • Findings: RL better integrates independently learned skills than SFT, especially in cross-task and OOD scenarios, although visual compositional reasoning remains challenging.RL-Ground combines caption-before-reasoning with progress rewards and yields strong in-distribution and OOD gains.

A.1 Benchmark Details

The benchmark uses synthetic image-and-question datasets for geometric, spatial, and compositional reasoning, with pure-text counterparts and OOD variants to test modality transfer and objective shifts.

  • Dataset Curation: The benchmark contains Shape Area, Grid Position, and Area-Position Composition tasks built from synthetic images, questions, reasoning paths, and final answers.The compositional task fuses geometric and spatial reasoning by combining area computation with nearest-neighbor selection.
  • Shape Area Task: Shape Area requires computing a queried shape’s area from images containing 2 to 6 labeled geometric shapes.Shapes include squares, rectangles, right triangles, and trapezoids, with symbolic formula reasoning and rounded-integer answers.
  • Grid Position Task: Grid Position requires identifying the grid index of the shape closest to a target using Manhattan distance on 3 × 3 to 10 × 10 grids.Each image contains 2 to 6 non-overlapping shapes placed in discrete grid cells.
  • Dataset Statistics: Each individual task has 4K training samples and 500 evaluation samples, while the compositional setting has 500 evaluation samples.Images are rendered at 512×512 resolution or higher.
  • Variants: Pure-text counterparts describe visual attributes directly in natural language while preserving the multimodal tasks’ reasoning objectives and answer formats.OOD variants change the objectives, such as largest-area selection, farthest-shape selection, and altered compositional comparisons.

A.2 Ablation Study on RL-Ground

The ablation shows that combining caption formatting with progress rewards yields the strongest compositional performance, while each component provides more targeted gains.

  • A.2 Ablation Study on RL-Ground: Caption formatting raises Shape Area from 74.6% to 87.4% and Grid Position from 83.2% to 84.6%, but slightly underperforms on composition.Explicitly verbalizing the visual scene improves perceptual grounding, whereas formatting alone may not support compositional multi-step reasoning.
  • A.2 Ablation Study on RL-Ground: 39.6% compositional accuracy from progress rewards exceeds the base RL setup's 31.2%, indicating stronger intermediate reasoning support.Progress rewards also moderately improve Grid Position.
  • A.2 Ablation Study on RL-Ground: 52.8% compositional accuracy and 88.4% Grid Position are achieved by full RL-Ground, despite lower Shape Area performance.The full configuration combines image-to-text conversion with progress-based reward supervision.

Limitations

The study is limited by its reliance on synthetic geometric and spatial tasks, structured intermediate signals, and unresolved transfer to natural multimodal settings.

  • Limitations: Synthetic shape-area, spatial-position, and composition tasks may not capture real-world visual complexity or ambiguity.The domain shift from synthetic data to natural images remains unaddressed.
  • Limitations: ComPABench focuses on geometric and spatial composition, leaving causal and commonsense multimodal reasoning outside its coverage.This focused design may limit conclusions about broader reasoning contexts.
  • Limitations: RL-Ground relies on structured captions and progress rewards, making application to open-ended real-world tasks less straightforward.Such settings may require unsupervised or new intermediate-signal mechanisms.

Broader Impacts

The work frames improved VLM compositional reasoning as useful for complex visual-input applications while warning that the same capabilities can increase misuse risks.

  • Broader Impacts: Improved VLM reasoning could benefit educational tools, scientific assistants, and other applications requiring complex reasoning over visual inputs.The benchmark is presented as a diagnostic and research resource for improving transparency and robustness.
  • Broader Impacts: More capable visual reasoning could also support persuasive visual misinformation or unsupervised automation in sensitive contexts.The authors encourage responsible use of the findings and dataset.
Loading 2505.19406v1…