Source-linked AI summary
AdaptMMBench: Benchmarking Adaptive Multimodal Reasoning for Mode Selection and Reasoning Process
Xintong Zhang, Xiaowen Zhang, Jingrong Wu, Zhi Gao, Shilin Yan, Zhenxin Diao, Kunpeng Gao, Xuanyan Chen, Yuwei Wu, Yunde Jia, Qing Li
TL;DR
Adaptive multimodal reasoning evaluation must distinguish appropriate mode selection from answer correctness and assess reasoning processes. AdaptMMBench addresses this gap, finding that high accuracy does not necessarily imply strong mode selection and that suboptimal tool invocation often limits performance.
Problem
Existing adaptive reasoning evaluation must separate mode appropriateness from answer correctness and assess process validity, since correct answers can mask logical errors or improper tool use.
Method
AdaptMMBench identifies tool-required versus tool-redundant cases and evaluates mode selection, reasoning processes, and efficiency across adaptive multimodal tasks.
Results
High accuracy does not necessarily imply strong mode selection, and adaptive reasoning trails oracle-visual reasoning, indicating suboptimal tool invocation often limits performance.
Takeaways & Limitations
Adaptive tool selection remains a key challenge for future multimodal reasoning models.
Abstract
from arXiv · showhide
Adaptive multimodal reasoning has emerged as a promising frontier in Vision-Language Models (VLMs), aiming to dynamically modulate between tool-augmented visual reasoning and text reasoning to enhance both effectiveness and efficiency. However, existing evaluations rely on static difficulty labels and simplistic metrics, which fail to capture the dynamic nature of difficulty relative to varying model capacities. Consequently, they obscure the distinction between adaptive mode selection and general performance while neglecting fine-grained process analyses. In this paper, we propose AdaptMMBench, a comprehensive benchmark for adaptive multimodal reasoning across five domains: real-world, OCR, GUI, knowledge, and math, encompassing both direct perception and complex reasoning tasks. AdaptMMBench utilizes a Matthews Correlation Coefficient (MCC) metric to evaluate the selection rationality of different reasoning modes, isolating this meta-cognition ability by dynamically identifying task difficulties based on models' capability boundaries. Moreover, AdaptMMBench facilitates multi-dimensional process evaluation across key step coverage, tool effectiveness, and computational efficiency. Our evaluation reveals that while adaptive mode selection scales with model capacity, it notably decouples from final accuracy. Conversely, key step coverage aligns with performance, though tool effectiveness remains highly inconsistent across model architectures.
2. Related Work
Early VLMs used text-only reasoning over fixed visual encodings, whereas newer models increasingly use tools for active multimodal information seeking. Existing benchmarks cover structured domains and reasoning correctness, while static difficulty levels overlook that difficulty depends on model capabilities.
- Multimodal reasoning: Early VLMs relied on text-only reasoning over fixed visual encodings, creating a “first-glance” bottleneck that limited access to fine-grained visual details.The passage contrasts this limitation with recent movement toward active, tool-augmented information seeking.
- Multimodal reasoning: Recent models, including GPT-5, Qwen3-VL, and InternVL, have shifted multimodal reasoning toward active, tool-augmented information seeking.This shift moves beyond passive visual interpretation.
- Benchmarking: Traditional VLM benchmarks emphasize structured domains with coarse visual content, including chart understanding, mathematical problem solving, and general-purpose VQA.The cited examples include chart understanding, mathematical problem solving, and other general-purpose visual question answering.
- Benchmarking: MME-CoT additionally evaluates the correctness of the text reasoning process, while existing methods rely on static difficulty levels despite difficulty being model-dependent.The static-versus-model-dependent difficulty distinction is illustrated in Figure 2.
3. AdaptMMBench
AdaptMMBench evaluates adaptive reasoning through both mode selection and reasoning-process analysis, using multimodal samples annotated with visual-tool requirements and verified solution steps. Its 1,420 samples span five domains and deliberately mix difficulty levels without prescribing the evaluation-time reasoning mode.
- AdaptMMBench evaluates two perspectives: adaptive reasoning mode selection and reasoning process.
- Each sample pairs an image, query, and answer with visual-tool annotation E and an ordered sequence K of human-verified key reasoning steps.E specifies target-region coordinates and transformations such as rotation and contrast adjustment, while K describes the solution path from (I, Q) to A.
- During inference, models observe only the image and query, invoking predefined visual tools through code execution or function calls when needed.A tool is represented as t(I, τ), where t is selected from the toolset and τ contains its execution arguments.
- 1,420 samples span five domains: real-world, OCR, GUI, math, and knowledge.The benchmark includes both direct perception and reasoning scenarios across diverse domains.
- Approximately 24%, 70%, and 6% of samples are respectively text-only solvable by Qwen2.5-VL-7B, solved adaptively by Qwen3-VL-235B after Qwen2.5-VL-7B fails, or unsolved by Qwen3-VL-235B.These subsets ensure difficulty diversity only; they do not determine the ground-truth reasoning mode, which the model selects during evaluation.
4. Evaluation Strategy
The evaluation strategy separates text-only, adaptive, and oracle-visual reasoning modes, then independently assesses mode-selection rationality and reasoning-process quality. It combines MCC-based calibration with metrics for key-step coverage, tool effectiveness, and computational efficiency.
- Evaluation Modes: Three evaluation modes compare text-only reasoning, adaptive tool selection, and text reasoning over gold-standard visual evidence.Text-Reasoning Mode provides a baseline for tool necessity, Adaptive Reasoning Mode records tool invocations and reasoning trajectories, and Oracle-Visual Mode estimates performance under perfect visual acquisition.
- Adaptive Mode Selection: Mode appropriateness is evaluated independently of answer correctness by labeling tasks Tool-Redundant or Tool-Required according to text-only solvability.Text-solvable tasks are Tool-Redundant, while tasks unsolvable through text reasoning alone are Tool-Required because visual tools provide necessary information.
- Adaptive Mode Selection: MCC evaluates adaptive mode-selection agreement despite model-dependent class imbalance, ranging from −1 for complete misalignment to 1 for perfect agreement.An MCC of 0 denotes chance-level performance, and a small constant ϵ is used for numerical stability.
- Adaptive Mode Selection: Only 0.02 of samples show inconsistent outcomes between text reasoning and adaptive text-only reasoning, indicating stable performance under prompt variations.Adaptive reasoning rarely degrades samples solvable through text reasoning alone.
- Reasoning Process Evaluation: Reasoning-process evaluation measures key-step coverage, tool effectiveness, and efficiency to assess coherence, execution fidelity, and computational cost beyond answer correctness.Key-step coverage tracks alignment with annotated solution structure, tool effectiveness judges semantic appropriateness and execution errors, and efficiency considers token numbers, reasoning turns, and tool frequency.
5. Experiments
Experiments evaluate adaptive multimodal reasoning through mode selection, reasoning-process quality and efficiency, and accuracy across reasoning modes. Results show that scaling improves selection and process quality, while imperfect tool use remains a major performance limitation.
- Experimental Setup: The evaluation covers mode selection, reasoning-process quality and efficiency, and accuracy across reasoning modes on AdaptMMBench.Evaluated models include GPT-5, Gemini3, Qwen3-VL at 8B, 32B, and 235B, and specialized adaptive reasoning models.
- Mode Selection: Mode selection is weakly coupled with accuracy: AdaptVision reaches MCC 0.17 despite modest accuracy, while GPT-5 achieves the highest MCC of 0.41.AdaptVision’s MCC exceeds that of other models trained on Qwen2.5-VL-7B backbones.
- Mode Selection: Larger Qwen3-VL models show more reliable mode selection, and large-scale closed-source models outperform open-source models.The results suggest increased capacity improves calibration about when tool-based reasoning is necessary.
- Mode Selection: Imbalanced tool invocation lowers mode-selection performance: Deepeyes v2 uses tools in all but one of 1,300 samples, whereas Thyme triggers tools in about 3% of cases.These imbalanced behaviors occur despite competitive accuracy.
- Reasoning Process: Key step coverage and tool effectiveness generally improve with model scale, but tool effectiveness varies across architectures because repeated, unnecessary, or complex code-based calls can reduce effectiveness.Qwen3-VL-235B is among the top models for key step coverage, while Qwen3 models show strong tool performance.
- Accuracy and Efficiency: Adaptive reasoning consistently outperforms text-only reasoning, yet GPT-5 rises from 78.69% adaptive accuracy to 88.69% with oracle tools, indicating substantial invocation headroom.Token usage does not necessarily decrease with fewer reasoning steps or tool calls; on auxiliary-line tasks, adaptive gains are limited or negative without visual generation.
6. Conclusion
AdaptMMBench evaluates adaptive multimodal reasoning in VLMs across diverse domains and scenarios, using model-dependent case identification and metrics for mode selection, reasoning processes, and efficiency. Its evaluation shows that accuracy does not necessarily indicate strong mode selection capability.
- Benchmark scope: AdaptMMBench benchmarks adaptive multimodal reasoning in VLMs across diverse domains and reasoning scenarios.The benchmark evaluates adaptive multimodal reasoning rather than only final task accuracy.
- Mode-dependent evaluation: The benchmark identifies tool-redundant and tool-required cases by comparing performance across reasoning modes.This comparison enables model-dependent identification of whether tool use is needed.
- Evaluation metrics: AdaptMMBench assesses mode selection quality, reasoning process quality, and efficiency with dedicated metrics.These metrics extend evaluation beyond final-answer performance.
- Main finding: High accuracy does not necessarily imply strong reasoning mode selection capability.The systematic evaluation of state-of-the-art models separates final accuracy from adaptive mode selection ability.
A. More Data Details · A.1. Data Source Distribution · A.2. Data Construction Pipeline
AdaptMMBench spans five multimodal VQA domains assembled from established and specialized datasets, covering perception, OCR, GUI, knowledge, and mathematical reasoning. Its construction separates tool-dependent from text-only tasks, adds visual transformations, and applies multi-stage expert and human verification.
- A.1. Data Source Distribution: Real-World VQA targets high-resolution natural scenes through VisualProbe for small-object search and a custom SA-1B subset for large-scale object reasoning.Queries test attributes, spatial relations, counting, physical state, and text recognition across distinct scales.
- A.1. Data Source Distribution: Text-Rich VQA combines ChartQA, DocVQA, ChartQA-Pro, MM-RealWorld, and Insight-o3 to require precise visual inspection and deep reasoning over charts, tables, and documents.
- A.1. Data Source Distribution: Math VQA consolidates samples from MathVista, MathVerse, We-Math, LogicVista, Visulogic, AuxSolidMath, and VTBench for mathematical reasoning in visual contexts.
- A.1. Data Source Distribution: GUI VQA covers iOS, Android, Web, macOS, Windows, and Linux by integrating GUI-Knowledge-Bench, MMBench-GUI, and WebWalker.
- A.1. Data Source Distribution: Knowledge VQA draws expert-level Physics, Chemistry, and Biology samples from MMMU and SciVerse to evaluate specialized knowledge integration with visual reasoning.
- A.2. Data Construction Pipeline: The construction workflow first partitions raw data by reasoning complexity, separating tasks requiring external tools from those suitable for text-only inference.
- A.2. Data Construction Pipeline: Visual inputs receive diverse transformations that mandate fine-grained perception and further challenge model adaptability.
- A.2. Data Construction Pipeline: A multi-stage verification pipeline uses expert annotation of transformation logic and key reasoning steps, followed by rigorous human review for high-fidelity benchmark ground truth.
B. Transform Results
The transformed subset is evaluated separately to assess robustness to data variations, reporting accuracy across five domains and aggregate results while prioritizing adaptive performance.
- Transform subset: Table 5 isolates accuracy on the Transformed subset across five domains and overall aggregates, complementing the original and aggregated results in Table 3.This separate analysis provides a comprehensive view of robustness against data variations.
C. Category Results … LLM Judge Prompt
The paper reports category-level accuracy across five domains, then specifies reasoning-mode prompts for text and oracle-visual settings and an LLM judge that checks answer consistency. The category results are divided between GUI/Realworld and Knowledge/Math/OCR tables, with enhancement-supporting models marked by an asterisk.
- C. Category Results: Category-level accuracy is presented in two detailed tables covering all benchmark sub-domains.The section separates results to accommodate the wide range of sub-domains.
- C. Category Results: Table 6 reports GUI and Realworld performance metrics.It is the first part of the detailed accuracy results.
- C. Category Results: Table 7 reports Knowledge, Math, and OCR performance metrics.It is the second part of the detailed accuracy results.
- C. Category Results: The category tables use N for the number of test samples and report accuracy values in decimal format.An asterisk indicates that a model supports enhancement operations.
- D. Reasoning Mode Prompt: The reasoning-mode section introduces the detailed prompts used in the experiments.These prompts define the formats for the subsequent text- and visual-reasoning settings.
- Text-Reasoning Mode Prompts: Text-reasoning prompts require step-by-step reasoning followed by a formatted final answer for multiple-choice and short-answer questions.Multiple-choice outputs use an option letter, while short-answer outputs use a single word or phrase.
- Oracle-Visual Mode Prompts: Oracle-visual prompts provide an original image together with a zoomed region, transformed enhancement, or auxiliary lines.Each setting asks for step-by-step reasoning and a single-word-or-phrase answer.
- LLM Judge Prompt: The LLM judge compares a model-predicted answer with the ground-truth answer and treats essentially equivalent meanings as consistent.The prompt gives “pink” and “it is pink” as an example of consistency.
E. Error Analysis
The error analysis identifies visual reasoning failures as the largest reported category, alongside context noise, forced-tool corrections, and degradation from incorrect mode selection. Representative cases show how transformation, region-selection, contextual, and tool-use errors alter intermediate reasoning and final predictions.
- Visual Reasoning Failures: 42.3% of errors stem from inability to correctly manipulate or locate visual information.The analysis presents representative visual reasoning failure scenarios.
- Visual Reasoning Failures: Incorrect region selection led the model to reason validly from irrelevant evidence by zooming into Question 235 instead of Question 238.This represents a spatial grounding failure in a dense document.
- Context Noise in Multi-step Reasoning: 7.3% of cases involved context noise overriding correct visual perception during multi-step reasoning.The model correctly enhanced the image and identified “two” objects, but accumulated context induced a hallucinated negation.
- Correction via Forced Tool Invocation: 7.0% of corrected errors were rectified by forcing tool invocation after an initial estimation failure.The example begins with an incorrect “Cerulean Blue” answer based on imprecise visual intuition before adopting a programmatic pixel-level strategy.
- Performance Degradation due to Incorrect Mode Selection: 8.3% of cases reflected performance degradation from unnecessarily invoking tools for tasks solvable by direct visual inspection.The adaptive mode replaced straightforward OCR or visual recognition with an unreliable OpenCV edge-detection approach for icon counting.
F. Process Evaluation Example … Key Step Coverage Prompt
The paper illustrates process-quality assessment through key step coverage and tool effectiveness examples, then standardizes both evaluations with specialized LLM-judge prompts. These prompts audit tool-call correctness and intent alignment while checking reasoning completeness against annotated ground truth.
- F. Process Evaluation Example: Detailed cases in Figures 14 and 15 illustrate the benchmark’s assessments of key step coverage and tool effectiveness.The examples are presented to clarify the process-reasoning quality evaluation protocol.
- G. Process Evaluation Prompt: Specialized LLM-based judges provide a reproducible and standardized framework for auditing adaptive reasoning processes.The protocol uses dedicated prompts for tool invocation effectiveness and key step coverage.
- Tool Invocation Effectiveness Prompt: The tool invocation effectiveness prompt evaluates each tool call for functional correctness and alignment with the model’s stated reasoning intent.The audit is conducted within the model’s multi-step reasoning process using the available tools.
- Key Step Coverage Prompt: Key step coverage verifies whether the model’s reasoning trajectory is logically complete relative to annotated ground truth.The prompt is designed for solutions to image-based problems and begins by dividing the solution into distinct logical reasoning steps.
- Tool Invocation Effectiveness Prompt: The tool-effectiveness audit treats each reasoning step as part of a visual-reasoning trajectory and compares tool usage with the stated intent.The prompt frames the judge as a visual reasoning audit expert with strong logical analysis skills.
- Key Step Coverage Prompt: The key step coverage prompt segments a solution into logical reasoning steps and matches ground-truth middle steps to the solution steps.Its inputs are the original problem, a continuous-paragraph solution, and the essential steps required for a correct answer.