Source-linked AI summary
Progressive Multimodal Reasoning via Active Retrieval
Guanting Dong, Chenghao Zhang, Mengjie Deng, Yutao Zhu, Zhicheng Dou, Ji-Rong Wen
TL;DR
Multimodal reasoning with multiple steps remains challenging for MLLMs, and adaptation of MCTS to these settings is underexplored. AR-MCTS combines active retrieval with MCTS to generate step-wise annotations and align a process reward model. The paper reports effectiveness across multimodal benchmarks and models, with optimized sampling diversity and verification accuracy.
Problem
MCTS adaptation to MLLMs remains underexplored, while multimodal misalignment can make internal-knowledge-based path expansion unreliable across dependent reasoning steps.
Source: §1 · p. 1Method
AR-MCTS uses active retrieval during MCTS expansion to obtain step-wise reasoning annotations and progressively align a process reward model for process-level verification.
Source: §1 · p. 2Results
Takeaways & Limitations
The framework provides a proposed approach to reliable multimodal reasoning through active retrieval, MCTS, and process-level verification.
Source: §6 · p. 9MCTS-based annotation requires significant computational resources, although AR-MCTS costs substantially less than manual annotation.
Source: D. Limitations and Future Work · p. 23
Abstract
from arXiv · showhide
Multi-step multimodal reasoning tasks pose significant challenges for multimodal large language models (MLLMs), and finding effective ways to enhance their performance in such scenarios remains an unresolved issue. In this paper, we propose AR-MCTS, a universal framework designed to progressively improve the reasoning capabilities of MLLMs through Active Retrieval (AR) and Monte Carlo Tree Search (MCTS). Our approach begins with the development of a unified retrieval module that retrieves key supporting insights for solving complex reasoning problems from a hybrid-modal retrieval corpus. To bridge the gap in automated multimodal reasoning verification, we employ the MCTS algorithm combined with an active retrieval mechanism, which enables the automatic generation of step-wise annotations. This strategy dynamically retrieves key insights for each reasoning step, moving beyond traditional beam search sampling to improve the diversity and reliability of the reasoning space. Additionally, we introduce a process reward model that aligns progressively to support the automatic verification of multimodal reasoning tasks. Experimental results across three complex multimodal reasoning benchmarks confirm the effectiveness of the AR-MCTS framework in enhancing the performance of various multimodal models. Further analysis demonstrates that AR-MCTS can optimize sampling diversity and accuracy, yielding reliable multimodal reasoning.
1. Introduction
AR-MCTS addresses multimodal reasoning verification by combining active retrieval with MCTS, whose path expansion is challenging when MLLMs’ internal knowledge is insufficient. Experiments on three complex multimodal reasoning benchmarks report improved performance across models, with gains in sampling diversity and verification accuracy.
1. Introduction
- MCTS adaptation to MLLMs remains largely unexplored, and reliance on internal knowledge for path expansion can compound errors caused by multimodal misalignment.Existing approaches often optimize simulation while leaving expansion to beam search.Source: §1 · pp. 1–2
- AR-MCTS dynamically retrieves problem-solving insights at each reasoning step, replacing beam-search sampling to diversify and improve the reliability of path expansion.Its motivation is that different steps in a complex task require different knowledge, unlike approaches that use retrieved knowledge as a whole.Source: §1 · p. 2
- The framework uses MCTS and active retrieval to generate step-wise annotations, then progressively aligns a process reward model for automated reasoning verification.The model is aligned with step-wise DPO and supervised fine-tuning objectives.Source: §1 · p. 2
- Experiments on three complex multimodal reasoning benchmarks demonstrate AR-MCTS’s effectiveness across various proprietary models.Further analysis reports optimized sampling diversity and verification accuracy.Source: §1 · p. 2
- The paper theoretically models MCTS mechanisms to reveal limitations in multimodal reasoning and presents step-wise retrieval as an alternative to model self-sampling.The authors describe this as the first introduction of retrieval at each multimodal reasoning step.Source: §1 · p. 2
2. Related Work
Related work covers structured reasoning, reflection, diverse-path selection, training data, external tools, and reward models for LLMs and MLLMs. It identifies multimodal reasoning and verification as less developed areas for MCTS and retrieval-augmented generation.
LLM and MLLM Reasoning
- Prior reasoning approaches include CoT, ToT, and PoT, alongside reflection, diverse-path selection, data augmentation, external tools, and reward models.The passage describes these as methods for improving reasoning or response quality across LLM and MLLM work.Source: §2 · p. 2
- MCTS and verification models have been used to simulate gradual reasoning in single-modal LLMs, while MLLM reasoning remains underexplored.The cited passage also notes multimodal task work using progressive decomposition and targeted training objectives.Source: §2 · pp. 2–3
Multimodal Retrieval-Augmented Generation
- Multimodal retrieval-augmented generation can support reasoning and question answering, but its reasoning process is described as largely unexplained and lacking verification mechanisms.The paper proposes retrieving multimodal information at each step to support PRM alignment and reasoning verification.Source: §2 · p. 3
3. Preliminary
The preliminary section frames MLLM reasoning as autoregressive generation from a multimodal query and retrieved insights, and describes sentence-level MCTS as a search process over reasoning steps.
Monte Carlo Tree Search
- MCTS constructs a search tree through repeated selection, expansion, evaluation, and back-propagation steps.During selection, it recursively chooses child nodes from the root using the Upper Confidence Bound.Source: §3 · p. 3
Problem Formulation
- The problem formulation assumes an MLLM generates a k-step reasoning path autoregressively from a multimodal query and retrieved problem-solving insights.Source: §3 · p. 3
- The paper models sentence-level MCTS as a Markov Decision Process, treating the current decoded reasoning step as a state and backtracking to sample the next step as an action.Intermediate trajectories are collected as decoding reaches a termination token.Source: §3 · p. 3
- Figure 1 presents statistics for the hybrid-modal retrieval corpus.Source: Figure 1 · p. 3
4. Methodology
AR-MCTS combines a hybrid-modal retrieval system with MCTS to generate step-wise reasoning annotations, then uses those annotations to progressively train a process reward model for fine-grained reasoning verification.
4. Methodology
- AR-MCTS uses active retrieval during MCTS expansion to obtain step-wise reasoning annotations, then progressively aligns a process reward model for fine-grained verification.The framework combines a hybrid-modal retrieval corpus and module with concept filtering before annotation generation.Source: §4 · p. 3, §4.4 · p. 4, §4.5 · p. 5
4.1. Hybrid-Modal Retrieval Corpus Construction
- The hybrid-modal corpus combines 22K text-only QA pairs and 12.5K multimodal pairs from six mathematical-reasoning sources with Wikipedia and COIG for general reasoning.The mathematical sources cover more than 20 sub-fields, and overlapping test-benchmark material is filtered out.Source: §4.1 · pp. 3–4
4.2. Unified Multimodal Retrieval Module
- The retrieval module uses Contriever for text-only queries and CLIP-based cross-modal retrieval to search the hybrid-modal corpus for top-K relevant knowledge.FAISS indexes the cross-modal embeddings; the retrieval database includes both multimodal and text-only samples.Source: §4.2 · p. 4
4.3. Knowledge Concept Filtering
- Knowledge concept filtering retains retrieved insights only when they meet both the query-relevance threshold and the concept-consistency threshold.This filters top-K hybrid-modal samples using similarity to the multimodal query and its knowledge-concept label.Source: §4.3 · p. 4
4.4. Progressive Multimodal Reasoning Annotation
- During MCTS expansion, AR-MCTS retrieves new candidate insights for each reasoning step, replacing the previous step’s insights to diversify sampled paths.One-step rollouts assess nodes by the probability of reaching the correct answer, and back-propagation updates visit counts and Q-values along the selected route.Source: §4.4 · p. 5
4.5. Curriculum Process Reward Modeling
- The PRM is trained in two phases: step-level DPO distinguishes preferred from dispreferred reasoning, followed by point-wise fine-tuning to score steps.Positive paths have value above 0.8, while paths with value 0 are negative; inference uses PRM scores and stops after round four.Source: §4.5 · pp. 5–6
- At inference, the fine-tuned PRM scores each step; AR-MCTS keeps the top-scoring path at each round and uses a four-round early-stopping criterion.This is intended to reduce computational complexity.Source: §4.5 · p. 6
5. Experiments
Across mathematical and general-reasoning benchmarks, AR-MCTS improves MLLM performance. Analyses also report gains in sampling accuracy and diversity, with component-removal results supporting the framework’s design.
5.1. Experimental Setup
- The evaluation covers MATHVISTA, WE-MATH, and GAOKAO-MM with proprietary and open-source MLLMs, using Self-Consistency, Self-Correction, and ORM as comparison strategies.Process annotations exclude sources under evaluation; GAOKAO-MM questions from 2022–2023 are used as the test set.Source: §5.1 · p. 6
5.2. Overall Results
- AR-MCTS improves performance across MLLMs and verification strategies on MATHVISTA and WE-MATH; Qwen2-VL-7B gains MATHVISTA (ALL: 5.3%↑) and WE-MATH (AVG: 8.3%↑) over zero-shot.The results also report stronger PRM than ORM performance on most WE-MATH S3 metrics, including GPT-4o (56.4% vs 50.3%) and Qwen2-VL (40.6% vs 34.6%).Source: §5.2 · p. 6, Table 1 · p. 7
5.3. General Reasoning Domain Verification
- On GAOKAO-MM, GPT-4o and Qwen2-VL-7B improve over their backbones and self-consistency; GPT-4o gains mathematics 12.5%↑, physics 7.7%↑, and history 20%↑.The paper presents these results as generalization across languages and reasoning disciplines.Source: Table 2 · p. 7, §5.3 · p. 7
5.4. Quantitative Analysis
- Removing PRM and active retrieval yields MATHVISTA drops of 3.1% and 2.2%, respectively; knowledge concept filtering also produces stable gains.Removing any AR-MCTS component reduces performance.Source: §5.4 · pp. 7–8, Table 3 · p. 8
- With more than 16 candidates, self-consistency fluctuates on WE-MATH; AR-MCTS outperforms ORM and self-consistency, increasingly as candidate count grows.Figure 4 compares inference-sampling strategies across candidate counts from 1 to 32; Random Choice averages random samples over that range.Source: Figure 4 · p. 8, §5.4 · p. 8
5.5. Does AR-MCTS Improve the Sampling Space?
- For 250 MATHVISTA problems sampled four times each, AR-MCTS produces more dispersed solutions and 46 versus 38 centroids compared with beam search.The visualization uses semantic embeddings reduced with PCA and clustered with DBSCAN.Source: Figure 5 · p. 8, §5.5 · pp. 8–9
- AR-MCTS improves solution-sampling accuracy over traditional beam search on both benchmarks, and accuracy rises with the number of candidate solutions.The analysis measures whether each question has at least one correct candidate solution.Source: Figure 6 · p. 8, §5.5 · p. 8
6. Conclusion
The conclusion presents AR-MCTS as a framework combining active retrieval and MCTS to improve complex multimodal reasoning. Its reported strengths include automated step-wise annotations, process-level verification, and improved sampling diversity and verification accuracy.
6. Conclusion
- AR-MCTS combines active retrieval with Monte Carlo Tree Search to progressively improve complex multimodal reasoning in MLLMs.The framework is presented as universal across MLLMs.Source: §6 · p. 9
- The framework automatically acquires step-wise reasoning annotations and progressively aligns a process reward model for process-level multimodal verification.Source: §6 · p. 9
- Experiments across MLLMs and benchmarks report improved sampling diversity and verification accuracy, supporting AR-MCTS as a promising approach to reliable reasoning.Source: §6 · p. 9
Supplementary Material
The supplementary workflow first builds a query-specific insight corpus from text-only and multimodal retrieval, then uses active retrieval and a process reward model during MCTS inference. It also includes statistics for the general-reasoning knowledge collection.
Supplementary Material
- Table 4 presents statistics for the general-reasoning knowledge collection.Source: Table 4 · p. 17
A.1. The Algorithm Workflow of AR-MCTS
- For each query, unified retrieval combines top-K results from separate text-only and multimodal routes, then filters them into a corpus of key insights for active retrieval.The workflow labels the combined results Dtop-K and the filtered insights Dins.Source: §A.1 · p. 17
- At each MCTS expansion step, the MLLM generates candidate states from retrieved insights; the PRM scores them, and the highest-scoring state extends the path until a terminal state yields the answer.Source: §A.1 · p. 17
B.1. Benchmarks and Datasets
The appendix describes the datasets used for evaluation and retrieval, the retrieval and inference procedures, the model comparisons, and the supporting training and corpus-processing setup.
B.1. Benchmarks and Datasets
- WE-MATH and MATHVISTA are evaluation benchmarks, while MathVision contributes mathematical problems to the reasoning knowledge base.WE-MATH focuses on decomposing complex problems using textbook knowledge units; MathVision contains competition problems with visual contexts.Source: §B.1 · pp. 17–18
- Unified Retrieval combines text-only and cross-modal retrieval, then filters candidate insights by query similarity and knowledge-concept consistency.The retrieval procedure uses text and image-text samples and returns top-K hybrid-modal samples before concept filtering.Source: §B.1 · p. 18
- AR-MCTS inference retrieves insights at each step, scores candidate continuations with a process reward model, and adds the highest-scoring step to the selected path.The procedure iterates until the maximum depth or a non-terminal path condition is reached.Source: §B.1 · p. 18
- MATHVERSE supplies text-only and vision-only knowledge-base data, while MATH and GSM8K contribute text-only reasoning data.For GAOKAO-MM, questions from 2010–2021 are used for the domain knowledge base and 2022–2023 questions form the test set.Source: §B.1 · p. 19
B.3. Implementation Details
- Retrieval uses mcontriever-mscoco for text and frozen CLIP for multimodal encoding, with FAISS indexing dense vectors for top-k retrieval.The retrieval setup incorporates text-to-text, text-to-image, image-to-image, image-to-text, and cross-modal similarity measures.Source: §B.3 · p. 20
B.4. Detailed Processing about Retrieved Corpus
- The corpus combines Chinese and English text resources with multimodal mathematics sources, and uses historical GAOKAO-MM questions to build knowledge separately from its later test years.For MATHVISTA and WE-MATH, the testmini sets are included in each other’s retrieval knowledge base; MATHVISTA test answers are supplemented with LLaVA-OneVision pseudo-answers.Source: §B.4 · p. 21
B.5. PRM Training Guideline; B.6. Detailes about Knowledge Concept Filtering
- The PRM is pre-aligned using text-only process-reward datasets, then finalized with AR-MCTS annotations; concept filtering uses InsTag labels on text and image-captioned multimodal data.The PRM is not fine-tuned on in-domain MLLM data; multimodal captions are generated with the corresponding evaluation backbone before InsTag annotation.Source: §B.5 · p. 21, §B.6 · p. 21
C.1. Results on More MLLMs backbones
The MATHVISTA generalization study reports stable AR-MCTS improvements for an open-source and a closed-source MLLM. Its findings align with the paper’s three core experimental conclusions.
C.1. Results on More MLLMs backbones
- On MATHVISTA, AR-MCTS achieves stable improvements with Llama3-Llava-Next-8B and GPT-4V.The study reports that the results align with the core findings that MLLMs struggle to self-correct, PRM outperforms ORM, and AR-MCTS better unlocks weaker MLLMs’ reasoning potential.Source: §C.1 · pp. 21–22
- The generalization results reaffirm that MLLMs struggle to self-correct reasoning errors and that PRM outperforms ORM on complex reasoning tasks.The authors also report that AR-MCTS better unlocks the reasoning potential of weaker MLLMs.Source: §C.1 · p. 22
C.2. The Composition Analysis of Retrieval Knowledge
The retrieval-composition analysis finds diverse text-only and multimodal sources among the top retrieved samples. The contamination analysis reports less than 5% overlap with MATHVISTA and WE-MATH for all sources.
C.2. The Composition Analysis of Retrieval Knowledge
- Top-50 retrieved samples for both MATHVISTA and WE-MATH span diverse text-only and multimodal sources, supporting the use of a diverse hybrid-modal retrieval library.The analysis ranks retrieved knowledge by similarity and visualizes source composition.Source: §C.2 · p. 22
C.3. Contamination Analysis on Hybrid-modal Retrieval Corpus
- At a 13-gram threshold, every retrieval source has less than 5% overlap with MATHVISTA and WE-MATH test sets.The authors interpret this analysis as indicating no overlap between the retrieval library and test sets.Source: §C.3 · p. 22
C.4. Ablations on Different Retrievers
Retriever ablations find varying downstream enhancement across retrievers, while the PRM objective comparison favors soft labels over hard labels on MATHVISTA.
C.4. Ablations on Different Retrievers
- Different text and multimodal retrievers yield varying degrees of downstream reasoning enhancement, supporting the retrieval module’s plug-and-play design and mixed retrieval library.The ablations concatenate the top two retrieval results for each sample.Source: §C.4 · p. 22, Table 9 · p. 23
- The authors identify feedback-driven supplementation of missing knowledge during stepwise reasoning as a future research direction.They describe dynamic supplementation based on feedback from multimodal models as an area with potential for further exploration.Source: §C.4 · p. 23
C.5. Comparison of Different Training Objectives for PRMs
- On MATHVISTA, the PRM trained with soft labels performs better than the PRM trained with hard labels.The comparison examines hard labels versus soft labels trained through a linear layer connected to a large model.Source: §C.5 · p. 23, Table 10 · p. 23
D. Limitations and Future Work
The authors identify MCTS annotation cost and challenges in training multimodal process reward models as limitations, alongside directions for reducing overhead and improving stepwise reasoning.
D. Limitations and Future Work
- MCTS-based annotation requires significant computational resources, although AR-MCTS costs substantially less than manual annotation.The authors describe computational cost as a common challenge in reasoning verification.Source: D. Limitations and Future Work · p. 23
- AR-MCTS avoids training multimodal foundation models, reducing computational overhead; the authors also point to vLLM as an efficiency aid.They characterize the framework as plug-and-play.Source: D. Limitations and Future Work · p. 23
- Training PRMs within multimodal models could improve image-text interaction, but scarce annotated data and high compute demands make this future work challenging.The authors describe this as a largely unexplored direction.Source: D. Limitations and Future Work · p. 23