PaperlayerView exact revision on arXiv

Source-linked AI summary

Mulberry: Empowering MLLM with o1-like Reasoning and Reflection via Collective Monte Carlo Tree Search

Huanjin Yao, Jiaxing Huang, Wenhao Wu, Jingyi Zhang, Yibo Wang, Shunyu Liu, Yingjie Wang, Yuxin Song, Haocheng Feng, Li Shen, Dacheng Tao

arXiv:2412.18319v2cs.CVcs.AI

TL;DR

MLLMs often struggle with complex tasks while giving brief answers with few explicit reasoning steps, and conventional tree search can be ineffective or inefficient for them. CoMCTS uses multiple models to search reasoning paths and train Mulberry on a dataset of explicit reasoning nodes. The paper reports that CoMCTS and Mulberry outperform or compare favorably with alternatives across search and benchmark evaluations.

  • Problem

    MLLMs often fail on complex tasks requiring in-depth reasoning, while current models predominantly give brief final answers with few explicit intermediate steps.

    Source: §1 · p. 1

  • Method

    CoMCTS uses collective knowledge from multiple MLLMs to search effective and reflective reasoning paths, which are used to construct Mulberry-260k and train Mulberry.

    Source: §1 · p. 2, §5 · p. 9

  • Results

    The paper reports that CoMCTS is superior in search effectiveness and efficiency and that its methods outperform alternatives across various benchmarks.

    Source: §4.5 · p. 9, §5 · p. 9

  • Takeaways & Limitations

    Mulberry-260k provides explicit reasoning nodes for each question, and Mulberry is presented as a model series with step-by-step reasoning and reflection capabilities.

    Source: §1 · p. 2, §5 · p. 9

Abstract

from arXiv · show

In this work, we aim to develop an MLLM that understands and solves questions by learning to create each intermediate step of the reasoning involved till the final answer. To this end, we propose Collective Monte Carlo Tree Search (CoMCTS), a new learning-to-reason method for MLLMs, which introduces the concept of collective learning into ``tree search'' for effective and efficient reasoning-path searching and learning. The core idea of CoMCTS is to leverage collective knowledge from multiple models to collaboratively conjecture, search and identify effective reasoning paths toward correct answers via four iterative operations including Expansion, Simulation and Error Positioning, Backpropagation, and Selection. Using CoMCTS, we construct Mulberry-260k, a multimodal dataset with a tree of rich, explicit and well-defined reasoning nodes for each question. With Mulberry-260k, we perform collective SFT to train our model, Mulberry, a series of MLLMs with o1-like step-by-step Reasoning and Reflection capabilities. Extensive experiments demonstrate the superiority of our proposed methods on various benchmarks. Code will be available at https://github.com/HJYao00/Mulberry

1. Introduction

MLLMs often fail on complex reasoning while producing brief answers with few explicit intermediate steps, and conventional MCTS can be ineffective or inefficient for MLLMs. CoMCTS uses collective knowledge to search effective and reflective reasoning paths, which support Mulberry-260k and the Mulberry models.

1. Introduction

  • MLLMs often fail on complex tasks and predominantly produce brief final answers with few explicit, well-defined reasoning steps.The paper contrasts this direct-prediction mode with its aim of learning to create each intermediate reasoning step.Source: §1 · p. 1
  • CoMCTS searches reasoning paths with collective knowledge from multiple MLLMs, jointly expanding candidates, simulating outcomes and positioning errors, backpropagating scores and visit counts, and selecting the next leaf by Upper Confidence Bound.Its iterations can expand candidate reasoning nodes through to the end from a given start node.Source: §1 · p. 2
  • Joint expansion searches across multiple MLLMs’ reasoning spaces, creating a unified tree of diverse, complementary nodes rather than remaining within one model’s reasoning space.The paper attributes CoMCTS’s search effectiveness and efficiency to these mechanisms.Source: §1 · p. 2
  • Joint simulation and error positioning can skip multiple intermediate steps and select the last correct step as the next start node, reducing search time while maintaining search effectiveness.The paper notes that other models can help identify and position errors a model may have difficulty recognizing itself.Source: §1 · p. 2
  • CoMCTS also builds reflective paths by integrating negative sibling nodes into effective paths, tracing a transition from an erroneous node to a positive one.These paths are intended to support step-wise reflection that dynamically calibrates reasoning toward a correct answer.Source: §1 · p. 2
  • CoMCTS-searched paths for multimodal inputs form Mulberry-260k, a dataset of rich, explicit reasoning nodes used for collective supervised fine-tuning of Mulberry models.The models are described as having o1-like step-by-step reasoning and reflection capabilities.Source: §1 · p. 2

2. Related Works

Related work covers MLLM reasoning, learning-based LLM reasoning, MCTS, and collective learning; the paper positions CoMCTS as bringing tree search and collective learning to MLLM reasoning.

2.1. Multimodal Large Language Model

  • Prior MLLM work explores prompting and structured reasoning, while this paper introduces tree search to MLLM reasoning through CoMCTS.The cited studies report that direct Chain-of-Thought prompting may yield limited gains or degradation.Source: §2.1 · pp. 2–3

2.2. Large Language Model Reasoning

  • LLM reasoning methods are categorized as prompt-based, plan-based, or learning-based, with learning-based approaches using search to build intermediate-thought trees and train step-by-step reasoning.Examples include GPTo1, Star, Iter-MCTS, and ReST-MCTS.Source: §2.2 · p. 3

2.3. Monte-Carlo Tree Search

  • MCTS is presented as a decision-search paradigm used in fields including games, robotics, theorem proving, and matrix multiplication.The paper proposes CoMCTS for reasoning-path search and learning on MLLMs.Source: §2.3 · p. 3

2.4. Collective Learning

  • Collective learning harnesses knowledge from multiple individuals to improve learning outcomes, with prior uses spanning classification, neural networks, and reinforcement learning.Examples include multiple classifiers for pseudo-labeling and multiple models in reinforcement learning.Source: §2.4 · p. 3

3. Methodology

The methodology introduces CoMCTS, which uses multiple MLLMs to search for reasoning paths, extends the search to reflective paths, and trains Mulberry on the resulting reasoning trees.

3.1. CoMCTS for effective reasoning

  • CoMCTS iterates collective expansion, joint error scoring and pruning, bottom-up node-statistic updates, and highest-UCB leaf selection until a correct path is found or the iteration limit is reached.The resulting question-dependent tree contains a correct reasoning path and supports step-by-step learning.Source: Figure 2 · p. 4, §3.1 · p. 5

3.2. CoMCTS for reflective reasoning

  • Reflective-path search inserts a negative sibling node and reflection prompt before a positive reasoning step, creating a trajectory from error toward correction.The negative sibling is selected among nodes at the same level under the same parent, and the prompt marks the previous step as wrong.Source: §3.2 · p. 5

3.3. Training with Collective MCTS

  • Mulberry-260k stores questions, effective paths, reflective paths, and reasoning trees; collective SFT trains models to learn both effective reasoning and step-wise reflection.The reflective objective uses negative information from the search to train calibration of negative reasoning nodes.Source: §3.3 · pp. 5–6

4. Experiment

The experiments assess Mulberry-260K, its construction and training, and the resulting models across benchmark comparisons, ablations, and tree-search analyses. They report gains over baselines and state-of-the-art comparisons, alongside findings on reasoning-step distributions and collective search.

4.1. Dataset

  • Reasoning paths average 7.5 steps overall, 6.8 for chart-related tasks, and 8.9 for geometry-related tasks.The dataset analysis associates fewer steps with simple tasks and more steps with complex tasks.Source: §4.1 · pp. 6–7, Figure 4 · p. 7

4.2. Implementation Detail

  • The experiments use four collective-search models, a maximum of 20 search iterations, and collective SFT over two epochs.The search group is GPT4o, Qwen2-VL-7B, LLaMA-3.2-11B-Vision-Instruct, and Qwen2-VL-72B; training uses batch size 128 and learning rate 1e-5, except 5e-6 for Qwen2-VL-7B.Source: §4.2 · p. 7

4.3. Main Results

  • +4.2% over Qwen2-VL-7B and +7.5% over LLaMA-3.2-11B-Vision-Instruct; searched data also yields +5.4% and +11.0% gains averaged on 8 benchmarks for Qwen2-VL-2B and LLaVA-NeXT-8B.The first gains are for models involved in collective search; the latter are for models not involved in it.Source: §4.3 · p. 7
  • +5.7% and +6.5% on MathVista, and +3.0% and +1.0% on MMMU, for Mulberry-8B over LLaVA-Reasoner-8B and Insight-V-8B, respectively; Mulberry-11B exceeds LLaVA-COT-11B by +6.3% on MathVista.The first four comparisons use LLaVA-NeXT-8B as the same base model; the last uses LLaMA-3.2-11B-Vision-Instruct.Source: §4.3 · p. 7
  • Mulberry outperforms most open-source MLLMs and achieves competitive results against closed-source ones, demonstrating step-by-step reasoning and reflection abilities.The comparison evaluates Mulberry trained on CoMCTS-searched data.Source: §4.3 · p. 7, Table 1 · p. 8

4.4. Ablation Study

  • CoMCTS with all four models reaches 80.2% S.S.R., while adding reflection data improves MathVista performance by 0.8%.The ablations assess collective-model contributions to search and the respective contributions of effective and reflective reasoning data.Source: §4.4 · p. 8, Table 2 · p. 8, Table 3 · p. 8

4.5. Discussion

  • CoMCTS shows greater search effectiveness and efficiency than existing tree-search methods, while qualitative examples contrast Mulberry’s explicit correct reasoning with shorter, incorrect baseline predictions.The authors attribute CoMCTS’s search advantage largely to joint expansion across models.Source: §4.5 · p. 9, Table 4 · p. 9, Figure 5 · p. 9

5. Conclusion

The paper presents CoMCTS, Mulberry-260k, and Mulberry as a collective-search approach, a multimodal reasoning dataset, and models trained for step-by-step reasoning and reflection.

  • CoMCTS introduces collective learning into tree search to search for effective reasoning paths for MLLMs.Source: §5 · p. 9
  • The authors use CoMCTS to search effective and reflective reasoning paths and construct Mulberry-260k with explicit reasoning nodes for each multimodal question.Source: §5 · p. 9
  • Using Mulberry-260k, the authors train Mulberry models with step-by-step reasoning and reflection capabilities.Source: §5 · p. 9

A. The Sources of Raw Data

The raw-data collection comprises 260K multimodal questions from varied domains, including mathematics, figure understanding, medical, science, and general question-answering data.

Loading 2412.18319v2…