PaperlayerView exact revision on arXiv

Source-linked AI summary

SoTA with Less: MCTS-Guided Sample Selection for Data-Efficient Visual Reasoning Self-Improvement

Xiyao Wang, Zhengyuan Yang, Chao Feng, Hongjin Lu, Linjie Li, Chung-Ching Lin, Kevin Lin, Furong Huang, Lijuan Wang

arXiv:2504.07934v3cs.CV

TL;DR

RFT has had less pronounced impact on VLMs, partly because multimodal reasoning differs from text-heavy pretraining and existing remedies often rely on cumbersome distillation and alignment. ThinkLite-VL uses MCTS-guided difficulty filtering to select appropriately challenging data for self-improvement, achieving strong gains with 11k or 7.5k samples and no knowledge distillation.

  • Problem

    RFT has had less pronounced impact on VLMs, while existing approaches often add knowledge distillation and supervised format alignment through cumbersome pipelines.

  • Method

    ThinkLite-VL estimates sample difficulty using the number of MCTS reasoning iterations required to solve each instance, then applies RFT to a curated subset of challenging samples without knowledge distillation.

  • Results

    Across eight VLM benchmarks, ThinkLite-VL models outperform strong baselines and existing SoTA models; ThinkLite-VL-7B improves its base model’s average performance by 7% and reaches 75.1 on MathVista, while ThinkLite-VL-72B reaches 79.7.

  • Takeaways & Limitations

    MCTS-guided difficulty filtering provides a data-efficient route to multimodal reasoning self-improvement using small, high-quality training subsets.

Abstract

from arXiv · show

We introduce ThinkLite-VL, a family of visual reasoning models that achieve state-of-the-art (SoTA) performance using an order of magnitude fewer training samples, relying purely on reinforcement fine-tuning (RFT) self-improvement without any knowledge distillation. Our central insight is that sample difficulty critically influences RFT effectiveness: appropriately challenging examples can drive substantial reasoning improvements, even in low-data regimes. However, quantifying sample difficulty in a reliable and scalable manner remains non-trivial. To address this, we repurpose Monte Carlo Tree Search (MCTS) to measure sample difficulty via the number of reasoning iterations a vision-language model (VLM) requires to solve each instance. This MCTS-based selection procedure identifies samples that induce deeper reasoning while remaining solvable, allowing us to filter a high-quality subset from 70k open-source examples spanning math, natural image understanding, and chart comprehension. Using this approach, we select just 11k challenging samples for RFT on Qwen2.5-VL-7B-Instruct and 7.5k samples for Qwen2.5-VL-72B-Instruct. The resulting models, ThinkLite-VL-7B and ThinkLite-VL-72B, significantly outperform their respective base models across eight visual reasoning benchmarks. In particular, ThinkLite-VL-7B improves the average performance of Qwen2.5-VL-7B-Instruct by 7\% and surpasses all existing 7B-level models, as well as much larger models such as GPT-4o, O1 and Qwen2.5-VL-72B, achieving a new SoTA score of 75.1 on MathVista. ThinkLite-VL-72B further advances the SoTA frontier, achieving an accuracy of 79.7 on MathVista and an average benchmark improvement of 4.42 over the open-source SOTA. These results demonstrate that MCTS-guided difficulty filtering provides a scalable and effective path toward data-efficient self-improvement in multimodal reasoning.

1. Introduction

ThinkLite-VL addresses limited VLM self-improvement by selecting appropriately challenging samples for RFT with MCTS, achieving strong visual-reasoning results from compact datasets without distillation.

  • Motivation: RFT has been less pronounced for VLMs, while prior pipelines use knowledge distillation and supervised format alignment before RFT.These pipelines are described as cumbersome and limiting self-training alone.
  • Approach: ThinkLite-VL uses appropriately challenging, high-quality samples to enable VLM self-improvement through RFT without knowledge distillation.The approach relies on data quality and alignment with the base model’s capability level.
  • Approach: MCTS estimates sample difficulty by counting the reasoning iterations a VLM requires to reach a solution, identifying examples that promote deeper reasoning.The pipeline ranks samples by reasoning steps in simulated MCTS inference traces.
  • Data-efficient RFT pipeline: 70k open-source samples across mathematical reasoning, natural image understanding, and chart interpretation yield filtered sets of 11k samples for 7B and 7.5k for 72B RFT.RFT is applied directly to these subsets without supervised fine-tuning or distillation.
  • Results: 75.1% on MathVista makes ThinkLite-VL-7B superior to several larger models, while ThinkLite-VL-72B reaches 79.7% on the same benchmark.The cited comparisons include GPT-4o, O1, and Qwen2.5-VL-72B.
  • Results: ThinkLite-VL-7B and ThinkLite-VL-72B outperform strong baselines across eight VLM benchmarks, with average performance gains over their respective base models.ThinkLite-VL-7B rises from 59.69% to 64.18%, while ThinkLite-VL-72B rises from 68.25% to 72.67%.

2. Related work

Prior work improves reasoning through structured outputs, supervised reasoning data, process rewards, search, and reinforcement learning, while ThinkLite-VL combines search-based difficulty estimation with reinforcement fine-tuning without SFT or distillation.

  • Reasoning methods include chain-of-thought or structured outputs, supervised fine-tuning on reasoning annotations, process reward models, search, and outcome-based reinforcement learning.
  • Vision-language models extend reasoning capabilities across diverse scenarios by combining visual inputs with language-based processing.
  • ThinkLite-VL combines search-based sample difficulty estimation with reinforcement fine-tuning for data-efficient visual reasoning at 7B and 72B scales without SFT or distillation.

3. Training Recipe

ThinkLite-VL builds a training pipeline that collects diverse multimodal data, uses MCTS to select appropriately challenging examples, and applies reinforcement fine-tuning directly to the filtered subsets.

  • Training Recipe: The pipeline’s core technical contribution is difficulty-aware sample selection using MCTS to predict sample difficulty for self-improving training.
  • Data Collection: 70k open-source examples cover multimodal mathematical reasoning, natural image understanding, and chart understanding.
  • Data Collection: Open-ended reformulation prevents answers from being obtained by guessing multiple-choice options, increasing task difficulty and filtering reliability.
  • MCTS-based Sample Selection: MCTS models reasoning states as prefixes, samples candidate steps with the VLM policy, and performs selection, expansion, and simulation without a separate process reward model.
  • MCTS-based Sample Selection: K measures the MCTS iterations required to solve each sample; selecting K greater than 5 and samples unsolved after 50 iterations yields 11k and 7.5k subsets for the 7B and 72B models.
  • Visual Reasoning Training: The filtered data are used for direct reinforcement fine-tuning with GRPO, without supervised fine-tuning or knowledge distillation.

4. Experiments

Experiments show that ThinkLite-VL improves visual reasoning across eight benchmarks using difficulty-curated RFT data, with MCTS selection outperforming random and self-consistency alternatives. The gains depend on model-specific sample difficulty, while online filtering offers negligible improvement beyond faster convergence.

  • Benchmark Evaluation: ThinkLite-VL-7B achieves SoTA performance among 7B-scale models on six of eight benchmarks, with only marginal gaps on MMBench and MMVet.
  • Benchmark Evaluation: 75.1 on MathVista is achieved by ThinkLite-VL-7B, exceeding GPT-4o and o1, while ThinkLite-VL-72B reaches 79.7 on MathVista and 64.3 on MathVerse.
  • Benchmark Evaluation: 7.5% and 6.5% are the average improvements over Qwen2.5-VL-7B-Instruct and Qwen2.5-VL-72B-Instruct, respectively, across eight benchmarks.
  • Effectiveness of MCTS-based sample selection: 5.4% and 4.4% are the average improvements over equal-sized random subsets at the 7B and 72B scales, respectively.
  • Ablation Studies: ThinkLite-VL-7B trained with MCTS-guided selection achieves the highest average performance, surpassing random, full-dataset, and self-consistency-filtered settings despite fewer samples.
  • Comparison with Online Data Selection: The online self-consistency filter offers negligible improvement except faster convergence, whereas MCTS better identifies genuinely challenging examples through explicit tree search.
  • Data Difficulty Analysis between 7B and 72B Models: 5.4k samples overlap between the 11k and 7.5k selected sets, but 3.6k are unsolved by either model within 50 MCTS iterations.
  • Data Difficulty Analysis between 7B and 72B Models: Cross-sample training yields markedly smaller gains than model-specific curation, indicating that selected samples transfer poorly between the 7B and 72B models.

5. Conclusion

The paper concludes that MCTS-guided difficulty selection enables self-improvement in visual-language models through RFT without external supervision or knowledge distillation. Small, high-quality subsets selected from 70k examples produce substantial benchmark gains for both model scales.

  • MCTS-guided sample selection enables VLM self-improvement through RFT without external supervision or knowledge distillation.
  • 11k and 7.5k challenging samples are selected for 7B-level and 72B-level models from 70k initial examples.
  • ThinkLite-VL-7B and ThinkLite-VL-72B significantly improve visual reasoning across multiple benchmarks, reaching SoTA accuracies of 79.7 on MathVista and 64.3 on MathVerse.

A. Prompts used in experiments

The appendix documents the prompts used for MCTS generation, critic evaluation, and RFT. It also identifies the corresponding prompt tables and their intended roles in the training pipeline.

  • A.1. Prompt for MCTS: The MCTS prompt is documented in Table 6 under subsection A.1.
  • A.1. Prompt for MCTS: Table 6 uses two in-context examples to make the VLM follow the specified reasoning format during MCTS.
  • A.2. Prompt for Critic Model: The critic-model prompt is documented in Table 7 under subsection A.2.
  • A.2. Prompt for Critic Model: Table 7 provides the critic prompt for evaluating MCTS simulation results.
  • A.3. Prompt for RFT: The RFT prompt is documented in Table 8 under subsection A.3.
  • A.3. Prompt for RFT: Table 8 contains the prompt template used for reinforcement learning fine-tuning.

B.1. Reward curves of VLM with different training data

Reward curves reveal that reward gains alone do not predict reasoning improvement. Difficulty-selected training produces lower rewards while outperforming easy-sample and full-set alternatives on benchmarks.

  • ThinkLite-VL-Random11k and ThinkLite-VL-Fullset achieve higher training rewards but inferior benchmark performance compared with ThinkLite-VL.
  • Easy samples rapidly increase rewards but fail to enhance the model’s reasoning ability.
  • Lower rewards for ThinkLite-VL than ThinkLite-VL-Iter5Only indicate that MCTS-selected unsolved data poses greater challenges to the VLM.

B.2. Ablation Study of Data Difficulty

Training-data difficulty strongly affects RFT outcomes: the Iter5+Unsolved setting performs best, while overly broad or easy-sample-heavy settings weaken reasoning performance.

  • Iter5+Unsolved presents the most challenging learning setting among the compared 7B RFT data conditions.
  • 63.89 is the highest average score, achieved by the Iter5+Unsolved subset and outperforming all other settings.
  • Performance consistently declines when the difficulty threshold expands to Iter10, Iter20, Iter30, and Iter40.
  • As training-data difficulty decreases, performance also declines, suggesting that excessive easy samples weaken the RFT training signal.

C. Case Studies

The case studies illustrate how MCTS-based selection distinguishes samples across multiple visual-reasoning datasets using the reasoning effort required for correct solutions.

  • Difficulty levels are determined by the number of reasoning iterations a VLM requires to reach the correct answer during MCTS.
  • The tables provide reference examples for understanding how the method distinguishes easy and challenging samples.
  • Examples of MCTS-assigned difficulty levels are presented for FigureQA, ScienceQA, OK-VQA, IconQA, TabMWP, GeoQA, Geos, and related datasets.
Loading 2504.07934v3…