PaperlayerView exact revision on arXiv

Source-linked AI summary

MM-HELIX: Boosting Multimodal Long-Chain Reflective Reasoning with Holistic Platform and Adaptive Hybrid Policy Optimization

Xiangyu Zhao, Junming Lin, Tianhao Liang, Yifan Zhou, Wenhao Chai, Yuzhe Gu, Weiyun Wang, Kai Chen, Gen Luo, Wenwei Zhang, Junchi Yan, Hua Yang, Haodong Duan, Xue Yang

arXiv:2510.08540v2cs.CV

TL;DR

Long-chain reflective reasoning in multimodal contexts remains underexplored and under-evaluated. The paper introduces a benchmark, reflective training data, and AHPO, which combines expert guidance with online exploration. It reports gains on its benchmark and general reasoning tasks.

  • Problem

    Current MLLMs largely generate outputs in a single pass, leaving their end-to-end multi-step reflective reasoning in multimodal contexts underexplored and under-evaluated.

    Source: §1 · p. 2

  • Method

    The paper constructs MM-HELIX and MM-HELIX-100K, then uses AHPO to adaptively combine off-policy expert guidance with on-policy exploration.

    Source: §1 · pp. 2–3, §3.3 · p. 7

  • Results

    +18.6% point improvement over Qwen2.5-VL-7B on MM-HELIX, with a +5.7% average accuracy increase on general math and logic tasks.

    Source: §1 · p. 3, §4.2 · p. 8

  • Takeaways & Limitations

    The findings establish reflective reasoning as a transferable skill that can be instilled and generalized in MLLMs.

    Source: §5 · p. 10

Abstract

from arXiv · show

While current Multimodal Large Language Models (MLLMs) have demonstrated proficiency in reasoning tasks such as mathematics and logic, their capacity for long-chain reflective reasoning, a prerequisite for solving complex real-world problems, remains largely underexplored. In this work, we first conduct an extensive empirical investigation to evaluate this capability. Leveraging a carefully designed data synthesis engine, we construct MM-HELIX, a multimodal benchmark consisting 1,260 samples of 42 challenging synthetic tasks that require iterative thinking and backtracking. Empirical results on this benchmark reveal that existing MLLMs exhibit significant performance deficits in long-chain reflective reasoning. To address this limitation, we generate post-training data and further explore learning paradigms for exploiting such data. We first develop the Step-Elicited Response Generation pipeline to create MM-HELIX-100K, a large-scale dataset of 100k high-quality, reflective reasoning traces for instruction-tuning stage. Given that standard Reinforcement Learning fails on complex tasks due to sparse reward signals and catastrophic forgetting after Supervised Fine-Tuning, we propose Adaptive Hybrid Policy Optimization (AHPO), a novel training strategy that dynamically unifies offline supervision and online optimization into a single stage. This strategy enables the model to learn from expert data when rewards are sparse and conduct independent exploration once proficient. When applied to the Qwen2.5-VL-7B baseline, our method achieves a +18.6\% accuracy improvement on MM-HELIX benchmark and demonstrates strong generalization with a +5.7\% average performance gain on general mathematic and logic tasks. Our work demonstrate that reflective reasoning in MLLMs can be effectively learned and generalized, paving the way for developing more capable MLLMs.

1 INTRODUCTION

The introduction identifies long-chain reflective reasoning in multimodal contexts as underexplored, then presents a benchmark, reflective training data, and AHPO as responses. The reported results show gains on MM-HELIX and general mathematics and logic tasks.

1 INTRODUCTION

  • MM-HELIX evaluates long-chain iterative reasoning across 42 multimodal tasks, with 1,260 samples requiring visual observation, rule understanding, reflection, and backtracking.Its procedural pipeline includes a task generator, solver, and verifier; the verifier also serves as a reward oracle for reinforcement learning.Source: §1 · pp. 2–3
  • Step-Elicited Response Generation integrates rule-based key-step knowledge to produce reflective Chain-of-Thought training traces.The resulting MM-HELIX-100K dataset contains 100,000 high-quality samples spanning 42 tasks and difficulty levels.Source: §1 · pp. 2–3
  • AHPO dynamically combines off-policy expert guidance with on-policy exploration, increasing supervision when rewards are sparse and attenuating it as rewards become dense.The introduction motivates this design by reporting catastrophic forgetting with instruction-tuning and reward sparsity with on-policy reinforcement learning.Source: §1 · pp. 2–3
  • +18.6% accuracy improvement on MM-HELIX and +5.7% average accuracy increase on general math and logic tasks are reported for the approach.The introduction attributes these gains to training with AHPO.Source: §1 · pp. 2–3
  • Evaluation on MM-HELIX finds that current state-of-the-art MLLMs have substantial difficulty with complex reflective reasoning.The benchmark is designed to assess long-chain, iterative, and reflective reasoning in multimodal tasks.Source: §1 · pp. 2–3

2 RELATED WORK

Related work covers advances in multimodal and reasoning-oriented models, long-chain reasoning and procedural benchmarks, and reinforcement-learning methods. It identifies limitations in prior evaluation settings and training approaches that motivate AHPO.

Multimodal Large Language Models

  • Recent MLLMs show strong general multimodal capabilities, while newer reasoning-oriented models explicitly emphasize structured thinking.The cited examples include Gemini 2.5, Qwen-2.5-VL, InternVL3, GLM-4.5V-Thinking, Seed1.5-VL, and Kimi-VL-A3B-Thinking.Source: §2 · p. 3

Exploration of Long-chain Reasoning

  • CoT and ToT use intermediate reasoning traces, while procedural generation supports logic-puzzle and multimodal QA benchmarks.Prior works primarily focus on text-only or puzzle-like tasks constrained to multiple-choice or fill-in-the-blank formats.Source: §2 · p. 3

Reinforcement Learning Method

  • On-policy and off-policy RL methods address different training needs, but prior RL methods are described as inefficient on hard tasks.The section presents AHPO as a way to simplify training and enhance multimodal reasoning.Source: §2 · p. 3

3 METHOD

The section develops MM-HELIX to assess multimodal long-chain reflective reasoning, then pairs reflective-trace generation with adaptive off-policy and on-policy training.

3.1 MM-HELIX: BENCHMARKING MULTIMODAL REFLECTIVE END-TO-END REASONING

  • MM-HELIX comprises 42 tasks across algorithms, graphs, puzzles, and games, with five progressive difficulty levels for evaluating long-chain reflective reasoning.Source: §3.1 · pp. 3–4, Figure 2 · p. 4
  • A procedural generator creates multimodal instances; a deterministic solver supplies feasible ground truth, and a verifier checks discrete answers or simulates multi-step action sequences.Source: §3.1 · pp. 4–5
  • 1,260 unique instances balance 42 tasks across five difficulty levels, with six instances per task at each level.Source: §3.1 · p. 5

3.2 MM-HELIX-100K: GUIDING MULTIMODAL REFLECTIVE REASONING

  • Qwen-2.5-VL-72B achieves only approximately 10% accuracy on MM-HELIX, motivating investigation of targeted training and transfer to mathematics and logic.Source: §3.2 · p. 5
  • SERG uses rule-based reasoning paths to scaffold Qwen3-235B’s reflective explanations, accepting each generated trajectory only when its final answer passes an automated verifier.Source: §3.2 · p. 5

3.3 AHPO: ADAPTIVE HYBRID ALGORITHM FOR GENERALIZING REFLECTION

  • AHPO combines expert-data off-policy loss with on-policy optimization, using an adaptive coefficient to gate expert supervision according to policy success.Source: §3.3 · pp. 6–7
  • Static expert weighting can conflict with the evolving on-policy distribution and degrade performance once the model surpasses the expert data’s proficiency.Source: §3.3 · p. 7
  • Using MM-HELIX-100K trajectories, AHPO learns reflective reasoning on MM-HELIX and generalizes those skills to broader mathematics and logic tasks.Source: §3.3 · p. 7

4 EXPERIMENT

The experiments expose substantial limits in MLLM reflective reasoning and show AHPO improving benchmark performance and generalization. SERG also reduces trace-generation time and produces data that improves downstream training.

4.1 EVALUATION RESULTS ON MM-HELIX

  • GPT-5 reached 58.1% accuracy, and no other evaluated model exceeded 50%, revealing a substantial deficit on multimodal reflective reasoning.Intern-S1-241B led open-source models at 33.3%, while InternVL-3-78B scored 9.9%.Source: §4.1 · p. 7, Table 1 · p. 8
  • Models performed best on Algorithm tasks, moderately on Graph and Puzzle tasks, and worst on Game tasks, where iterative state tracking is required.GPT-5 accuracy rose from 58.1% on multimodal tasks to 84.5% on text-only equivalents.Source: §4.1 · pp. 7–8

4.2 MAIN RESULTS OF AHPO

  • 24.9% accuracy on MM-HELIX was AHPO’s highest score among the compared methods, +18.6 percentage points over Qwen2.5-VL-7B; it also gained +5.7 points on general math and logic tasks.AHPO exceeded larger models Qwen2.5-VL-72B and GLM-4.5V-106B on MM-HELIX.Source: §4.2 · p. 8, Table 2 · p. 9

4.3 COMPARISON OF GENERATION PIPELINE

  • SERG reduced generation time by 90% versus unguided Qwen3-235B roll-outs and improved downstream performance by 4.9% over equivalent Rule-Based data.The comparison used 1,000 samples for generation and 22k samples for downstream fine-tuning.Source: Table 3 · p. 9, Table 4 · p. 9, §4.3 · p. 9

4.2 MAIN RESULTS OF AHPO

  • SFT improved in-domain performance but caused catastrophic forgetting on general reasoning tasks, while GRPO after SFT did not recover that deficit.GRPO alone showed negligible improvement, and LUFFY’s gains were smaller than AHPO’s.Source: §4.2 · p. 9

5 CONCLUSION

The conclusion presents MM-HELIX as evidence of current MLLM limitations in long-chain reflective reasoning, then describes training data and AHPO as the response. The resulting model gains on the benchmark and general reasoning tasks support the paper’s conclusion that reflective reasoning can be learned and generalized.

5 CONCLUSION

  • MM-HELIX confirmed profound limitations in current models’ long-chain reflective reasoning.Source: §5 · p. 9
  • The authors developed MM-HELIX-100K and AHPO to provide training data and cultivate reflective reasoning.Source: §5 · p. 9
  • MM-HELIX-7B gained on the in-domain benchmark and general reasoning tasks, supporting the conclusion that reflective reasoning can transfer across tasks.Source: §5 · p. 10

A.1 THE USE OF LARGE LANGUAGE MODELS (LLMS)

The authors report that an LLM was not involved in developing the article’s ideas. They used an LLM for minor grammar corrections and rewrote reasoning during data construction with Qwen3-235B.

A.1 THE USE OF LARGE LANGUAGE MODELS (LLMS)

  • An LLM was not involved in developing the article’s ideas.Source: §A.1 · p. 13
  • The authors used an LLM to correct minor writing errors, such as grammatical errors.Source: §A.1 · p. 13
  • During data construction, the authors rewrote reasoning processes using Qwen3-235B.Source: §A.1 · p. 13

A.2 EXPERIMENT SETTINGS

The appendix describes evaluation and training settings, including separate generation limits for models with and without thinking steps. It also specifies datasets and batch/trajectory settings for supervised fine-tuning and reinforcement learning.

A.2 EXPERIMENT SETTINGS

  • Evaluations used VLMEvalKit, with MM-HELIX generation limits and temperatures adjusted for models with versus without thinking steps.Thinking models used a 32,768-token limit (or 16,384 for smaller context windows) and temperature 0.6; models without thinking used 8,192 tokens and temperature 0.0.Source: §A.2 · p. 13
  • Generalization evaluations included MathVision, MathVerse-VisionOnly, LogicVista, and WeMath.Source: §A.2 · p. 13
  • RL used a global batch size of 128 and five response trajectories per sample; AHPO’s success-rate threshold was 2.SFT used 22k MM-HELIX-100K samples, while RL used an approximately 37k-sample mix including MM-HELIX-CoT-100k and MMK12.Source: §A.2 · p. 13

A.3 DETAILS OF COMPARISON OF COT GENERATION PIPELINE

SERG produces higher-quality, less redundant CoTs than the compared generation approaches, with lengths between Rule-Based CoT and model-rollout CoTs.

A.3 DETAILS OF COMPARISON OF COT GENERATION PIPELINE

  • SERG CoTs average 5,715.61 tokens, compared with 2,728.83 for Rule-Based CoT and 7,552.17 for model-rollout CoTs.Fig. 8 presents the distributions for the three approaches.Source: §A.3 · p. 13, Figure 8 · p. 13
  • SERG is proposed to address Rule-Based CoT’s rigid, mechanical reasoning and model rollout’s inefficiency, redundancy, and lack of supervision for reasoning correctness.The paper describes SERG’s resulting reasoning as more accurate and concise, with reduced redundancy and improved quality.Source: §A.3 · p. 13
  • SERG CoTs have fewer extraneous details and higher quality than rollout CoTs while incorporating rule-based structure; Fig. 8 compares all three length distributions.The authors attribute this to the structured logical reasoning steps of Rule-Based CoT.Source: §A.3 · p. 13, Figure 8 · p. 13

A.4 MM-HELIX-100K STATISTICS

The appendix reports a uniform difficulty distribution for MM-HELIX-100K and statistics on question, answer, and CoT lengths.

A.4 MM-HELIX-100K STATISTICS

  • MM-HELIX-100K has a uniform difficulty distribution across tasks, reported in Table 5.The supplied table passage identifies Table 5 as the difficulty distribution of tasks in MM-HELIX-100K.Source: §A.4 · p. 14, Table 5 · p. 14
  • Question, answer, and CoT token-length statistics for MM-HELIX-100K are reported in Table 6.The supplied table passage identifies Table 6 as statistics for questions, answers, and chain-of-thoughts.Source: §A.4 · p. 14, Table 6 · p. 14

A.5 TASK DIFFICULTY SETTINGS

Task difficulty is set using task-specific parameters when constructing initial states, with map size illustrated at several scales.

A.5 TASK DIFFICULTY SETTINGS

  • Difficulty is divided using different parameters for each task when constructing its initial state.The section gives map size as one example of a task setting.Source: §A.5 · p. 14
  • Map-size examples include 6 × 6, 7 × 7, and 8 × 8 grids.These are among the listed map-size settings.Source: §A.5 · p. 14
  • The listed map-size examples also include 9 × 9 and 10 × 10 grids.These are the largest map sizes shown in the examples.Source: §A.5 · p. 15

A.6 MM-HELIX BENCHMARK EXAMPLES

This appendix section illustrates MM-HELIX benchmark examples, spanning image-based puzzles and structured reasoning tasks across grid, path-finding, arithmetic, and sequence problems, with explicit rules and answer formats.

A.6 MM-HELIX BENCHMARK EXAMPLES

  • Examples include water-grid, Kakuro, Nonogram, and Numbrix puzzles, with solutions constrained by region levels, sum clues, block clues, or sequential adjacency.The prompts request filled-cell coordinates or digits, a text grid, or a completed numbered grid.Source: §A.6 · pp. 15, 16, 18, 19
  • Snake tasks ask for either a safe route that eats all apples or a non-intersecting path from S to E satisfying row and column counts.The apple-eating task returns directional moves, while the path task returns coordinates.Source: §A.6 · pp. 17, 22
  • Other examples require a closed Shingoki loop, a sequence of 15-puzzle moves, or Sokoban moves that push boxes onto target positions.Each prompt defines movement or loop constraints and specifies a segment-list or move-sequence response.Source: §A.6 · pp. 20, 21, 23
  • Word-search and Tapa examples test straight-line word location and black-cell placement under connectivity and shape constraints.The word search accepts forward or backward words in eight directions; Tapa clues describe connected groups among neighboring cells.Source: §A.6 · pp. 24–25
  • Game examples ask models to find maze paths, solve Tower of Hanoi, identify Minesweeper locations, or connect Bridges islands under connectivity and move constraints.Campsite examples require tent placements that satisfy tree-adjacency, non-adjacency, and row-and-column counts.Source: §A.6 · pp. 26, 27, 30, 33
  • Examples include visual grid puzzles such as Hitori, inequality Latin squares, Eulero, Calcudoku, and Kukurasu, with uniqueness or arithmetic constraints and structured grid outputs.Other grid examples include Sudoku, Binairo, and skyscraper puzzles.Source: §A.6 · pp. 28, 29, 36, 37
  • Other examples include a 24 game, CryptoMath, a Word Ladder transformation, and Eulerian cycle or path decisions, each with a specified expression or list-style answer.The section also gives a sample hills-and-valleys counting task.Source: §A.6 · pp. 33, 36, 37, 38
  • Bar-based tasks ask for stock-trading profit, trapped water, h-index, largest histogram rectangle, longest increasing subsequence, container capacity, or counts of hills and valleys.The prompts request integer answers.Source: §A.6 · pp. 34–35

Graph Isomorphism

This appendix illustrates MM-HELIX task prompts and a reflective Snake solution, then identifies the benchmark’s category-level breakdown tables.

Graph Isomorphism

  • The Graph Isomorphism prompt asks whether two connected undirected planar graphs are isomorphic based on their planar structure.Source: Graph Isomorphism · p. 39

Reference Answer

  • The reference-answer prompts cover Hamiltonian cycles and paths, maximum flow, shortest distance, and topological ordering.The prompts request a cycle or path when one exists, a flow value or distance, or one valid ordering.Source: Reference Answer · pp. 39–40

A.7 EXAMPLES IN MM-HELIX-100K

  • A Snake example traces a 12-move strategy that collects all three apples while avoiding wall and self-collisions.The example emphasizes nearest-first planning, space preservation, growth planning, and checking the completed path.Source: §A.7 · pp. 43, 46

A.8 COMPLETE BREAKDOWN RESULTS

  • The appendix provides complete MM-HELIX results across four task types, with breakdown tables for Algorithms, Graphs, Puzzles, and Games.The supplied passages identify the tables’ scope but do not provide their cell values.Source: §A.8 · p. 47

A.8.1 BREAKDOWN RESULTS ON THE ALGORITHMS CATEGORY

A.8.2 BREAKDOWN RESULTS ON THE GRAPHS CATEGORY

A.8.3 BREAKDOWN RESULTS ON THE PUZZLES CATEGORY

A.8.4 BREAKDOWN RESULTS ON THE GAMES CATEGORY

Loading 2510.08540v2…