Source-linked AI summary
LlamaV-o1: Rethinking Step-by-step Visual Reasoning in LLMs
Omkar Thawakar, Dinura Dissanayake, Ketan More, Ritesh Thawkar, Ahmed Heakl, Noor Ahsan, Yuhao Li, Mohammed Zumri, Jean Lahoud, Rao Muhammad Anwer, Hisham Cholakkal, Ivan Laptev, Mubarak Shah, Fahad Shahbaz Khan, Salman Khan
TL;DR
Existing methods provide limited support for evaluating and performing step-by-step multimodal reasoning, especially when reasoning-chain errors are not accurately reflected. This paper introduces a benchmark, step-level metric, and curriculum-trained LlamaV-o1, which outperforms recent open-source visual reasoning methods and performs favorably against proprietary models.
Problem
Existing approaches struggle with step-by-step multimodal reasoning and reference-free reasoning-chain evaluations can assign high scores despite disruptive minor errors.
Method
The paper introduces VRC-Bench, a step-level visual reasoning metric, and LlamaV-o1 trained with progressive curriculum learning and Beam Search.
Results
67.33% average score across six established benchmarks makes LlamaV-o1 the highest-scoring open-source model, surpassing LLaVA-CoT at 63.50%.
Takeaways & Limitations
The framework combines structured training, efficient inference, and rigorous evaluation to support accurate, coherent, robust multimodal reasoning across diverse tasks.
Abstract
from arXiv · showhide
Reasoning is a fundamental capability for solving complex multi-step problems, particularly in visual contexts where sequential step-wise understanding is essential. Existing approaches lack a comprehensive framework for evaluating visual reasoning and do not emphasize step-wise problem-solving. To this end, we propose a comprehensive framework for advancing step-by-step visual reasoning in large language models (LMMs) through three key contributions. First, we introduce a visual reasoning benchmark specifically designed to evaluate multi-step reasoning tasks. The benchmark presents a diverse set of challenges with eight different categories ranging from complex visual perception to scientific reasoning with over 4k reasoning steps in total, enabling robust evaluation of LLMs' abilities to perform accurate and interpretable visual reasoning across multiple steps. Second, we propose a novel metric that assesses visual reasoning quality at the granularity of individual steps, emphasizing both correctness and logical coherence. The proposed metric offers deeper insights into reasoning performance compared to traditional end-task accuracy metrics. Third, we present a new multimodal visual reasoning model, named LlamaV-o1, trained using a multi-step curriculum learning approach, where tasks are progressively organized to facilitate incremental skill acquisition and problem-solving. The proposed LlamaV-o1 is designed for multi-step reasoning and learns step-by-step through a structured training paradigm. Extensive experiments show that our LlamaV-o1 outperforms existing open-source models and performs favorably against close-source proprietary models. Compared to the recent Llava-CoT, our LlamaV-o1 achieves an average score of 67.3 with an absolute gain of 3.8\% across six benchmarks while being 5 times faster during inference scaling. Our benchmark, model, and code are publicly available.
1 Introduction
The paper targets the difficulty of evaluating and improving step-by-step visual reasoning in multimodal models. It introduces a benchmark, step-level metric, and curriculum-trained model to address this gap.
- Motivation: Existing methods improve step-by-step reasoning in text tasks but struggle with multimodal reasoning across sequential visual problems.Visual reasoning requires connecting diverse information across images, text, and multiple reasoning steps.
- Contributions: VRC-Bench evaluates multi-step visual reasoning across eight categories, more than 1,000 samples, and 4,173 manually verified reasoning steps.The categories include visual, mathematical, scientific, medical, cultural, chart, OCR, and complex perception tasks.
- Contributions: The proposed metric evaluates individual reasoning steps for correctness and logical coherence rather than relying only on final-answer accuracy.This provides more detailed insight into the quality of generated reasoning chains.
- Contributions: LlamaV-o1 combines multi-step curriculum learning with Beam Search to develop reasoning skills progressively while improving inference efficiency.Training progresses from simpler captioning-related tasks to more complex multi-step reasoning scenarios.
- Results: 3.8% absolute average-score gain across six benchmarks and 5× faster inference are reported for LlamaV-o1 versus Llava-CoT.The comparison is reported for the combined curriculum-learning and Beam Search approach.
2 Related Works
Prior work spans language, visual, and multimodal reasoning benchmarks, but existing visual benchmarks do not provide step-by-step reasoning evaluation. VRC-Bench is positioned to cover diverse visual reasoning contexts with explicit reasoning chains.
- Reasoning with LLMs: LLM reasoning research has progressed from neural-symbolic methods toward approaches that leverage the inherent reasoning abilities of large language models.The related work discusses explicit formal-language reasoning and newer inference-time computation approaches.
- Reasoning with VLMs: Visual reasoning requires both visual perception and higher-level cognitive abilities across science, mathematics, robotics, and question answering.These applications motivate reasoning methods specialized for vision-language models.
- Benchmarks for Visual Reasoning: Existing benchmarks test capabilities such as counting, logical inference, science reasoning, mathematical reasoning, and multi-hop question answering across visual contexts.Examples include CLEVR, StrategyQA, ScienceQA, MathVista, and ShareGPT-4o-Reasoning.
- Benchmarks for Visual Reasoning: Existing visual reasoning benchmarks do not provide step-by-step reasoning evaluation, motivating a benchmark focused on reasoning chains.The proposed benchmark addresses this limitation by evaluating sequential reasoning rather than only task outcomes.
3 Step-by-Step Visual Reasoning Benchmark: VRC-Bench
VRC-Bench is a structured benchmark for assessing both reasoning-chain progression and final answers across diverse multimodal domains. Its construction combines heterogeneous datasets, semi-automatic chain generation, manual verification, and reference-based step-level evaluation.
- Benchmark Overview: VRC-Bench evaluates both the logical progression of reasoning chains and the accuracy of final outcomes generated by LMMs.The benchmark is designed for complex multi-step visual reasoning scenarios.
- Benchmark Domains: The benchmark integrates datasets spanning mathematics, science, medicine, social sciences, and data interpretation to cover diverse reasoning scenarios.Its dataset structure includes multiple specialized domains and real-world multimodal applications.
- Benchmark Domains: Benchmark samples cover mathematical and logical reasoning, scientific reasoning, medical imaging, cultural and social understanding, visual perception, OCR, and document understanding.Examples include MathVista, DynaMath, ChartQA, ScienceQA, MMMU-Medical, ALM-Bench, LogicVista, Blink-IQ, and Doc-VQA.
- Benchmark Creation: GPT-4o generates detailed reasoning chains and answers, after which verifiers review them and add or correct steps to establish a trustworthy ground truth.Examples with fewer than three reasoning steps are dropped after verification, with stated exceptions.
- Evaluation Framework: Reference-free metrics can assign high scores despite minor errors disrupting a reasoning chain, so the paper uses ground-truth comparison for evaluation.The proposed reference-based metric uses GPT-4o and measures attributes including faithfulness, informativeness, hallucination, redundancy, missing steps, semantic coverage, and reasoning alignment.
4 Proposed Step-by-Step Visual Reasoning Model: LlamaV-o1
LlamaV-o1 advances step-by-step visual reasoning through curriculum learning that progressively moves from foundational multimodal understanding to detailed reasoning and final answers. Its reasoning pipeline combines structured stages with Beam Search to improve accuracy, interpretability, and inference efficiency.
- Model Overview: LlamaV-o1 combines curriculum learning with Beam Search to develop multimodal reasoning skills progressively while optimizing reasoning paths.The approach is designed to balance reasoning accuracy with computational complexity.
- Curriculum Learning: Curriculum learning progresses from simpler multimodal tasks toward increasingly complex multi-step reasoning scenarios.The progression is intended to build foundational skills before detailed reasoning and final answer generation.
- Reasoning Design: Breaking reasoning into explicit stages supports systematic processing, error reduction, and interpretability of multimodal outputs.The structured pathway provides a basis for verifying and refining model outputs.
- Multi-Step Reasoning Stages: The five reasoning stages are task understanding, task summarization, detailed caption generation, logical reasoning, and final answer generation.The stages move from interpreting the question and visual context to extracting relevant data, reasoning over it, and producing the answer.
- Two-Stage Training: Stage 1 trains approach summaries and detailed captions, while Stage 2 adds detailed reasoning and final answer generation.Stage 1 uses PixMo and Geo170K data; Stage 2 uses the original Llava-CoT dataset.
5 Experiments
Experiments evaluate LlamaV-o1 on VRC-Bench and six established multimodal benchmarks spanning visual, mathematical, scientific, and hallucination-related reasoning. The model achieves strong category-wise and cross-benchmark performance, with Beam Search further improving results over curriculum learning with multi-step CoT alone.
- VRC-Bench Results: 56.49 final-answer accuracy and 68.93% step score show strong VRC-Bench performance against evaluated models.The results combine final-answer correctness with step-level reasoning quality.
- Category-Wise Results: 93.44% on OCR & Document Understanding, 86.75% on Scientific Reasoning, and 83.18% on Chart & Diagram Understanding are reported category scores.These results illustrate strong performance across visually and logically demanding categories.
- Benchmark Results: 67.33% average score makes LlamaV-o1 the best-performing open-source model across six established multimodal benchmarks.It surpasses LLaVA-CoT’s 63.50% average score and performs strongly on MMVet and Hallusion.
- Comparative Analysis: The proposed approach outperforms recent open-source visual reasoning methods and achieves favorable results against close-source counterparts.The qualitative comparison also reports stronger reasoning steps and final answers than Llava-CoT on VRC-Bench examples.
- Ablations: Beam Search raises the six-benchmark average from 66.08% with curriculum learning and multi-step CoT to 67.33%.The gains are reported on MMVet, MMStar, and AI2D among other benchmarks.
- Inference Efficiency: Beam Search provides higher MMVet scores with lower inference time through linear scaling, whereas Llava-CoT’s stage-level beam search has quadratic scaling.The comparison is conducted on a single NVIDIA A100 GPU.
6 Conclusion
The paper advances multimodal reasoning through a benchmark, metric, and curriculum-trained model. It reports improved performance, efficiency, generalization, and robustness across diverse tasks.
- The framework combines a new benchmark, a novel metric, and a curriculum-trained multimodal reasoning model.
- LlamaV-o1 achieves state-of-the-art performance on challenging benchmarks while maintaining inference efficiency.
- Curriculum learning progressively develops foundational reasoning skills and improves generalization and robustness across diverse tasks.
- The evaluation combines the proposed benchmark with widely recognized datasets to assess the model’s design choices.
A.1 Generating reasoning Steps from Closed Sourced Models
The closed-source-model prompt enforces explicit, image-grounded, step-by-step reasoning before producing a final answer. It also accommodates variable task complexity and invalid answer choices.
- The prompt requires models to describe each action and explain its execution in a clear, logical progression.
- Models may take as many reasoning steps as necessary to handle tasks with varying complexity.
- When no answer option is valid, the prompt instructs the model to respond “None of the choices provided.”
- The final response includes a solution or conclusion derived from the reasoning process.
- Each reasoning step must explain how the image or images guide the solution.
A.2 System Prompt used to Evaluate Reasoning Steps
The evaluation prompt compares model reasoning with ground-truth reasoning using multiple 1–10 metrics covering fidelity, information content, coherence, alignment, and completeness. It requires direct comparison, consistent scoring, and an averaged overall score.
- The framework evaluates reasoning quality with metrics including faithfulness, informativeness, repetition, hallucination, redundancy, semantic coverage, alignment, commonsense, and missing steps.
- Faithfulness and informativeness assess whether steps remain true to the source and capture relevant information.
- Repetition, hallucination, and redundancy identify unnecessary paraphrasing, invented content, and steps that add no value.
- Semantic coverage, reasoning alignment, and commonsense assess essential meaning, correspondence with the reference chain, and necessary background reasoning.
- Missing Step measures whether necessary reasoning steps are absent and whether omissions affect the outcome.
- Evaluators must compare ground truth and model responses directly, follow scoring thresholds, and compute the Overall Score as the average of all metric scores.
A.3 Response format used to generate structured evaluation scores
The response format uses a strict JSON schema to capture structured reasoning evaluations. It standardizes metric scores and related outputs for comparison, interpretability, reproducibility, and future extension.
- The JSON schema enforces a structured format for detailed, systematic, and transparent evaluation scores.
- Standardization facilitates model comparison, reduces ambiguity, and improves reproducibility of results.
- Responses are organized around reasoning steps, metric scores, and final answers to support fine-grained performance analysis.
- The modular schema supports adding metrics or evaluation criteria as the benchmark evolves.
- The schema defines numeric fields for ten evaluation metrics and the Overall Score, with all fields required and additional properties disallowed.
A.4 Evaluating reasoning steps using gpt-4o as a judge
The evaluate_steps function uses GPT-4o-mini to assess model-generated reasoning steps against task questions and ground-truth reasoning. It structures the evaluation as a deterministic conversation and returns the judge’s evaluation content.
- The evaluate_steps function compares a model response with the task question and ground-truth reasoning.It takes the question, ground-truth reasoning, and LLM response as inputs.
- The evaluation packages a system prompt and user content containing the question, ground truth, and model response.
- GPT-4o-mini performs the assessment with temperature 0.0 and a maximum output of 500 tokens.The function returns the model’s message content.
A.5 Evaluating final answer accuracy
The final-answer evaluation uses a secondary prompt to compare model predictions with ground truth. It produces a binary semantic-match score, complementing the reasoning-step evaluation.
- The comparison function evaluates whether the model prediction matches the ground-truth answer semantically.
- The evaluator assigns 1 for a match and 0 for a mismatch.
- The final-answer prompt requires the evaluator to return only the numeric score without additional commentary.
- The result is obtained from the model response after submitting the evaluation conversation.