PaperlayerView exact revision on arXiv

Source-linked AI summary

LLaVA-CoT: Let Vision Language Models Reason Step-by-Step

Guowei Xu, Peng Jin, Ziang Wu, Hao Li, Yibing Song, Lichao Sun, Li Yuan

arXiv:2411.10440v6cs.CV

TL;DR

Vision-language models often struggle with systematic and structured reasoning on complex visual question-answering tasks. LLaVA-CoT introduces autonomous four-stage reasoning, structured training data, and stage-wise retracing search. It reports improved multimodal reasoning performance over its base model and larger or closed-source models, while overly complex images can still cause failures.

  • Problem

    Existing VLMs often lack sufficiently systematic and structured reasoning for complex multimodal tasks.

  • Method

    LLaVA-CoT uses four reasoning stages, the LLaVA-CoT-100k dataset, and stage-wise retracing search for test-time scaling.

  • Results

    LLaVA-CoT achieves superior performance across multimodal reasoning benchmarks relative to many larger and closed-source models, with a 5.8% average benchmark score increase over its base model.

  • Takeaways & Limitations

    Structured autonomous reasoning and retracing provide an approach for enhancing multimodal reasoning and scaling inference at test time.

  • Takeaways & Limitations

    Overly complex input images can exceed the model’s visual understanding capabilities, and retracing may still fail to produce the correct answer.

Abstract

from arXiv · show

Large language models have demonstrated substantial advancements in reasoning capabilities. However, current Vision-Language Models (VLMs) often struggle to perform systematic and structured reasoning, especially when handling complex visual question-answering tasks. In this work, we introduce LLaVA-CoT, a large VLM designed to conduct autonomous multistage reasoning. Unlike chain-of-thought prompting, LLaVA-CoT independently engages in sequential stages of summarization, visual interpretation, logical reasoning, and conclusion generation. This structured approach enables LLaVA-CoT to achieve marked improvements on reasoning-intensive tasks. To accomplish this, we construct the LLaVA-CoT-100k dataset, integrating samples from various visual question answering sources and providing structured reasoning annotations. Besides, we propose a test-time stage-wise retracing search method (SWIRES), which enables effective and efficient test-time scaling. Remarkably, with only 100k training samples and test-time scaling, LLaVA-CoT not only outperforms its base model by 9.4% on a wide range of multimodal reasoning benchmarks, but also surpasses the performance of larger and even closed-source models, such as Gemini-1.5-pro, GPT-4o-mini, and Llama-3.2-90B-Vision-Instruct. The code, dataset, and pre-trained weights are publicly available at https://github.com/PKU-YuanGroup/LLaVA-CoT.

1. Introduction

LLaVA-CoT addresses systematic and structured reasoning shortcomings in VLMs by organizing inference into four stages and adding stage-wise retracing search. It improves reasoning performance across multimodal benchmarks relative to its base model and other models.

  • Existing VLMs often begin without organizing the problem and may introduce premature conclusions that lead to flawed reasoning paths.
  • LLaVA-CoT uses summary, caption, reasoning, and conclusion stages to address questions in a structured, multistage manner.The stages respectively outline the task, describe relevant image content, analyze the question, and summarize the answer.
  • LLaVA-CoT trains on structured responses generated with GPT-4o and uses dedicated tags to mark stage boundaries.The model can transition between stages without additional test-time prompting or intervention.
  • SWIRES generates multiple candidates at each reasoning stage, retains promising responses with a reward model, and retraces when a stage’s candidates are suboptimal.Retracing regenerates an earlier stage to improve error correction during reasoning.
  • LLaVA-CoT substantially outperforms traditional CoT prompting on complex reasoning tasks and supports scalable, more reliable inference.Experiments cover MMStar, MMBench, MMVet, MathVista, AI2D, and HallusionBench.

2. Related Works

Prior VLMs use visual reasoning mechanisms and increasingly leverage LLM reasoning or chain-of-thought prompting. Test-time scaling methods include voting, best-of-N search, and beam search, each with stated task or evaluation limitations.

  • Visual reasoning in VLMs combines visual perception with high-level cognition and has been approached through neural-symbolic modeling and improved visual encoding.
  • Chain-of-thought prompting decomposes difficult questions into reasoning steps and has improved VLM performance on reasoning tasks.
  • Majority voting suits standard-answer problems, whereas best-of-N and beam search generate and evaluate multiple candidate responses.Evaluating the accuracy of complete responses can be challenging for best-of-N search.

3. Method

LLaVA-CoT decomposes answer generation into four structured stages, trains on a GPT-4o-annotated VQA dataset, and scales inference through stage-wise search with retracing. The method supports autonomous stage transitions and error correction during test-time search.

  • 3.1.1. Reasoning Stages: LLaVA-CoT’s four stages summarize the question, describe relevant visual elements, perform logical reasoning, and synthesize the final answer.Only the conclusion is directly returned to the user; the preceding stages are internal reasoning stages.
  • 3.1.1. Reasoning Stages: The model initiates and completes all four stages in one inference pass using dedicated summary, caption, reasoning, and conclusion tags.
  • 3.1.1. Reasoning Stages: LLaVA-CoT can transition between stages without external intervention and independently manage its reasoning process after training.
  • 3.1.2. Data Preparation and Model Training: The LLaVA-CoT-100k dataset contains 99k image-question-answer pairs with GPT-4o-generated summary, caption, reasoning, and conclusion annotations.The data integrates samples from several widely used VQA datasets and is intended for supervised fine-tuning.
  • 3.2.1. Stage-wise beam search: Stage-wise beam search generates M candidates per reasoning stage, retains the top N with a reward model, and adapts search granularity to task complexity.
  • 3.2.2. Stage-wise retracing search: SWIRES retraces to an earlier stage when no current candidate exceeds a reward threshold, regenerating responses for continued search.This mechanism targets local optima and selection biases in ordinary stage-wise beam search.

4. Post-Training Performance

Post-training with structured data improves LLaVA-CoT across diverse multimodal benchmarks, with gains concentrated in reasoning-intensive skills. Ablations show that both the multistage dataset format and structured tags contribute to performance.

  • Benchmark Results: The evaluation covers general visual question answering, mathematical and scientific reasoning, and hallucination and visual-illusion handling.MMStar, MMBench, and MMVet assess general VQA; MathVista and AI2D assess mathematical or scientific reasoning; HallusionBench targets hallucinations and illusions.
  • Benchmark Results: LLaVA-CoT improves the average benchmark score by 5.8% over Llama-3.2-11B-Vision-Instruct across general VQA, mathematical reasoning, scientific VQA, and hallucination control.The improvement uses only 100k training examples.
  • Ablation Study: Training directly on original Q&A pairs yields lower average performance, and performs worse than the base model on the detailed-response MMVet benchmark.This comparison supports the importance of the multistage format in LLaVA-CoT-100k.
  • Ablation Study: Removing structured tags causes a significant performance drop, indicating that tags facilitate reasoning and improve model performance.The complete LLaVA-CoT configuration uses structured tags in the LLaVA-CoT-100k training data.
  • Skill Analysis: On MMStar, gains are larger for instance reasoning, logical reasoning, math, and science & technology than for coarse or fine-grained perception.The reported pattern suggests that the method mainly improves reasoning capabilities.

5. Test Time Scaling

SWIRES provides stronger test-time scaling than stage-wise beam search and best-of-N under comparable computational conditions. Its backtracking design continues improving beyond the point where the other methods plateau.

  • Approximate-Scale Comparison Analysis: SWIRES performs better than stage-wise beam search, which outperforms best-of-N under similar test conditions.The comparison uses a shared reward model and comparable computational constraints.
  • Approximate-Scale Comparison Analysis: Adding a backtracking step to stage-wise beam search enhances model performance.
  • Scaling Trend Analysis: All three methods show scaling relative to the baseline accuracy of 57.6 on MMStar without test-time scaling.
  • Scaling Trend Analysis: Stage-wise beam search and best-of-N plateau around 10,000 seconds, while best-of-N slightly declines beyond that point.
  • Scaling Trend Analysis: SWIRES continues scaling beyond 10,000 seconds, demonstrating the strongest scaling effect as computation time increases.The Figure 5 time axis is logarithmic and the evaluation uses MMStar on a single A800 node.

6. Comparison to State-of-the-Art VLMs

On reasoning-focused benchmarks, LLaVA-CoT consistently surpasses many similarly sized or larger open-source VLMs and even certain closed-source models. The comparisons position it as competitive for multimodal reasoning tasks.

  • Evaluation Setting: The comparison uses six benchmarks requiring advanced reasoning, including filtered MMStar-R, MMBench-R, and MMVet-R variants that remove perception- and OCR-only tasks.
  • State-of-the-Art Comparison: LLaVA-CoT surpasses the closed-source models GPT-4o-mini and Gemini-1.5-pro.

7. Conclusion

LLaVA-CoT combines autonomous multistage reasoning, structured training annotations, and stage-wise retracing search to improve multimodal reasoning. The paper identifies reinforcement learning as a possible direction for further work.

  • Conclusion: LLaVA-CoT performs structured, autonomous reasoning through four distinct stages.The paper presents the LLaVA-CoT-100k dataset and SWIRES as its two main contributions.
  • Conclusion: The LLaVA-CoT-100k dataset supports training on systematic, structured responses with detailed reasoning annotations.
  • Conclusion: Stage-wise retracing search enables effective test-time scaling for multimodal reasoning.
  • Conclusion: Future research could explore reinforcement learning to further improve complex multimodal reasoning.

Supplementary Material

The supplementary material illustrates two recurring VLM reasoning failures and documents a structured data-generation and verification process.

  • Illustrative Cases of Reasoning Challenges in VLMs: VLMs may begin without organizing the problem and available information, or deviate from logical reasoning toward premature conclusions.Figure 6 presents these two challenges in Llama-3.2-11B-Vision-Instruct responses.
  • Data Generation Scheme: The data-generation process gives GPT-4o an image, question, and original answer, then requests responses in four prescribed sections.The sections are SUMMARY, CAPTION, REASONING, and CONCLUSION.
  • Data Verification: Generated examples are checked for format compliance before their conclusions are filtered for refusals or disagreement with the standard answer.The verification prompt labels responses valid when they are non-refusals and semantically align with the standard answer.

C. Training Hyperparameters

This section describes the training framework and points to the tabulated hyperparameter configurations used for training.

  • Training Hyperparameters: Training uses the llama recipes framework.The section introduces the framework used for training.
  • Training Hyperparameters: The training hyperparameter configurations are listed in Table 6.The table is titled “Hyperparameter configurations used in training.”
  • Training Hyperparameters: Table 6 is the designated reference for the training settings described in this section.The supplied passage identifies the table but does not enumerate its parameter values.

D. Implementation Details of Stage-wise Retrace

Stage-wise Retrace generates and scores candidates across caption, reasoning, and conclusion stages, with preset conditions determining whether to stop or backtrack.

  • Stage-wise Retrace Algorithm: The algorithm starts with one summary, generates M captions, scores them, and retains the top N captions.These operations form the initial candidate-generation stages of Stage-wise Retrace.
  • Stage-wise Retrace Algorithm: For each retained caption, the algorithm generates and reward-scores M reasonings, collecting candidates and scores for later selection.The pseudocode evaluates each reasoning and appends it to the candidate and score lists.
  • Stage-wise Retrace Algorithm: If preset conditions are not satisfied, the procedure increments the backtracking counter until the configured limit is reached.After the loop, it selects the top N reasonings, generates one conclusion per reasoning, scores all conclusions, and returns the best one.
  • Reward Model: The reward model evaluates stage outputs online, using IXC-2.5-Reward to assess their quality during inference.The implementation notes that this multimodal reward model is used because few open-source multimodal reward models align with human preferences.
  • Backtrack Condition: Backtrack cutoff equals reward_mean + Z × reward_std, with Z set to 0.2533 in the reported configuration.The authors state that scores above this threshold represent 40% of the distribution and use the second-largest reasoning score to decide when backtracking is unnecessary.
  • Backtrack Condition: The reward-score mean and standard deviation are estimated from the reasoning-phase model outputs on MMStar, whose distribution is described as close to Gaussian.The preset condition is motivated by retaining sufficiently good reasoning references for conclusion selection.

F. Further Experiments on the Effectiveness of CoT

Further experiments examine the structured CoT design, stage ordering, test-time scaling, and behavior on fill-in-the-blank and open-ended questions, while documenting a limitation.

  • Effectiveness of CoT: Structured CoT prompting improves GPT-4o but not Llama-3.2-Vision, whereas supervised fine-tuning on structured CoT data improves Llama-3.2-Vision.The comparison indicates that prompting alone was insufficient for the Llama model in this experiment.
  • Stage Ordering: Training with shuffled SUMMARY, CAPTION, REASONING, and CONCLUSION stages produces almost no improvement, indicating that their proper order is crucial for response quality.The experiment tests whether the natural reasoning order contributes to the effectiveness of structured CoT.
  • Test-Time Scaling: The scaling experiments vary candidates or retracing iterations across Best of N, Stage-wise Beam Search, and SWIRES.The reported points use N values of 1, 3, 4, and 8; candidate counts of 1, 4, 6, and 19; and SWIRES retracing iterations of 0, 1, and 3, respectively.
  • Test-Time Scaling: Test-time scaling effectively corrects errors made during generation in the comparison of LLaVA-CoT with and without scaling.Figure 7 reports the comparison and supports the effectiveness of the proposed scaling approach.
  • Additional Question Types: Llama-3.2-11B-Vision-Instruct often responds inadequately to fill-in-the-blank and open-ended questions through vague or nonspecific reasoning, causing factual inaccuracies or incomplete answers.Figure 8 extends the comparison beyond reasoning and science-based multiple-choice questions.
  • Limitations: In rare cases, overly complex images exceed the model’s visual understanding, and retracing may still fail to produce the correct answer.The authors also report that LLaVA-CoT can get lost during retracing or hallucinate while attempting to reach an answer.
Loading 2411.10440v6…