PaperlayerView exact revision on arXiv

Source-linked AI summary

Improve Vision Language Model Chain-of-thought Reasoning

Ruohong Zhang, Bowen Zhang, Yanghao Li, Haotian Zhang, Zhiqing Sun, Zhe Gan, Yinfei Yang, Ruoming Pang, Yiming Yang

arXiv:2410.16198v1cs.AIcs.CV

TL;DR

VLM training often relies on short answers with limited rationales, leaving open whether direct prediction teaches detailed chain-of-thought reasoning. The paper distills GPT-4o rationales for supervised fine-tuning and uses DPO on model-generated correct and incorrect rationale pairs; these methods improve CoT reasoning and generalization to direct answers.

  • Problem

    VLM training datasets often contain short answers with minimal rationales, and direct-prediction training improves CoT accuracy only marginally.

  • Method

    The paper distills 193k GPT-4o-generated CoT examples for supervised fine-tuning, then applies DPO to positive and negative model-generated rationale pairs.

  • Results

    Combining direct and CoT data improves average direct prediction by +7.3 (65.5 → 72.8) and CoT prediction by +11.7 (62.7 →74.4), while DPO adds +1.1 in CoT prediction.

  • Takeaways & Limitations

    Detailed rationale data strengthens VLM chain-of-thought reasoning, while model-generated preference data further improves reasoning accuracy and cross-task generalization.

Abstract

from arXiv · show

Chain-of-thought (CoT) reasoning in vision language models (VLMs) is crucial for improving interpretability and trustworthiness. However, current training recipes lack robust CoT reasoning data, relying on datasets dominated by short annotations with minimal rationales. In this work, we show that training VLM on short answers does not generalize well to reasoning tasks that require more detailed responses. To address this, we propose a two-fold approach. First, we distill rationales from GPT-4o model to enrich the training data and fine-tune VLMs, boosting their CoT performance. Second, we apply reinforcement learning to further calibrate reasoning quality. Specifically, we construct positive (correct) and negative (incorrect) pairs of model-generated reasoning chains, by comparing their predictions with annotated short answers. Using this pairwise data, we apply the Direct Preference Optimization algorithm to refine the model's reasoning abilities. Our experiments demonstrate significant improvements in CoT reasoning on benchmark datasets and better generalization to direct answer prediction as well. This work emphasizes the importance of incorporating detailed rationales in training and leveraging reinforcement learning to strengthen the reasoning capabilities of VLMs.

1 INTRODUCTION

The paper argues that short-answer VLM training provides limited transfer to detailed CoT reasoning and addresses this gap with distilled rationales, SFT, and DPO-based refinement.

  • CoT supports VLM interpretability and trustworthiness, but existing training data often contains short answers with limited rationales.
  • Training on 26k direct predictions improved direct accuracy by 2.9 points (70.2 to 73.1) but CoT accuracy by only 0.6 points (71.2 to 71.8).
  • The method uses GPT-4o to generate reasoning paths from short annotations across nine datasets requiring diverse reasoning skills.
  • The resulting SHAREGPT-4O-REASONING dataset contains 193k CoT examples for supervised fine-tuning.
  • DPO aligns VLM reasoning using positive and negative model-generated rationales selected by whether their predictions match short annotations.
  • The paper reports improved CoT performance and better direct-answer generalization, without requiring additional human-labeled data.

2 RELATED WORK

Prior work studies VLM reasoning across mathematics, science, charts, and localization, while alignment research applies preference optimization to factuality, hallucination reduction, and reasoning.

  • VLM reasoning has been evaluated across mathematical, college-level, scientific, chart-based, and object-localization tasks.
  • Related methods train VLMs to generate step-by-step solutions for mathematics, chart calculations, and object localization.
  • VLM alignment commonly uses DPO and PPO to improve factual accuracy and reduce hallucination.
  • Iterative or step-level DPO has also been used to improve mathematical CoT reasoning in language and vision-language models.

3 METHOD

The method distills GPT-4o rationales from short-answer VQA data, trains VLMs on CoT and direct formats, and uses preference pairs for DPO refinement.

  • The pipeline has three stages: GPT-4o CoT distillation, SFT with CoT and direct data, and RL for further CoT enhancement.
  • 3.1 REASONING DATA DISTILLATION: The distilled SHAREGPT-4O-REASONING data contains 193k visual CoT instances spanning world knowledge, charts, documents, text extraction, mathematics, and science.
  • 3.1 REASONING DATA DISTILLATION: CoT responses peak around 100 tokens, whereas direct answers are typically under 5 tokens after filtering incorrect GPT-4o predictions.
  • 3.2 SUPERVISED FINE-TUNING: Training uses separate direct and CoT prompts, with CoT responses placing the rationale before an extractable formatted answer.
  • 3.3 REINFORCEMENT LEARNING FOR ENHANCED REASONING: DPO forms preference data by comparing 32 SFT-generated candidate predictions per question with ground truth to identify positive and negative responses.
  • 3.3 REINFORCEMENT LEARNING FOR ENHANCED REASONING: The SFT experiments compare format-aligned, direct-only, CoT-only, and combined training data under both direct and CoT inference prompts.

4 SFT EXPERIMENTS FOR CHAIN-OF-THOUGHT LEARNING

The SFT experiments test whether CoT can be learned from short answers and how GPT-4o-distilled reasoning data affects VLM performance. CoT training improves reasoning and can also generalize to direct prediction, while combining CoT and direct data gives the strongest overall results.

  • Can reasoning be implicitly learnt from direct prediction?: Direct-only training improves direct accuracy by +5.6 (65.5 →71.1) but CoT performance remains lower at 65.6 versus 71.1 for direct prediction.The results suggest that short-answer training does not effectively teach CoT reasoning.
  • How effective is CoT reasoning data?: CoT-only training improves direct prediction by +4.2 (65.5 →69.7) and CoT prediction by +10.5 (62.7 →73.2).CoT performance exceeds direct prediction for this model, with especially large gains on ChartQA and MathVista.
  • How effective is CoT reasoning data?: Combining CoT and direct data yields the best overall performance, with gains of +7.3 in direct prediction and +11.7 in CoT prediction.The combined model reaches 72.8 direct and 74.4 CoT performance.
  • Ablation tests on data composition: CoT data composition matters for mathematics: MathVision adds +3.1 CoT performance over the format-only baseline, while G-LLaVA adds a further +1.5.The composition study also evaluates text-only math, science, and chart datasets.
  • Ablation tests on data composition: Combining SQA and AI2D provides additional gains on both science datasets, and adding ChartQA improves AI2D by +0.7.These results indicate that the science datasets are mutually beneficial in CoT training.
  • Comparing with SOTA model and GPT-4o: The SFT model outperforms Cambrian-7B on most benchmark datasets despite using fewer than 400k instruction examples versus Cambrian’s 7 million.The authors frame the comparison as evidence for CoT learning effectiveness rather than a claim of state-of-the-art performance.

5 RL EXPERIMENTS FOR ENHANCED CHAIN-OF-THOUGHT REASONING

The RL experiments use DPO preference pairs derived from model-generated rationales and short-answer feedback to refine CoT reasoning. The resulting model improves generation, verification, reranking, and general benchmark performance, with evidence that it penalizes early reasoning errors and hallucinations.

  • DPO training setup: The authors construct DPO preference pairs from correct and incorrect model-generated rationales across A-OKVQA, ChartQA, and mathematical reasoning tasks.The preference data contains 64.8k pairs, including 24.5k ChartQA, 18.3k A-OKVQA, and 22.0k math examples.
  • Can DPO calibrate reasoning?: The reasoning-pair DPO model improves CoT prediction by +1.1 and direct prediction by the same amount, exceeding the smaller gains from RLAIF-V.Gains appear across 7 of 8 datasets, with a slight decrease on SQA from 92.9 to 92.6.
  • DPO as verifier for CoT reasoning reranking: Reasoning-data DPO improves both best-of-N and weighted-voting reranking on ChartQA, A-OKVQA, and MathVista, whereas RLAIF-V shows less consistent gains.The DPO reward compares the DPO and SFT policies for candidate answers.
  • Additional DPO CoT performance on general datasets: Both DPO models outperform the SFT baseline on OCRBench, MMStar, and MMMU, with the reasoning-pair model slightly better.This extends the observed DPO benefit beyond the datasets used to construct preference pairs.
  • Additional DPO CoT performance on general datasets: On MMMU, reasoning-pair DPO generalizes to both weighted-voting and best-of-N reranking, while RLAIF-V improves CoT prediction without improving reranking metrics.The evaluation uses 988 multiple-choice questions from the Dev+Val split.
  • DPO credit assignment: Token-level DPO rewards are especially negative for early chart-interpretation mistakes and hallucinations, including incorrect labels and moon-phase answers.The model is reported to be more sensitive to knowledge-based errors than the SFT model.

6 CONCLUSION

The paper introduces SHAREGPT-4O-REASONING and uses it to improve VLM CoT reasoning through SFT, then further refines reasoning and direct-answer generalization with DPO. The authors conclude that these approaches enhance the robustness and interpretability of multimodal reasoning models.

  • Conclusion: DPO further improves the fine-tuned models’ reasoning and generalization to direct answer prediction without requiring additional human-labeled data.The conclusion presents DPO as a reinforcement-learning refinement stage after CoT data distillation and SFT.

A SHAREGPT-4O-REASONING Data for VLM CoT Reasoning;

The SHAREGPT-4O-REASONING data is produced by prompting GPT-4o to generate solutions from reference annotations and examining disagreements with annotated answers. The appendix documents filtering and examples of annotation errors in A-OKVQA and ChartQA.

  • Data generation: GPT-4o receives an annotation as a reference answer and generates its own solution for CoT distillation.The appendix includes the GPT-4o system and task prompts used for this process.
  • Data quality inspection: Human examination found annotation errors when GPT-4o answers differed, including misread text, incorrect object identification, and wrong chart values.Examples include “dentist” versus “heart,” beer versus water, and incorrect ChartQA answers.
  • Data filtering: Examples whose GPT-4o-generated answers differed from annotations were filtered to improve consistency and reduce potential errors.The released resource includes the SFT CoT data together with original distillation and filtered examples for reference.

B GPT-4O EVALUATION AND PROMPT OPTIMIZATION

The GPT-4o evaluation uses manually optimized direct and CoT prompts, with separate formatting requirements and development-set checks. The appendix documents these prompts and cautions that tuning does not guarantee optimality.

  • B GPT-4O EVALUATION AND PROMPT OPTIMIZATION: GPT-4o evaluation reports both direct and CoT predictions using prompts selected after comparing alternative prompt designs.The process follows prior prompt-optimization work and selects the best-performing prompts for reporting.
  • B GPT-4O EVALUATION AND PROMPT OPTIMIZATION: Prompt optimization uses a random 200-instance development set to inspect refusals, formatting errors, and mismatches with ground-truth labels.Manually designed prompts are compared against ground truth before benchmark evaluation.
  • B GPT-4O EVALUATION AND PROMPT OPTIMIZATION: ChartQA direct-prediction prompts were refined to produce short answers without units or special tokens.The appendix states that prompt #4 addressed full-sentence outputs and unwanted units or special tokens.
  • B GPT-4O EVALUATION AND PROMPT OPTIMIZATION: CoT prompts add output-format instructions so the final answer can be extracted, while direct and CoT prompts are evaluated with separate criteria.The appendix reports that a system prompt typically increases CoT scores by 0.5 points across datasets but does not improve direct prediction.
  • B.1 GPT-4O PROMPTS FOR EVALUATION: The prompt-tuning procedure does not guarantee optimal prompts and instead aims to roughly match previously reported benchmark values.The prompts are included as reference material for reproducing GPT-4o results.
  • B.1 GPT-4O PROMPTS FOR EVALUATION: The appendix provides optimized prompts for direct prediction across DocVQA, TextVQA, InfoVQA, OCRBench, MathVista, and MMMU.The prompt tables are included to support replication of GPT-4o benchmark results.
  • B.1 GPT-4O PROMPTS FOR EVALUATION: Separate tables provide the optimized CoT prompts, including continued pages for the full benchmark prompt set.The appendix lists the CoT prompt tables and their continuation pages.

C BASELINE EVALUATION

The baseline evaluation shows that LLAVA-NEXT-8B struggles to follow CoT instructions, while format-aligned training enables structured reasoning and answer extraction. However, the induced reasoning can remain incorrect, limiting gains over direct prediction.

  • Baseline behavior: LLAVA-NEXT-8B struggles with CoT prompting, including refusals, answer-first explanations, and descriptions without answers.These behaviors are illustrated with randomly sampled ChartQA outputs at temperature 1.0.
  • Evaluation procedure: CoT evaluation uses ChatGPT to extract letter choices or short answers from long-form model outputs.The extraction prompts are shown in the appendix, including a fallback output of NA when no answer can be derived.
  • Baseline results: LLAVA-NEXT-8B performs significantly worse for CoT predictions than for direct predictions and the format-aligned model.The comparison is attributed to the baseline’s inability to accurately follow the CoT format.
  • Format alignment: Training on 450 format-aligned examples enables LLAVA-NEXT-FORMAT to verbalize reasoning and place a short answer after “### Answer:”.This structure permits a rule-based extractor to retrieve answers.
  • Format alignment: Format alignment does not ensure correct reasoning: 32 sampled outputs produced only 3 correct answers, or 9.4% accuracy, on one ChartQA case.The resulting CoT gain over direct prediction is therefore slight on ChartQA.
  • Baseline results: LLAVA-NEXT-FORMAT has similar direct-prediction performance to LLAVA-NEXT-8B.The baseline is reported as LLAVA-NEXT-8B to avoid the evaluation and extraction difficulties of the LLAMA3-LLAVA-NEXT-8B baseline.

D NEARLY ZERO DATA LEARNING FOR COT REASONING

Minimal CoT format-aligned data can support self-taught reasoning through rejection sampling. The method improves CoT and direct prediction overall, while RFT substantially outperforms DPO on the reported ChartQA comparison.

  • Minimal CoT training: LLaVA-Next-STaR uses 450 CoT format-aligned examples with rejection sampling fine-tuning to learn from sampled reasoning chains.LLAVA-NEXT-DIRECT provides the baseline and direct-prediction training data.
  • RFT results: RFT improves both CoT reasoning and direct predictions overall, except for two reported data points.The study reports notable CoT gains on ChartQA, DocVQA, InfoVQA, AI2D, and MathVista, with roughly 1% direct-prediction gains on those datasets.
  • RFT results: TextVQA shows a significant drop in CoT performance after RFT, which the authors leave for future work.This is the explicitly noted exception among the reported dataset-level outcomes.
  • DPO versus RFT: On ChartQA, DPO reaches 72.3 (+0.5) for CoT and 74.2 (+0.5) for direct prediction, whereas RFT reaches 77.9 (+6.1) and 74.6 (+0.9).The comparison is conducted under the same conditions before the RFT experiments.

E SFT ABLATION EXPERIMENTS

The SFT ablations compare direct-only, CoT-only, and combined training across datasets, while format-aligned data teaches the model to follow the requested inference format.

  • SFT ablation settings: SFT ablations compare direct-only, CoT-only, and direct-plus-CoT training settings across each dataset.The experiments also incorporate format-aligned data for the corresponding direct or CoT inference format.

F ADDITIONAL DPO EXPERIMENTS

Additional experiments show that DPO performance depends on response truncation, while RFT exhibits dataset-specific trade-offs and weaker cross-dataset generalization.

  • Truncating Responses for DPO: 90-token truncation empirically yields the best DPO results, whereas no truncation causes performance to decline.This setting was therefore used for the DPO experiments.
  • DPO vs. RFT: RFT alone achieves the best A-OKVQA result but entirely loses the model’s ability to generate short answers.Adding format-aligned data introduces a trade-off between A-OKVQA and other datasets.
  • DPO vs. RFT: Combining RFT datasets improves only ChartQA, while A-OKVQA and MathVista performance declines.Balancing RFT across datasets is challenging when the SFT model already performs relatively well on basic tasks.
  • DPO vs. RFT: DPO demonstrates consistent gains across datasets and better generalization than RFT.The comparison table presents SFT and DPO alongside RFT ablations trained on individual and combined datasets.
Loading 2410.16198v1…