Measuring and Improving Chain-of-Thought Reasoning in Vision-Language Models

Introduction

• The study measures reasoning performance as correct high-level inference beyond immediately perceived information, and consistency as identifying the reasoning chains that lead to that inference.The authors position grounded and consistent visual reasoning as a key direction for improving VLMs that generate human-like responses using multimodal instructional data and LLMs (Li et al., 2023a; Liu et al., 2023c,a; Chen et al., 2023).Source: §1 · p. 1
• Earlier consistency evaluation used coarse-grained rationales (Zellers et al., 2019), human evaluation (Lu et al., 2022a), or similarity measures (Wei et al., 2023), limiting scalability and precision.Fine-grained reasoning-step annotations are costly to collect and can be inconsistent across annotators.Source: §1 · p. 1
• An LLM-Human-in-the-Loop pipeline builds CURE from Sherlock, using limited human instructions and filtering rules to produce 1,622 human-verified inference samples with CoT chains.The pipeline is designed to reduce annotation cost while generating high-quality data for zero-shot evaluation.Source: §1 · p. 1
• CURE uses progressive reasoning subquestions and a multiple-choice format to evaluate both overall inference for a designated image area and the intermediate chain leading to the answer.The chains range from recognition to cognition; multiple choice enables automatic evaluation given the difficulty of evaluating natural-language generation.Source: §1 · p. 2
• Even BLIP-2 falls short of humans in overall visual reasoning; success depends on LLM components, visual inputs, and instruction finetuning, while intermediate-step reliability can diverge from final-answer accuracy.The authors report that intermediate reasoning can be unreliable even when the final inference is accurate, and vice versa.Source: §1 · p. 2
• Approximately 4% relative improvement over the SOTA in both reasoning performance and consistency is reported for the proposed two-stage rationale-training framework.The first stage trains on LLM-generated step-by-step reasoning samples; the second incorporates LLM feedback to examine reasoning and address inconsistencies or hallucinations in rationales.Source: §1 · p. 2

2 Related Work

• Comprehensive diagnostic studies of VLM reasoning consistency remain scarce despite prior work measuring visual reasoning performance and coarse-grained rationale quality.Prior approaches include Sherlock, multiple-choice evaluation such as VCR, human evaluation of generated rationales, and similarity to ground-truth rationales.Source: §2 · p. 2
• VLMs can fail on component subquestions even when those questions relate to the main problem (Ray et al., 2019; Jing et al., 2022; Selvaraju et al., 2020; Wang et al., 2022f; Lu et al., 2022a; Wei et al., 2023).For example, a model may identify a mountain as large but classify it as small when asked whether the mountains are small (Ray et al., 2019).Source: §2 · p. 2
• The paper uses coherent chains of subquestions that support high-level inference to measure how consistent and grounded VLM reasoning is.This contrasts with prior work focused on coarse-grained rationale evaluation or individual subquestions.Source: §2 · p. 2

3 CURE Benchmark

• CURE builds on Sherlock (Hessel et al., 2022), adding fine-grained CoT reasoning chains and candidate answers to assess both high-level inference and its supporting reasoning.The chains comprise coherent subquestions linked to the high-level inference, while plausible incorrect alternatives make inference and subquestion evaluation multiple-choice.Source: §3 · p. 3

3.1 LLM-Human-in-the-Loop Data Generation Pipeline

• The two-stage construction pipeline uses LLMs for most tasks, with researchers iteratively correcting errors during generation and filtering.The first stage generates a preliminary dataset that may contain failures; the second filters error cases.Source: §3.1 · p. 3

3.1.1 Stage-1: Preliminary Generation

• Stage 1 iteratively refined prompts on 10,000 Sherlock examples, using GPT-4 and an in-context demonstration to generate 1.6k reasoning chains and six candidate answers.The reasoning chains typically progress from perception questions to more complex visual inference; candidate alternatives are relevant to the image but factually inaccurate relative to the ground truth.Source: §3.1.1 · pp. 3–4

3.1.2 Stage-2: Filtering

• Stage 2 repeatedly applied failure-mode-specific LLM filters for six rounds, then human-reviewed the full dataset for validity and sample diversity.Filtering was estimated to reduce labor by 50%, but human review was retained because failures remained, including examples relying on simplistic visual shortcuts.Source: §3.1.2 · p. 4

3.2 Human Evaluation

• Only 3% of CURE evaluation samples had identified issues; 2% showed inconsistent reasoning chains and 1% had incorrect subquestion answers.Three annotators assessed 200 samples, answering multiple-choice questions and flagging known or other failure modes; the findings validate CURE’s quality and the pipeline’s filtering.Source: §3.2 · pp. 4–5

3.3 Evaluation Metrics

• CURE frames each image’s high-level inference and each reasoning-chain subquestion as multiple-choice, with six candidates for each.The proposed metrics assess both VLM reasoning ability and reasoning consistency.Source: §3.3 · p. 5

3.3.1 Metrics for Reasoning Performance

• The reasoning-performance metrics assess high-level inference accuracy (Rh), correct answers across all CoT subquestions (Rcot), and success on both for each image (Ro).Ro requires successful high-level and CoT reasoning for every image.Source: §3.3.1 · p. 5

3.3.2 Metrics for Reasoning Consistency

• The consistency metrics assess high-level inference correctness given correct subquestions (Cf) and all subquestion answers being correct given a correct inference (Cb).Cf measures forward reasoning consistency; Cb measures backward reasoning consistency.Source: §3.3.2 · p. 5

4 Approach

• CoTBLIP combines SFT and RLAIF to generate visually detailed, reasonable rationales intended to facilitate high-level inference and mitigate hallucination.The framework uses BLIP-2-T5xl as its backbone (Xu et al., 2023; Fu et al., 2023).Source: §4 · pp. 5–6, Figure 3 · p. 6

Stage-1: SFT

• SFT trains CoTBLIP on refined LLaVA reasoning samples, converting 77K GPT-4-generated examples grounded in five human-annotated COCO captions and bounding boxes into logical, consistent, succinct chains.The source samples were produced from COCO images (Lin et al., 2014); LLaVA is cited as Liu et al. (2023c).Source: §4 · p. 6

Stage-2: RLAIF

• RLAIF addresses potentially inaccurate, inconsistent, or hallucinated rationales and SFT’s dependence on dense human annotations, which limits scalability and broad visual-concept generalizability.The approach inspects reasoning chains, which the authors describe as more straightforward than directly scrutinizing high-level inferences.Source: §4 · p. 6
• For wild image-caption pairs, CoTBLIP generates three chains per image, and GPT-3.5-Turbo evaluates their sophistication, consistency, and groundedness against the caption.Pairwise comparisons rank the chains, a consistency check removes instances with conflicting rankings, and SBU Captions yields around 27K preference samples; pairwise comparison adapts Ouyang et al. (2022).Source: §4 · p. 6
• Conditional Reinforcement Learning trains CoTBLIP with <Good> on the top-ranked chain and <Bad> on the other two, maximizing each chain’s likelihood without a separate reward model.The method is selected for stability observed in prior work (Lu et al., 2022b; Liu et al., 2023b; Laskin et al., 2022).Source: §4 · pp. 6–7

Inference

• At inference, CoTBLIP rationales are used only for high-level visual inference and are placed before the top-tier question, which the original BLIP-2-T5xl answers.The authors note that rationales may be less effective for basic perception subquestions and text-only inference from supplied visual details.Source: §4 · p. 7

5.1 Model

• Evaluation spans text-only, non-LLM, LLM-based, and instruction-tuned VLMs; CoTBLIP appends generated CoT to frozen BLIP-2-T5xl for high-level visual inference.The comparison includes GPT-3.5-Turbo-0613, OFA, BLIP-2, InstructBLIP, LLaVA, and miniGPT-4.Source: §5.1 · p. 7

5.3 Further Analysis

• Existing VLMs often struggle with initial visual perception, while smaller drops on later subquestions suggest partially derived but insufficiently grounded inference and low Cb.The study presents examples with three subquestions and also discusses forward reasoning consistency in Appendix E.Source: Figure 5 · p. 8, §5.3 · pp. 8–9
• Increasing RLAIF training samples continuously improves reasoning performance and consistency, indicating that this stage could scale with web-scale image-caption data.The Rcot results are omitted because they are identical across the varying data amounts.Source: Figure 4 · p. 8, §5.3 · p. 8

5.2 Experimental Results

• CoTBLIP improves high-level visual inference and forward reasoning consistency by 4% each, though even BLIP-2-T5 remains below human performance.Its CoT rationales combine extracted visual details with potential inference.Source: §5.2 · p. 7, Table 2 · p. 8

5.3 Further Analysis

• Ro reaches 56.91 with SFT+RLAIF, versus 54.93 for baseline and 55.06 for SFT alone; both stages help, with RLAIF further improving results.RLAIF calibrates rationales for greater cohesiveness and substantiation; without SFT initialization, it produces only captions or trivial rationales and is not feasible.Source: §5.3 · pp. 7–8, Table 3 · p. 8

• The two-stage training framework combines supervised fine-tuning with learning from LLM feedback and improves VLM reasoning performance and consistency.Source: §6 · p. 9
• An LLM-Human-in-the-Loop pipeline is used to identify deficiencies in existing VLMs’ reasoning performance and consistency.Source: §6 · p. 9

Limitation

• CoTBLIP still has a significant gap from human annotators in reasoning performance and consistency.The paper says substantial effort is needed for VLMs to perform robust visual inference like humans.Source: Limitation · p. 9
• CoTBLIP generates general visual inference without considering instructions; future work should enable instruction-guided reasoning chains for high-level inference.Source: Limitation · p. 9

A Dataset Statistics

• CURE contains 1,622 evaluative instances averaging 2.91 reasoning chains; candidate inferences, subquestions, and answers average 7.05, 9.97, and 2.96 words, respectively.These elements are generated by LLMs from visual clues provided by human annotators; the clues center on human-oriented entities, activities, and occurrences.Source: A Dataset Statistics · p. 14
• CURE includes varied question types, with “What” questions dominating because Sherlock uses them to elicit holistic context comprehension, detailed narratives, and visual inference.Source: A Dataset Statistics · p. 14

B Related Work

• VLMs’ large-scale pretraining supports downstream performance; early models relied heavily on object detectors for image comprehension (Gan et al., 2022; Uppal et al., 2022; Li et al., 2019; Tan and Bansal, 2019).Source: B Related Work · p. 14
• Later VLM work sought to bypass resource-intensive detectors, streamline inference, incorporate more visual data, and add object-grounding tasks during pretraining (Dou et al., 2022; Huang et al., 2020; Kim et al., 2021; Huang et al., 2021; Xu et al., 2021; Yang et al., 2022; Yao et al., 2022; Li et al., 2021a; Radford et al., 2021; Jia et al., 2021).Source: B Related Work · p. 14
• Other efforts pursued unified VLM architectures and multimodal instruction tuning to support multiple tasks and user-friendly responses without task-specific adjustments (Wang et al., 2021, 2022b; Li et al., 2023a; Liu et al., 2023c; Zhu et al., 2023).Source: B Related Work · p. 14
• Chain-of-thought prompting elicits rationales before answers, has expanded across domains and complex tasks, and has also been used to improve reasoning performance (Wei et al., 2022; Poesia et al., 2023; Li et al., 2023b; Chen et al., 2022; Jin and Lu, 2023; Yao et al., 2023b,a; Saparov et al., 2023; Wang et al., 2024; Wang et al., 2022d).Prior studies report mixed results on how consistent LLM reasoning is (Wang et al., 2022a; Lanham et al., 2023; Madaan and Yazdanbakhsh, 2022; Saparov and He, 2023; Sahu et al., 2022).Source: B Related Work · p. 14

C Implementation Details of Evaluation

• Because the evaluated VLMs were not trained on grounded data, evaluation images receive annotated bounding boxes and prompts direct models to the delineated regions.For each top-tier question or subquestion, models choose the highest-probability option from A–F.Source: C Implementation Details of Evaluation · p. 14
• Figure 6 presents a word cloud of visual clues, while Figure 7 shows the question distribution.Source: Figure 6 · p. 15, Figure 7 · p. 15

Human Verification

• Three annotators review each LLM-generated example for reasoning and consistency relevance, using known failure modes and identifying additional exclusion reasons.They also label highly analogous cases to reduce redundant reasoning paths, updating their understanding of dataset distribution during annotation; any example classified as a failure by an annotator is excluded.Source: D Human Annotation · p. 15
• Annotators receive 100 samples to learn the dataset distribution, and abundant groups undergo validation to ensure high-quality examples proportional to group size.Source: D Human Annotation · p. 15

Dataset Evaluation

• A separate set of three annotators cross-validates the verified dataset and answers CURE’s visual-inference and CoT questions to establish human performance scores.Source: D Human Annotation · p. 15

E Forward Reasoning Consistency

• The analysis selects BLIP-2 and CoTBLIP as the highest-performing models, citing their significant improvements over text-only models.Source: E Forward Reasoning Consistency · p. 15
• Two Figure 8 examples illustrate BLIP-2's lack of forward reasoning consistency and CoTBLIP's potential to help.Source: E Forward Reasoning Consistency · p. 15
• CoTBLIP generates coherent rationales that progress from image-relevant visual elements to sophisticated inferences that affect predictions.In one example, it identifies motorcyclists gathered on a street before inferring they may belong to a community interested in such vehicles.Source: E Forward Reasoning Consistency · p. 16
• Incorporating rationales explicitly into the context enhances VLM reasoning consistency.Source: E Forward Reasoning Consistency · p. 16

F Prompts

• The prompt collection instructs LLMs to perform designated tasks in the implementation.Source: F Prompts · p. 16

F.1 Candidate Answers Generation for CoT Subquestions

• Candidate-answer prompts request five semantically similar but factually erroneous answers of 1–5 words for each CoT subquestion.Source: §F.1 · p. 16

F.2 Filtering

• Filtering prompts present a human-annotated scene and sequential CoT question-answer pairs, then ask whether the reasoning chain is coherent and consistent.The judgment format requests a direct yes-or-no answer.Source: §F.2 · p. 16

F.3 Baseline Evaluation

• Baseline evaluation prompts ask models to select among six answers, with variants focusing on a red-boxed region, an inference question, or a generated reasoning chain.Source: §F.3 · p. 16

F.4 CoT Reasoning Chains Generation

• CoT-generation prompts require short, cohesive step-by-step questions and answers that derive a high-level inference from a scene description.Chains should contain 2–4 questions with 1–3-word answers; questions should begin with visual information, use atomic nonredundant steps, avoid yes/no answers, and not mention the description.Source: §F.4 · pp. 16–17

F.5 Postprocessing of LLaVA Dataset

• LLaVA postprocessing prompts use scene descriptions and human-annotated question-answer pairs to generate concise step-by-step reasoning chains.Each step must contain fewer than 15 words, and the chain must have fewer than four steps.Source: §F.5 · p. 17

F.6 Feedback Generation

• Feedback prompts ask an LLM to choose the superior of two image inferences using fluency, image grounding, logical coherence, concision, and completeness as criteria.Source: §F.6 · pp. 17–18