Source-linked AI summary
Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoning
Hao Shao, Shengju Qian, Han Xiao, Guanglu Song, Zhuofan Zong, Letian Wang, Yu Liu, Hongsheng Li
TL;DR
MLLMs can struggle with interpretability and complex visual inputs, particularly when relevant information lies in small regions. VisCoT introduces a 438k-pair grounded dataset, a multi-turn pipeline for dynamically focusing on visual regions, and a benchmark for local-region identification; experiments validate the framework’s effectiveness.
Problem
MLLMs often lack interpretability and struggle with complex visual inputs, while existing datasets lack intermediate visual chain-of-thought supervision.
Method
VisCoT combines a 438k-pair dataset with bounding-box annotations, detailed reasoning for about 98k pairs, a multi-turn dynamic visual pipeline, and a local-region benchmark.
Results
Extensive experiments validate the framework’s effectiveness, including improved token efficiency and performance sensitivity to bounding-box selection precision.
Takeaways & Limitations
VisCoT provides interpretable reasoning stages and resources for studying visual chain-of-thought reasoning in MLLMs.
Abstract
from arXiv · showhide
Multi-Modal Large Language Models (MLLMs) have demonstrated impressive performance in various VQA tasks. However, they often lack interpretability and struggle with complex visual inputs, especially when the resolution of the input image is high or when the interested region that could provide key information for answering the question is small. To address these challenges, we collect and introduce the large-scale Visual CoT dataset comprising 438k question-answer pairs, annotated with intermediate bounding boxes highlighting key regions essential for answering the questions. Additionally, about 98k pairs of them are annotated with detailed reasoning steps. Importantly, we propose a multi-turn processing pipeline that dynamically focuses on visual inputs and provides interpretable thoughts. We also introduce the related benchmark to evaluate the MLLMs in scenarios requiring specific local region identification. Extensive experiments demonstrate the effectiveness of our framework and shed light on better inference strategies. The Visual CoT dataset, benchmark, and pre-trained models are available on https://hao-shao.com/projects/viscot.html to support further research in this area.
1 Introduction
Existing MLLMs often process visual inputs as black boxes with static, fixed-grain image contexts, leaving visual chain-of-thought reasoning and interpretability underexplored. The paper addresses these gaps with a grounded dataset, a dynamic multi-turn pipeline, and a benchmark for local-region reasoning.
- Existing MLLMs commonly use decoder-only black-box designs that can produce inaccurate information or hallucinations and hinder interpretability.
- Visual chain-of-thought reasoning and multi-turn in-context capabilities remain insufficiently explored in multimodal visual understanding.
- 438k visual question-answer pairs receive intermediate bounding boxes identifying key image regions, while about 98k additionally include detailed reasoning steps.The annotations train models to identify a final relevant region and generate an answer through logical, step-by-step processing.
- The proposed multi-turn pipeline dynamically focuses on visual inputs and provides intermediate interpretable thoughts.
- The visual chain-of-thought benchmark evaluates MLLMs in scenarios requiring specific local-region focus or object identification.
2 Related Works
The related work situates VisCoT among multimodal language models and chain-of-thought methods, while motivating datasets that ground reasoning in specific image regions. Figure 1 illustrates the dataset’s coverage across five visual domains and its bounding-box annotations.
- MLLMs integrate vision and language, evolving from dispatching vision experts toward models that directly process multimodal inputs.
- Figure 1 covers chart, text/document, general VQA, fine-grained understanding, and relation reasoning domains, with red boxes marking critical regions.
- Chain-of-thought prompting enables language models to generate intermediate reasoning steps, building on in-context learning and demonstrations.
3 Visual CoT Dataset
The Visual CoT dataset targets region-focused visual reasoning by pairing questions and answers with intermediate bounding boxes, spanning five domains and diverse source datasets. Its key regions are often small, motivating precise localization for efficient and accurate processing.
- Dataset Overview: About 98k samples include detailed reasoning steps, while the red bounding box identifies the important image region for answering.These examples combine step-by-step reasoning with localized visual evidence.
- Dataset Overview: 438k data items pair each question and answer with an intermediate visual bounding box, with some items also containing detailed reasoning steps.The dataset spans five distinct domains.
- Data Composition: The dataset integrates text/doc, fine-grained understanding, charts, general VQA, and relation reasoning to cover varied analytical tasks.It reuses twelve source datasets across five domains and combines linguistic and visual annotation.
- Data Generation: Source-specific construction uses OCR regions, object bounding boxes, bird-part annotations, and relation-relevant boxes as visual CoT supervision.These procedures support text recognition, fine-grained discrimination, chart answering, general VQA, and spatial reasoning.
- Dataset Analysis: The average CoT bounding box is 247.82 pixels and covers about 13.2% of image area, with many key regions occupying only a small portion of images.The reported region size aligns with common vision-encoder input resolutions of 224–336 pixels, whereas original images often require down-sampling.
4 Enhancing MLLMs with Chain-of-Thought Capabilities
VisCoT augments standard MLLMs with a multi-stage visual chain-of-thought process that identifies a question-relevant region, processes it, and combines localized and global information. The framework supports optional CoT inference and uses standard models without specialized modifications.
- VisCoT Pipeline: VisCoT prompts the MLLM to identify the most informative image region and generate its bounding box before localized visual processing.During training, the ground-truth bounding box is used to extract localized information in subsequent steps.
- Visual Sampler: The visual sampler crops a square region sized to cover the predicted box and the vision encoder’s receptive-field requirement.At inference, it adjusts the crop center when boundaries would otherwise be exceeded.
- Inference: Users can run VisCoT with a standard image-question input or activate the visual CoT process by appending the additional prompt.The framework is adaptable to annotated and non-annotated data formats.
- VisCoT Pipeline: The pipeline extracts visual tokens, pinpoints a key region, processes localized information, and integrates localized and overall images for the final answer.This sequence is illustrated as a multi-stage visual reasoning process.
- Model Training: VisCoT is trained in two stages: initial caption training with vision and language weights frozen, followed by training with all weights trainable.The first stage follows the LLaVA-1.5 training setup.
5 Experiments
Experiments evaluate VisCoT on its visual CoT benchmark, compare inference and sampling strategies, and report improved performance, token efficiency, and qualitative region-focused reasoning.
- 5.1 Visual CoT Benchmark: VisCoT is evaluated on a benchmark emphasizing questions that require focusing on specific image regions, using data from 12 source datasets.The benchmark includes zero-shot evaluation on test splits from SROIE, DUDE, and Visual7W.
- 5.2 Performance Evaluation: VisCoT significantly improves doc/text-related tasks and high-resolution image processing, including an 8× performance gain over direct inference on SROIE.The comparison includes VisCoT, LLaVA-1.5, and a standard pipeline without chain-of-thought reasoning.
- 5.3 Ablation Study: Ground-truth bounding-box selection gives the highest performance and represents an upper bound for the model.The ablation compares annotated boxes with model predictions and alternative selection strategies.
- 5.3 Ablation Study: At 224 resolution, VisCoT surpasses the standard pipeline at 448 resolution while using half the visual tokens.The pipeline uses double the visual tokens for answer generation, motivating evaluation across resolutions 224, 336, and 448.
- 5.3 Ablation Study: Random box selection performs similarly to removing CoT, while center selection performs better than random selection.These results indicate that arbitrary or inaccurate box selection has limited benefit and that prediction precision affects overall performance.
- 5.3 Ablation Study: Expanded and centered cropping provide more image context, which the authors suggest can mitigate detection inaccuracies.The visual sampler ablation compares these cropping designs with the default sampler.
- 5.4 Qualitative Results: Qualitative results show VisCoT identifying critical regions and combining original-image and zoomed-in contexts when generating answers.The section compares VisCoT with ground-truth-box and without-CoT configurations.
6 Conclusion
The conclusion presents VisCoT as a visual chain-of-thought framework addressing interpretability and dynamic visual-input processing in MLLMs. It combines a large annotated dataset, a multi-turn pipeline, and a benchmark, with experiments supporting its effectiveness.
- Contributions: VisCoT addresses interpretability and dynamic visual-input processing in multi-modal large language models.The conclusion frames these as critical gaps in existing MLLMs.
- Contributions: The visual CoT dataset contains 438k annotated question-answer pairs for detailed visual analysis.The dataset is part of the framework introduced in the paper.
- Contributions: VisCoT’s multi-turn processing pipeline dynamically focuses on and interprets visual data through more interpretable reasoning stages.The conclusion describes this as mirroring human cognition.
- Contributions: The visual CoT benchmark evaluates MLLMs’ ability to focus on specific image areas.The conclusion identifies the benchmark as an advance for evaluation in this setting.
- Conclusion: Extensive experiments validate the framework’s effectiveness and provide a starting point for further exploration of visual chain-of-thought.The conclusion characterizes the work as a promising starting point for future research.
Checklist
The checklist records reported compliance, implementation details, and released artifacts, including the model configuration and two-stage training setup.
- Checklist: The authors report that the paper describes its limitations, ethics considerations, and asset licensing or consent information.These items are marked as addressed in the checklist, with limitations referenced in Appendix F and ethics details in Appendix G or H.
- Checklist: The released artifacts include code, training data, benchmarks, checkpoints, and Apache-2.0-licensed model weights.The checklist provides GitHub and Hugging Face locations for these resources.
- Checklist: The model uses CLIP ViT-L/14 as its vision encoder, Vicuna-7/13B as its language model, and a linear projection into the language embedding space.The projected visual features form visual tokens compatible with the language model.
- Checklist: Training comprises one pre-training epoch followed by one fine-tuning epoch on the visual CoT dataset, using batch size 128 in both stages.The reported learning rates are 2e-3 and 2e-5, respectively, with Adam, zero weight decay, and cosine scheduling.
B.3 Training data details
The training data combines existing vision-language resources with Visual CoT data, while the supplementary material documents dataset composition and bounding-box detection evaluation.
- B.3 Training data details: The reorganized training set combines LLaVA second-stage data, Shikra second-stage data, and the authors’ visual CoT data.Shikra contributes positional-annotation datasets such as RefCOCO and visual genome grounding captions.
- B.3 Training data details: Table 6 provides an overview of the training dataset.The supplied passage identifies the table’s purpose but does not provide its entries.
D.1 Performance evaluation
Evaluation covers standard multimodal benchmarks, visual grounding, and visual CoT behavior, with results indicating broad performance and stronger gains for smaller queried regions.
- D.1 Performance evaluation: VisCoT achieves the best performance on most of eight listed benchmarks and ranks second on the others under direct-response evaluation.The comparison includes ScienceQA, TextVQA, MME, POPE, and MMBench variants.
- D.1 Performance evaluation: The model achieves comparative results across standard MLLM benchmarks and performs better with lower image resolution and fewer visual tokens when visual CoT is enabled.The authors interpret this as evidence of improved efficiency and effectiveness in visual understanding.
- D.1 Performance evaluation: VisCoT outperforms previous state-of-the-art methods on RefCOCO, RefCOCO+, and RefCOCOg referring-expression comprehension benchmarks.The reported advantage persists with a 7B language model and 224-pixel resolution, compared with methods using larger models or higher resolutions.
- D.1 Performance evaluation: Visual CoT improves the baseline over answering without CoT, with greater improvement when the corresponding bounding box is relatively smaller.Evaluation datasets are divided into five equal parts by relative bounding-box size.
- D.1 Performance evaluation: Figure 7 illustrates a failure case in which an incorrectly predicted CoT region leads to a wrong answer.Predicted boxes are shown in red and ground-truth boxes in blue.
E.4 Evaluation for the visual CoT benchmark using the ChatGPT
The evaluation procedure scores a model answer against a standard answer on a 0-to-1 scale, with higher scores assigned to closer matches.
- E.4 Evaluation for the visual CoT benchmark using the ChatGPT: The evaluator outputs a score between 0 and 1 by comparing the model’s answer with the standard answer for the given question.The required format is “score: <score>”.
F Limitations
The paper identifies limitations in both VisCoT’s region selection and its GPT-4-based data pipeline. Complex images or questions can mislead the model, while generated annotations may contain inaccuracies or biases.
- VisCoT may struggle to identify the most relevant region when images contain extensive information or questions are particularly complex.This difficulty can mislead the model and produce incorrect responses.
- GPT-4-generated data may be inaccurate and spread misinformation, despite a post-processing filtering script intended to improve content quality.
- Because the GPT-4 training data are inaccessible, generated instructional data may reflect biases that reinforce social or cultural inequalities.
G Potential negative societal impacts
The paper acknowledges societal risks associated with Visual CoT and MLLMs, including privacy, bias, misinformation, employment, accountability, and consent concerns. It limits the released resources to research and reproducibility rather than deployed use.
- Visual CoT and MLLMs may contribute to privacy invasion, bias perpetuation, misinformation, job displacement, and accountability or consent concerns.
- Dataset examples use red bounding boxes to mark critical image regions containing information necessary or related to answering questions.
- The dataset, code, and checkpoints are intended for future visual-language research and reproducing the reported experiments.
- Deployed use cases, whether commercial or otherwise, are outside the intended scope of the models and evaluations.
- Visualization results distinguish model-generated bounding boxes in red from ground-truth bounding boxes in blue.