PaperlayerView exact revision on arXiv

Source-linked AI summary

Bootstrapping Grounded Chain-of-Thought in Multimodal LLMs for Data-Efficient Model Adaptation

Jiaer Xia, Bingkui Tong, Yuhang Zang, Rui Shao, Kaiyang Zhou

arXiv:2507.02859v1cs.CV

TL;DR

MLLMs are difficult to adapt to specialized vision tasks because object-centric pre-training mismatches charts, tables, receipts, and reports, while distilled CoT can contain factual errors. The paper introduces GCoT, which bootstraps self-verified bounding boxes into CoT, and reports significantly better data-limited performance than fine-tuning and distillation. Its scope is constrained by reliance on external CoT models and difficulty grounding abstract visual elements.

  • Problem

    MLLMs’ object-centric pre-training mismatches specialized visual data, making adaptation difficult without large-scale retraining; distilled CoT also contains factual inaccuracies.

  • Method

    GCoT bootstraps an MLLM to generate, verify, and iteratively refine bounding boxes, then injects the verified grounding information into distilled CoT.

  • Results

    Across five specialized vision datasets, GCoT significantly outperforms zero-shot, fine-tuning, and distillation under data-limited regimes.

  • Takeaways & Limitations

    Self-verified grounding enables higher-quality reasoning data and supports data-efficient adaptation to specialized vision tasks.

  • Takeaways & Limitations

    The method is mainly suited to text- and number-rich images, relies on external models for CoT bootstrapping, and is difficult to scale up.

Abstract

from arXiv · show

Multimodal Large Language Models (MLLMs) have demonstrated remarkable capabilities in interpreting images using natural language. However, without using large-scale datasets for retraining, these models are difficult to adapt to specialized vision tasks, e.g., chart understanding. This problem is caused by a mismatch between pre-training and downstream datasets: pre-training datasets primarily concentrate on scenes and objects but contain limited information about specialized, non-object images, such as charts and tables. In this paper, we share an interesting finding that training an MLLM with Chain-of-Thought (CoT) reasoning data can facilitate model adaptation in specialized vision tasks, especially under data-limited regimes. However, we identify a critical issue within CoT data distilled from pre-trained MLLMs, i.e., the data often contains multiple factual errors in the reasoning steps. To address the problem, we propose Grounded Chain-of-Thought (GCoT), a simple bootstrapping-based approach that aims to inject grounding information (i.e., bounding boxes) into CoT data, essentially making the reasoning steps more faithful to input images. We evaluate our approach on five specialized vision tasks, which cover a variety of visual formats including charts, tables, receipts, and reports. The results demonstrate that under data-limited regimes our approach significantly improves upon fine-tuning and distillation.

1. Introduction

MLLMs struggle with specialized vision tasks because their object-centric pre-training mismatches chart-, table-, receipt-, and report-focused data. The paper proposes grounded CoT and finds it improves data-limited adaptation by making reasoning more faithful to images.

  • Problem: Distilled CoT often contains factual errors, even when its final answer is correct.These errors motivate adding image-grounded evidence rather than relying on reasoning text alone.
  • Motivation: Object-centric pre-training leaves MLLMs weak on specialized, non-object images such as charts and tables.Although MLLMs interpret images in natural language, adapting them to specialized tasks typically requires large task-specific datasets.
  • Motivation: CoT data can facilitate adaptation from generic to specialized vision tasks, particularly when labeled training data are limited.The benchmark spans charts, tables, receipts, and reports and uses distilled CoT to fine-tune models under few-shot settings.
  • Method: GCoT injects self-verified bounding boxes into CoT so reasoning steps are more faithful to the input image.The approach bootstraps an MLLM to generate and refine grounding labels through self-verification.
  • Evaluation: Across five specialized vision datasets, GCoT significantly outperforms zero-shot, fine-tuning, and distillation under data-limited regimes.The evaluation covers charts, tables, receipts, and reports, with accurate grounding helping identify image content such as item prices.

2. Related Work

Prior work establishes MLLMs, chart-specific adaptation, and the value of CoT reasoning, while leaving vision-based CoT verification open. This paper addresses that gap by verifying grounded bounding boxes within a bootstrapped CoT pipeline.

  • Multimodal LLMs: MLLMs combine a vision encoder with an LLM, projecting image features into the text space for multimodal understanding.LLaVA and related systems use large image-caption datasets to train the connection between vision and language components.
  • Grounded CoT: GCoT’s bootstrapping loop generates candidate boxes, checks cropped regions against target content, and retains consistent boxes for iterative refinement.Correct boxes are collected to fine-tune localization, progressively improving the model’s grounding ability.
  • Chart Understanding: Chart-focused datasets adapt MLLMs to chart interpretation, question answering, and reasoning because pre-training provides limited exposure to charts.Prior datasets include ChartLLaMA’s 160,000 chart samples and ChartAssistant’s 39 million chart-text annotations.
  • Chain-of-Thought Reasoning: CoT improves complex reasoning by decomposing problems into intermediate steps and can also align models with desired behaviors.These benefits motivate using distilled CoT for adapting multimodal models.
  • Chain-of-Thought Reasoning: Verification methods are established for textual CoT, but extending CoT verification to vision remains an open question.The paper fills this gap by combining CoT with grounded bounding boxes that a detection-based MLLM can verify.

3. Methodology

GCoT bootstraps grounding information into distilled Chain-of-Thought data so reasoning steps become more faithful to images before fine-tuning. The method combines visual-grounding pre-training, iterative box generation and self-verification, and grounded-CoT fine-tuning.

  • Grounded Chain-of-Thought: GCoT injects bounding boxes into distilled CoT to reduce factual inaccuracies in reasoning and provide verifiable visual evidence.The augmented reasoning is intended to make model outputs more faithful to the input image.
  • Pre-training for Visual Grounding: The method first pre-trains a base MLLM on visual-grounding data so it can localize text or objects mentioned in prompts.The implementation can use a model such as VisCoT-7B, pre-trained on ten visual-grounding datasets spanning text recognition, VQA, infographic understanding, and relation reasoning.
  • Generating Grounded Chain-of-Thought: For each distilled CoT, meaningful nouns and numerical terms become targets for “Where is the <target>?” sub-questions that request corresponding bounding boxes.The targets are extracted from the generated reasoning process, and each sub-question corresponds to one target.
  • Generating Grounded Chain-of-Thought: The bootstrapping loop crops each candidate box, asks the MLLM to identify the enclosed content, and retains boxes whose contents consistently match the target.Mismatched proposals are automatically filtered, producing verified boxes for subsequent iterations.
  • Generating Grounded Chain-of-Thought: Verified boxes are used to fine-tune the MLLM, progressively improving localization and enabling it to identify more accurate boxes in later cycles.The process ultimately develops grounding ability before the final grounded-CoT construction.
  • Grounded Chain-of-Thought: After a fixed number of bootstrap iterations, correct box coordinates are appended after their corresponding CoT targets, and the resulting GCoT is used for fine-tuning.Self-verification also checks reasoning and final answers, filtering noisy examples through box-content consistency.

4. Experiments

Experiments evaluate GCoT on five specialized vision datasets under data-limited settings, comparing it with zero-shot, fine-tuning, and CoT distillation. GCoT consistently leads the baselines, while ablations show benefits from augmentation, box verification, iterative bootstrapping, and varied distillation sources.

  • Main Results: GCoT consistently outperforms fine-tuning and distillation across all five datasets and training sample sizes.The main comparison uses the same backbone and LoRA-based training setup; GCoT also outperforms the baselines on the challenging TAT-QA dataset.
  • Main Results: With 8 training samples, GCoT exceeds zero-shot performance by around 2% on average, and its advantage over training-based baselines grows with more samples.Performance improves across methods as sample size increases, but fine-tuning remains relatively unchanged on ChartQA and TabMWP.
  • Main Results: Distillation generally outperforms fine-tuning because distilled CoT provides richer training information than simple question-answer pairs.The advantage is clearest as sample size increases, although low-shot exceptions occur on datasets such as TabMWP and DVQA.
  • Ablation Studies and Analyses: Removing augmentation lowers performance by 1.63–2.36 percentage points, while removing box verification causes larger drops of 5.94–10.36 points.The ablation on TabMWP indicates that generating multiple annotations and verifying bounding boxes are both important for filtering noisy CoT data.
  • Ablation Studies and Analyses: Increasing bootstrapping iterations steadily raises the number of true bounding boxes, with larger gains on ChartQA and TabMWP than on TAT-QA and DVQA.The iterative process is presented as a way to obtain more accurate grounding without manual labeling.
  • Ablation Studies and Analyses: Across distillation sources, GCoT improves over normal distillation by +1.54 to +5.62 percentage points and reduces average-result variance from 2.37 to 0.91.The strongest reported gains occur with Claude 3.5 (+5.62) and Qwen2-VL (+4.95), suggesting the approach is not tied to one source model.

5. Conclusion

The study finds that MLLMs can adapt to specialized vision tasks in data-limited settings, while grounded CoT addresses factual errors in distilled reasoning data. It evaluates this approach across five specialized tasks and identifies efficient adaptation as important for broader deployment.

  • The work is presented as the first study of MLLMs in data-limited settings for specialized vision tasks.
  • Across five specialized vision tasks, training with grounded CoT improves model adaptation under limited data and supports efficient MLLM deployment.The study describes data-efficient adaptation as important for democratization and large-scale deployment.
  • CoT data can facilitate adaptation from generic to specialized vision tasks, but distilled reasoning often contains multiple factual errors.
  • The study introduces GCoT, which injects self-verified grounding information into CoT to make reasoning more faithful to input images.

6. Limitations and Future Work

GCoT is constrained by its reliance on bounding-box examination and external models for CoT generation, limiting applicable image formats and scalability.

  • Bounding-box verification is difficult for abstract objects such as lines and icons, restricting GCoT mainly to images rich in text and numbers.
  • Reliance on existing external models to provide CoT data limits training and makes the method difficult to scale up.The paper suggests reinforcement learning as a possible direction for improving CoT and grounding abilities.

A.1. Evaluation of Generalization Ability

Cross-model evaluation shows that clearer, higher-quality GCoT training data improves generalization beyond the training dataset. A model trained on ChartQA generalizes improvements to other datasets, with GCoT producing greater enhancements.

  • GCoT produces greater cross-dataset generalization enhancements than the comparison training approaches.The model is trained on ChartQA and evaluated across other datasets.
  • Training on detailed CoT structures, relationships, and logical patterns supports clearer alignment than training only on simple answer annotations.
  • GCoT improves the quality of distilled reasoning data by eliminating noisy information before training.

A.2. Evaluation of Chain-of-Thought Accuracy

Manual examination of 64 ChartQA CoT samples supports the use of box verification for improving reasoning-data accuracy. The verification process reduces errors in distilled CoT and yields more accurate training samples.

  • Box-based verification substantially reduces errors in distilled CoT, producing more accurate training data.The manual examination covered 64 ChartQA CoT samples and supports the ablation findings.
  • The analysis was conducted on 64 manually examined ChartQA CoT samples to validate the process-verification approach.

A.3. Visualization

This section presents illustrative examples of self-verified GCoT across different distillation sources and datasets, highlighting its improvements.

  • The examples compare self-verified GCoT across multiple distillation-data sources and datasets.

A.3.1. Different source of Distillation CoT

GCoT reduces inaccuracies in distilled reasoning data while standardizing format and generating diverse, reliable CoT datasets across chart-understanding tasks.

  • Different source of Distillation CoT: GCoT markedly reduces reasoning inaccuracies compared with distilled CoT, producing higher-quality training data.
  • Different source of Distillation CoT: GCoT generates diverse CoT data while maintaining high accuracy across ChartQA, TAT-QA, and SROIE examples.
  • Different source of Distillation CoT: GCoT standardizes the data format, expands limited datasets, and enhances training efficiency for diverse tasks.
Loading 2507.02859v1…