PaperlayerView exact revision on arXiv

Source-linked AI summary

CogCoM: A Visual Language Model with Chain-of-Manipulations Reasoning

Ji Qi, Ming Ding, Weihan Wang, Yushi Bai, Qingsong Lv, Wenyi Hong, Bin Xu, Lei Hou, Juanzi Li, Yuxiao Dong, Jie Tang

arXiv:2402.04236v3cs.CVcs.CL

TL;DR

Conclusive-response tuning can leave VLMs without the intermediate visual reasoning needed for subtle visual questions. CogCoM trains models to produce step-by-step manipulation-based evidence, and reports improvements across visual benchmarks while preserving interpretability.

  • Problem

    Conclusive-response tuning can cause VLMs to overlook intermediate visual reasoning, contributing to failures on subtle visual tasks and unfaithful answers.

    Source: §1 · p. 2

  • Method

    CogCoM uses Chain of Manipulations, in which a VLM solves problems step by step by producing visual manipulations and their results as evidence.

    Source: §1 · p. 3

  • Results

    Across nine benchmarks in four categories, CogCoM reports up to 9.0 accuracy improvement on detailed VQA benchmarks and 1.09 on grounding benchmarks, plus superior general multimodal performance.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The authors report that the 17B CogCoM model produces informative reasoning content while achieving superior performance across nine public benchmarks.

    Source: §7 · p. 10

    Solving-step diversity remains insufficient, and coarse grounding boxes or OCR errors can produce negative paths.

    Source: C LIMITATION AND IMPACT · p. 15

Abstract

from arXiv · show

Vision-Language Models (VLMs) have demonstrated their broad effectiveness thanks to extensive training in aligning visual instructions to responses. However, such training of conclusive alignment leads models to ignore essential visual reasoning, further resulting in failures in meticulous visual problems and unfaithful responses. Drawing inspiration from human cognition in solving visual problems (e.g., marking, zoom in), this paper introduces Chain of Manipulations, a mechanism that enables VLMs to solve problems step-by-step with evidence. After training, models can solve various visual problems by eliciting intrinsic manipulations (e.g., grounding, zoom in) with results (e.g., boxes, image) actively without involving external tools, while also allowing users to trace error causes. We study the roadmap to implement this mechanism, including (1) a flexible design of manipulations upon extensive analysis, (2) an efficient automated data generation pipeline, (3) a compatible VLM architecture capable of multi-turn multi-image, and (4) a model training process for versatile capabilities. With the design, we also manually annotate 6K high-quality samples for the challenging graphical mathematical problems. Our trained model, \textbf{CogCoM}, equipped with this mechanism with 17B parameters achieves state-of-the-art performance across 9 benchmarks from 4 categories, demonstrating the effectiveness while preserving the interpretability. Our code, model weights, and collected data are publicly available at https://github.com/THUDM/CogCoM.

1 INTRODUCTION

Conclusive-response tuning can leave VLMs short of the intermediate visual reasoning needed for subtle details, contributing to unfaithful answers. CogCoM introduces step-by-step reasoning with visual manipulations and evidence, and reports gains across visual benchmarks while preserving interpretability.

1 INTRODUCTION

  • Conclusive-response tuning can make VLMs overlook intermediate visual reasoning, leading to failures on subtle details, unfaithful responses, and hallucinations.The paper illustrates this with a model answering “NO SMOKING” for text on a pillar.Source: §1 · p. 2
  • The approach addresses three implementation obstacles: flexible manipulation definitions, abundant training data, and a compatible multi-turn, multi-image structure.The paper draws inspiration from human behaviors such as marking, locating references, and zooming in on image regions.Source: §1 · p. 3
  • CogCoM’s Chain of Manipulations (CoM) lets VLMs solve problems step by step, generating visual manipulations and their results as evidence.The paper studies manipulations design, data collection, architecture, and training to develop this mechanism.Source: §1 · p. 3
  • The work collects 70K CoM training samples and manually annotates 7K graphical-math samples, while training CogCoM for chat, captioning, grounding, and reasoning.The 7K samples each include a CoM reasoning process.Source: §1 · p. 3
  • The paper reports up to 9.0 accuracy improvement on detailed VQA benchmarks and 1.09 on grounding benchmarks, plus superior general multimodal performance.Experiments cover nine benchmarks across four categories; the authors say the outputs preserve interpretability.Source: §1 · p. 3

2 TERMINOLOGY

The paper identifies six atomic image manipulations from pilot analysis and defines a CoM format that pairs each reasoning step with a manipulation and its result. The format supports evidence-bearing samples while remaining compatible with linguistic reasoning.

2 TERMINOLOGY

  • Pilot analysis prompted GPT-4 to propose image-assisted solving steps for 170K TextVQA questions, using four manually written demonstrations for stability.The researchers extracted verb phrases referring to actions and analyzed their frequencies.Source: §2 · p. 3
  • Analysis found that most proposed actions map to six fundamental manipulations, with their distribution shown across 465 generated actions in Figure 3.The mapping includes OCR, Grounding, CropZoomIn, Counting, Calculate, and Line.Source: §2 · p. 3, Figure 3 · p. 3
  • The six atomic manipulations are OCR, Grounding, CropZoomIn, Counting, Calculate, and Line.Their inputs and outputs are specified in the paper’s formal definition.Source: §2 · p. 4
  • The model may create new manipulations at inference time, and the paper reports that more complex goals can be derived from the six fundamentals.The predefined set may be developed during pre-training or learned through fine-tuning by imitating human behaviors.Source: §2 · p. 4
  • A CoM reasoning chain is a sequence of steps, each pairing a manipulation with its corresponding result, alongside free-form textual descriptions.The format explicitly represents symbolic manipulation execution while remaining compatible with linguistic reasoning steps.Source: §2 · p. 4

3 DATA COLLECTION

The data collection approach combines automated CoM sample generation with human annotation for graphical mathematics. Samples are structured as manipulation-assisted reasoning chains and adapted into multi-turn VQA training examples.

3 DATA COLLECTION

  • Figure 4’s pipeline has an LLM propose solving steps, visual foundational models fill in manipulation results, and a final tree traversal find positive paths to the answer.The pipeline starts from an original VQA sample and produces standard CoM samples.Source: Figure 4 · p. 4

3.1 AUTOMATED DATA GENERATION

  • The automated pipeline generated 70K CoM samples from TextVQA, ST-VQA, and TDIUC by traversing manipulation-result trees and retaining paths ending in the golden answer.GroundingDINO and PaddleOCR supply visual results; depth-first search finds positive paths that the authors describe as mostly error-free.Source: §3.1 · p. 5

3.2 HUMAN ANNOTATION

  • Ten human experts annotated 7K high-quality CoM math samples on MathVista and ChartQA, covering geometric and chart math problems.Each expert annotated linguistic solving steps, manipulation use, and manipulation results on images.Source: §3.2 · p. 5
  • CoM samples are converted into multi-turn VQA examples, with intermediate steps associated with successive images and prompts that ask the model to answer using conversation history.This transformation makes the samples compatible with existing VLM training corpora.Source: §3.2 · p. 5

4 MODEL TRAINING

CogCoM combines a memory-based multi-turn, multi-image VLM architecture with two training stages for general multimodal task-solving and visual reasoning.

4.1 ARCHITECTURE

  • Figure 5 presents a compatible multi-turn, multi-image architecture; CogCoM uses CogVLM's visual encoder, MLP adapter, LLM backbone, and visual expert module.The visual expert module adds 6.5B parameters for modality fusion.Source: §4.1 · p. 5, Figure 5 · p. 6
  • Across turns, each LLM layer retains accumulated KV memories; subsequent images are cropped from the previous image and bicubically amplified.Concatenated representations are truncated when their sequence length exceeds a predefined threshold.Source: §4.1 · p. 6

4.2 TRAINING

  • Pre-training proceeds through 1.5B image-text pairs and then 40M grounded image-question-answer triples, using next-token prediction for the visual experts.The two phases use 120,000 and 60,000 iterations, respectively.Source: §4.2 · p. 6
  • Alignment fuses CoM data with instruction-following, text-recognition, and detailed-captioning corpora into 570K samples, with multi-turn CoM answers and optional manipulation prompts.Training uses 14,000 iterations; the visual experts are trained with next-token prediction.Source: §4.2 · p. 6

5 EXPERIMENT

The experiments evaluate CogCoM on detailed VQA, visual grounding, general multimodal ability, hallucination, and reasoning accuracy and time, using VQA and grounding model versions alongside qualitative analysis. Across these settings, the reported results include gains over generalist baselines and strong grounding performance.

5.1.1 GQA, TEXTVQA, ST-VQA, TALLYVQA

  • CogCoM achieves state-of-the-art performance among generalist models on detailed VQA, with up to 5.97 and 9.0 percentage-point gains over the baseline on complex-reasoning and detailed-recognition benchmarks.It is comparable with large-scale specialist SOTAs on GQA and TextVQA.Source: Table 1 · p. 7, §5.1.1 · p. 8

5.1.2 EXPERIMENTS FOR REASONING ACCURACY AND TIME COMPLEXITY

  • 48.41 accuracy at 2K steps and 55.59 at 8K show CoM-test performance from training on 70K automated CoM examples alone.CoM-test uses TextVQA test-set reasoning chains and a keypoints-aware metric; the reasoning score combines a manipulation score and paragraph BLEU.Source: Figure 6 · p. 8, §5.1.2 · p. 8
  • On 218 MM-Vet questions, CogCoM produces information-intensive reasoning without infeasible time overhead; it outputs 262.9 informative tokens in approximately 9 seconds.The analysis groups samples into eight intervals by time expenditure and averages time and token counts.Source: Figure 6 · p. 8, §5.1.2 · p. 8

5.2 EXPERIMENTS ON VISUAL GROUNDING

  • CogCoM achieves the best performance on 6 of 8 visual-grounding subsets and is on par with specialist SOTAs.It also surpasses ONE-PEACE on RefCOCO+ test-A.Source: Table 2 · p. 9, §5.2 · p. 9

5.3 EXPERIMENTS ON GENERAL MULTIMODAL EVALUATION AND HALLUCINATION EXAMINATION

  • CogCoM improves MM-Vet performance by 0.6 points and achieves superior POPE performance consistent with the baseline model.The authors report lower hallucination alongside maintained general multimodal effectiveness.Source: Table 3 · p. 9, §5.3 · p. 9

5.4 ABLATION EXPERIMENT FOR TRAINING W/WO COM DATA

  • Removing the 70K CoM training data while keeping other settings unchanged shows that CoM data benefits detailed recognition, multimodal reasoning, and mathematical-capability benchmarks.The ablation uses CogCoM-chat and retains public VQA data and 500K training examples from MultiInstruct, LLaVAR, and ShareGPT4V.Source: Table 4 · p. 10, §5.4 · p. 10

6 RELATED WORKS

Related work covers VLM pretraining and instruction tuning, then groups visual-reasoning research into grounding, artificial-scene understanding, hallucination, and detail recognition.

6 RELATED WORKS

7 CONCLUSION

The conclusion presents CoM as a step-by-step visual reasoning mechanism and reports a 17B model with informative reasoning across diverse multimodal problems. The discussion distinguishes CoM from related systems and identifies efficient backtracking as future work.

7 CONCLUSION

  • CoM combines a flexible data structure, data-generation framework, memory-based compatible architecture, and training process; the paper also annotates 7K graphical-math samples.Experiments on nine public benchmarks support the reported performance across diverse multimodal problems.Source: §7 · p. 10

A DISCUSSION WITH CLOSELY RELATED WORKS

  • Unlike ViperGPT’s training-free framework combining external VLMs with a code LLM, CoM trains one end-to-end VLM to reason and manipulate images.The discussion contrasts CoM with V*, DualFocus, VPD, VisProg, Ferret-v2, and video reasoning work on their architectures, training, or visual-reasoning approaches.Source: A DISCUSSION WITH CLOSELY RELATED WORKS · p. 15

B DISCUSSION FOR FUTURE WORKS

  • Backtracking may help form reliable reasoning paths, but its high time complexity makes improving efficiency without sacrificing performance a future-work challenge.The paper identifies efficient backtracking as a crucial direction.Source: B DISCUSSION FOR FUTURE WORKS · p. 15

C LIMITATION AND IMPACT

The paper identifies limited reasoning-step diversity, visual-tool inaccuracies, and slow re-input of manipulated images as limitations, while outlining potential benefits of its mechanism, data framework, and architecture.

C LIMITATION AND IMPACT

  • Limited diversity in solving steps and inaccurate visual tools, including coarse boxes and OCR errors on slanted text, lead to negative paths.The authors suggest diversified prompts and enhanced visual tools as possible improvements.Source: C LIMITATION AND IMPACT · p. 15
  • Re-inputting manipulated images with fixed prompts can slow the process; the authors suggest direct physical manipulations in vector-space calculations as an optimization.Source: C LIMITATION AND IMPACT · p. 15
  • The paper expects its reasoning mechanism, data-production framework, and memory-based architecture to benefit complex visual problem solving, data-driven training scenarios, and multi-turn long contexts.Source: C LIMITATION AND IMPACT · p. 15

D ADDITIONAL ANALYSIS

The additional analysis reports inference-turn statistics and data-quality controls, examines where reasoning-path construction succeeds, describes the data-generation pipeline, and defines a metric for evaluating CoM chains.

D.1 THE NUMBER OF INFERENCE TURNS AND THE QUALITY CONTROL

  • The TextVQA/ST-VQA training data averages 1.42 multi-image turns, with a maximum of 7; TextVQA test outputs average 1.54 turns.Training capped turns at 4 to prevent OOM, and some images require no manipulation.Source: §D.1 · p. 16

E DETAILS OF DATA PRODUCTION

  • The synthesis pipeline traverses intermediate reasoning paths to retain positive chains reaching the golden answer; manual verification found most paths correct.The 70K training data excludes negative paths that fail to reach the golden answer, though overly large grounding boxes can still cause errors.Source: §D.1 · p. 16, E DETAILS OF DATA PRODUCTION · p. 17

D.2 THE DISTRIBUTION OF THE SUCCESSFUL AND UNSUCCESSFUL PATHS

  • Most image classes have approximately consistent successful and unsuccessful path proportions, while common prominent objects have higher construction success than smaller, less common objects.The figure shows the top 200 classes with positive frequencies log10-transformed; the authors attribute lower success for smaller objects to visual-tool accuracy limits.Source: §D.2 · p. 16, Figure 7 · p. 16

E.1 ALGORITHM FOR THE AUTOMATE DATA GENERATION PIPELINE

  • The automated pipeline prompts a linguistic annotator for reasoning steps, obtains visual content for referenced boxes, builds a tree, and traverses it for positive chains.The pseudocode is provided to clarify the synthesis algorithm and support reproduction.Source: §E.1 · p. 17

E.2 THE COM-TEST BENCHMARK AND EVALUATION METRIC

  • The keypoints-aware metric evaluates ordered manipulation elements with normalized Levenshtein distance and combines that score with BLEU matching between chains.The reported combination weights the sequence score by 0.6 and BLEU by 0.4, then divides by 2.Source: §E.2 · p. 17

E.3 DATA STATISTICS

The appendix describes data-statistics and annotation materials, outlines linguistic and visual annotation settings, lists data-production limitations, and gives example prompts for eliciting manipulations.

E.3 DATA STATISTICS

  • The data-statistics materials include predicate-phrase extraction, implicit reasoning chains for absurd unanswerable questions, and a table of training and evaluation data statistics.The figure caption describes the top 50 generated manipulations out of 465 under 4-shot prompting, while another figure shows linguistic and visual annotation inputs and outputs.Source: §E.3 · p. 18, Table 5 · p. 18, Figure 8 · p. 18

E.4 DETAILS OF THE LINGUISTIC/VISUAL ANNOTATIONS

  • The annotation setup uses GPT-4-turbo for solving steps and GroundingDINO and PaddleOCR for requested visual contents.The paper reports API use from 2023.9 to 2023.12 and points to an example-based annotation figure.Source: §E.4 · p. 19

E.5 LIMITATION ANALYSIS FOR THE DATA PRODUCTION

  • The data-production limitations are insufficient diversity in linguistic steps and visual-tool errors such as coarse boxes and recognition failures on slanted letters or long sentences.The authors suggest diversified prompts and stronger visual-tool semantic understanding as remedies.Source: §E.5 · pp. 19–20

F.1 LAUNCHING PROMPTS

  • The example prompts ask models to solve incrementally and optionally use grounding, OCR, crop-and-zoom, calculation, or newly devised manipulations.Source: §F.1 · p. 20

F.2 TRAINING SETTINGS

Table 6 presents the training details for all stages.

F.2 TRAINING SETTINGS

G.1 QUALITATIVE ANALYSIS

CogCoM is shown handling detailed visual reasoning through task-specific manipulation sequences, alongside a range of multimodal scenarios.

G.1 QUALITATIVE ANALYSIS

  • CogCoM locates a plane logo through grounding before zooming into its cropped region, reads time after locating the device, and counts truck wheels using grounding and counting.Source: §G.1 · p. 20

G.2 REPRESENTATIVE CASES

  • Figure 10 presents CogCoM in evidential visual reasoning, visual grounding, grounded captioning, image captioning, multiple choice, and detailed captioning scenarios.Source: Figure 10 · p. 21
Loading 2402.04236v3…