PaperlayerView exact revision on arXiv

Source-linked AI summary

Chain-of-Visual-Thought: Teaching VLMs to See and Think Better with Continuous Visual Tokens

Yiming Qin, Bomin Wei, Jiaxin Ge, Konstantinos Kallidromitis, Stephanie Fu, Trevor Darrell, XuDong Wang

arXiv:2511.19418v3cs.CVcs.AIcs.LG

TL;DR

VLMs struggle with perception-intensive reasoning because discrete text space poorly represents dense spatial and geometric cues. COVT teaches them to reason with compact continuous visual tokens aligned to lightweight vision experts, improving diverse vision-centric benchmarks while supporting efficient and interpretable inference.

  • Problem

    Current VLMs often struggle with counting, spatial correspondence, and relative depth estimation because dense perceptual cues are lost or poorly represented in discrete text space.

  • Method

    COVT trains VLMs to predict compact continuous visual tokens aligned with lightweight experts for segmentation, depth, edges, and visual representations.

  • Results

    COVT consistently improves fine-grained visual reasoning across diverse perception benchmarks, including gains of 5.5% on CV-Bench and 4.5% on HRBench8K.

  • Takeaways & Limitations

    Compact continuous visual thinking provides a self-contained, interpretable framework for integrating fine-grained perceptual reasoning into multimodal systems.

  • Takeaways & Limitations

    COVT has not exhaustively explored visual-model or token-combination designs and does not yet fully interleave textual and visual reasoning.

Abstract

from arXiv · show

Vision-Language Models (VLMs) excel at reasoning in linguistic space but struggle with perceptual understanding that requires dense visual perception, e.g., spatial reasoning and geometric awareness. This limitation stems from the fact that current VLMs have limited mechanisms to capture dense visual information across spatial dimensions. We introduce Chain-of-Visual-Thought (COVT), a framework that enables VLMs to reason not only in words but also through continuous visual tokens-compact latent representations that encode rich perceptual cues. Within a small budget of roughly 20 tokens, COVT distills knowledge from lightweight vision experts, capturing complementary properties such as 2D appearance, 3D geometry, spatial layout, and edge structure. During training, the VLM with COVT autoregressively predicts these visual tokens to reconstruct dense supervision signals (e.g., depth, segmentation, edges, and DINO features). At inference, the model reasons directly in the continuous visual token space, preserving efficiency while optionally decoding dense predictions for interpretability. Evaluated across more than ten diverse perception benchmarks, including CV-Bench, MMVP, RealWorldQA, MMStar, WorldMedQA, and HRBench, integrating COVT into strong VLMs such as Qwen2.5-VL and LLaVA consistently improves performance by 3% to 16% and demonstrates that compact continuous visual thinking enables more precise, grounded, and interpretable multimodal intelligence.

1. Introduction

COVT addresses the loss of dense perceptual information caused by reasoning through discrete text by introducing compact continuous visual tokens into VLM reasoning. It aligns these tokens with lightweight vision experts, enabling grounded and interpretable visual reasoning while improving perception benchmarks.

  • Motivation: Current VLMs lose boundaries, layout, depth, and geometry when continuous visual information is projected into discrete text space.This limitation affects counting, spatial correspondence, and relative depth estimation.
  • Motivation: External vision tools can restore some fine-grained perception but add computational cost and constrain outcomes to the tools’ capabilities.Image generation or cropping still projects visual information into text space.
  • COVT: COVT inserts compact continuous visual tokens into VLM reasoning so models can process fine-grained perceptual cues directly rather than translating everything into words.The framework targets self-contained visual reasoning while remaining efficient.
  • COVT: COVT aligns visual-token groups with segmentation, depth, edge, and representation experts to reconstruct corresponding dense supervision signals.Task-oriented signals use prompt-level alignment, while representation-based signals use feature-space alignment.
  • Interpretability: COVT visual tokens can be decoded into dense predictions for human-interpretable visualizations, or retained in latent space when visualization is unnecessary.This supports both interpretability and efficient inference.
  • Results: 5.5% overall gain on CV-Bench, 14.0% on its depth sub-task, and 4.5% overall gain on HRBench demonstrate consistent improvements across perception benchmarks.COVT also maintains competitive performance on general non-vision-centric benchmarks and can decode dense predictions for visualization.

2. Related Work

Prior multimodal reasoning methods often rely on text bottlenecks, supplementary images, or external tools. COVT instead embeds flexible, expert-linked visual reasoning directly in dense continuous latent space.

  • Tool-Augmented Reasoning: External vision tools provide specialized perception but introduce computational overhead and bound performance by tool capabilities.COVT considers self-contained visual reasoning as an alternative.
  • Text Space Reasoning: Dense-caption approaches translate visual content into language, making the reasoning process inherently lossy.This preserves the discrete text-space bottleneck for visual reasoning.
  • Multimodal Reasoning Paradigms: Visual CoT, MCoT, and VChain address visual reasoning through textual interpretations or supplementary images but retain information-loss or compute limitations.The cited comparison distinguishes these approaches from COVT’s continuous visual-space design.
  • Latent Space Reasoning: COVT uses continuous visual tokens tied to perceptual experts, extending latent-space reasoning to dense visual information.Its visual tokens are linked to specific expert functions rather than serving as generic latent reasoning states.
  • COVT Pipeline: COVT’s visual-token pipeline aligns SAM, DepthAnything, PIDINet, and DINO with token groups for masks, depth, edges, and patch-level features.The figure specifies allocations of 8, 4, 4, and 4 tokens respectively.
  • Comparison: COVT is described as uniquely combining continuous visual-space reasoning, dense visual cues, arbitrary perception types, and operation without external tools.These properties are presented as a comparison with prior multimodal reasoning methods.

3. Chain-of-Visual-Thought (COVT)

COVT augments VLMs with continuous visual tokens that encode fine-grained perceptual information and can be used within multimodal reasoning chains. It trains these tokens through staged data formatting and reconstruction supervision, improving vision-centric performance while supporting flexible token integration.

  • Motivation: COVT addresses text-only reasoning’s accumulated errors and weak supervision for low-level perceptual cues by adding visual information to the reasoning process.The framework targets edges, depth, regions, spatial relationships, and geometry that text-dominated supervision inadequately captures.
  • Overall Pipeline: COVT lets VLMs output continuous visual tokens that encode fine-grained representations and support reasoning over rich perceptual information.Each token group corresponds to a lightweight perceptual expert, while task-specific decoders reconstruct visual outputs under supervision.
  • COVT Tokens: COVT retains next-token prediction while allowing each generated token to represent either visual or textual content.During training, task-specific decoders reconstruct visual outputs from the generated visual tokens; visual thinking tokens are not decoded during inference.
  • COVT Tokens: COVT combines segmentation, depth, edge, and DINO tokens to supervise complementary perceptual abilities with task- or feature-space alignment.Segmentation and depth use decoder prompts, edge tokens reconstruct edge maps, and DINO tokens align with patch-level features.
  • Extensibility: COVT supports adding new visual token types with minimal modification because its pipeline follows a clean next-token prediction paradigm.The framework also uses four data-formatting stages to preserve text-space ability while progressively teaching visual-token use.
  • Training Data: Four training stages teach visual-token comprehension, generation, reasoning, and efficient selection within visual thought chains.The efficient reasoning stage randomly drops token-type sets so the model learns to use visual tokens dynamically.

4. Experiments

Experiments evaluate COVT across vision-centric and non-vision-centric benchmarks, qualitative visual-token behavior, alignment, grounding, and token-count ablations. COVT improves vision-centric performance, generalizes across baselines, and benefits from complementary, visually aligned tokens.

  • Benchmark Results: COVT improves by 5.5% on CV-Bench, 14.0% on CV-Bench Depth, 3.7% on MME-RealWorld, and 4.5% on HRBench8K.On LLaVA-v1.5-13B, COVT also exceeds Aurora-depth by 12.9% on BLINK relative-depth and Aurora-count by 26.6% on BLINK-count.
  • Qualitative Results: Decoded segmentation, depth, and edge tokens provide complementary 2D, 3D, and fine-grained boundary cues for perceptual reasoning.Examples show segmentation localizing regions, depth representing relative distance, and edges delineating object positions or line structures.
  • Ablation Studies: Text-only CoT often degrades vision-centric reasoning, whereas COVT consistently enhances performance across vision-centric benchmarks.The ablation keeps the training-data formatting consistent while removing visual tokens from the text-only condition.
  • Ablation Studies: Eight segmentation tokens perform best with four Depth and four DINO tokens, while too few or 32 segmentation tokens reduce performance.The poor 16-empty-token result further highlights the importance of visually aligned token embeddings.
  • Analysis: COVT yields higher answer-token similarity to image features and more precise, consistent attention over task-relevant image regions.These analyses associate COVT with stronger visual grounding and intermediate states that are more predictive of the final answer.
  • Benchmark Results: COVT remains comparable on non-vision-centric benchmarks, with a 1.2% improvement across eight such benchmarks.This result indicates that gains on vision-centric tasks do not require sacrificing performance on the broader evaluated set.
  • Ablation Studies: Decoder-based tailored alignment outperforms direct feature alignment, which loses important perceptual details from the image.The comparison is reported in the alignment ablation and supports using task-specific alignment strategies.

5. Conclusions

The paper introduces COVT, enabling VLMs to reason beyond discrete linguistic space with compact, dense visual representations. Across diverse perception benchmarks, COVT improves visual-centric reasoning and shows that different visual-token types contribute complementary aspects of multimodal understanding.

  • Conclusions: COVT enables VLMs to reason beyond discrete linguistic space using compact, dense visual representations.The framework is presented as a chain of continuous visual thoughts.
  • Conclusions: COVT consistently improves visual-centric reasoning across diverse perception benchmarks.The conclusion also reports complementary contributions from different visual-token types.
  • Conclusions: Different visual-token types contribute to complementary aspects of multimodal understanding.The paper presents COVT as a general framework for integrating fine-grained perceptual reasoning into broader multimodal systems.

Supplementary Material

The supplementary material details COVT’s projection and alignment mechanisms for segmentation, depth, edge, and representation tokens, along with reproducibility information. It also describes the dataset composition and expert-supervision pipeline.

  • Supplementary Material: The supplementary material presents detailed projection-layer, token-alignment, and dataset-composition frameworks for COVT.These materials cover segmentation, depth, edge, and DINO-related components.
  • Projection Layer: A single linear layer maps VLM latent features into decoder prompt space, or encoder feature space for DINO-token alignment.A learnable query attends to the mapped feature, which serves as both key and value.
  • Segmentation Alignment: Eight predicted segmentation tokens are projected into SAM prompt space, decoded into masks, and matched to eight filtered SAM ground truths using Hungarian assignment.The matching cost combines Dice and Focal losses, with α set to 1 in the experiments.
  • Implementation Details: The supplementary tables include the fine-tuning hyperparameter setup and list the visual-model encoders used for supervision.The listed experts include SAM, DepthAnything v2, PIDINet, and DINO v2.
  • Depth Alignment: Four depth tokens are projected into DepthAnything prompt space and combined with intermediate features to reconstruct four depth maps that are averaged.Depth supervision uses an L1 reconstruction loss against the DepthAnything v2 depth map.
  • Edge Alignment: Four edge tokens act as 1 × 1 convolutional kernels over PIDINet intermediate features, producing averaged edge maps trained with L1 loss.The aggregated edge prediction is followed by sigmoid normalization and supervised by PIDINet’s edge map.

A.5. COVT Dataset Composition

The COVT dataset combines vision-centric subsets of LLaVA-OneVision with filtered counting data and newly generated relative-depth samples.

  • COVT Dataset Composition: The COVT dataset integrates selected vision-centric LLaVA-OneVision subsets, a re-filtered 150k-sample TallyQA subset, and 5k ADE20K relative-depth samples.The TallyQA filtering reduces the proportion of zero-count answers, while the relative-depth samples follow Aurora’s methodology.

B. Additional Experiments

Additional experiments document the experimental settings, compare supervision-type configurations, and report the role of the four-stage training strategy.

  • Additional Experiments: The additional-experiments section covers experimental settings, early-stage ablations, token-count studies, and qualitative output examples.These studies are organized across Sections B.1 through B.4.
  • More Settings: The hyperparameter configurations distinguish alignment with 1, 3, or 4 supervision types.Three types jointly use segmentation, depth, and DINO tokens; four types additionally include edge tokens.
  • Training Strategy: The full four-stage training strategy provides more stable improvements than the reduced strategy using only the final two stages.Table 7 summarizes this stability comparison.

B.2. Training Stage Impact Ablation

The training-stage ablation finds that retaining all four COVT training stages produces consistent improvements, whereas using only the final two stages degrades performance on some benchmarks.

  • B.2. Training Stage Impact Ablation: The full four-stage model shows consistent and robust improvements across all evaluated benchmarks.The comparison tests the complete training strategy against a variant trained only on Stages 3 and 4.
  • B.2. Training Stage Impact Ablation: Restricting training to Stages 3 and 4 causes notable degradation on BLINK and only marginal gains elsewhere.This supports the contribution of the first two stages to the overall training strategy.

B.3. Token Numbers Ablation

The ablation varies segmentation-token count while fixing depth and DINO tokens at four, revealing a performance optimum before computational cost continues rising.

  • With 4 depth and 4 DINO tokens fixed, increasing segmentation tokens from 1 to 32 initially improves performance, then gradually reduces it.Computational overhead rises steadily as segmentation-token count increases.

B.4. More Results

Additional experiments report stable performance across varied vision-centric tasks, while the token ablation identifies eight segmentation tokens as the best configuration with four depth and four DINO tokens.

  • COVT provides more fine-grained captioning, stronger instance identification, and better counting than the Qwen2.5-VL-7B baseline in example comparisons.The examples include detailed captioning, identifying a white car hood, and counting.
  • 8 segmentation tokens produce the best performance when combined with 4 depth and 4 DINO tokens.Using more segmentation tokens causes slight performance diminishing and increases computational cost.
  • COVT identifies farthest objects, handles common-sense VQA, and maintains stable OCR performance across additional examples.The OCR examples include detecting “Sales Tax 4.24” and partially visible text “Mer”.

C. Limitations and Future Work

The paper identifies open design and reasoning-integration limitations while positioning COVT as a foundation for multimodal systems that combine language and vision.

  • Limitations and Future Work: The current study does not exhaustively explore visual-model designs or token combinations beyond representative segmentation, depth, edge, and DINO features.Alternative or hybrid visual experts may provide more expressive or complementary tokens.
  • Limitations and Future Work: COVT does not yet fully interleave continuous visual thoughts with freeform textual reasoning in a unified sequence.Developing such an integrated chain is identified as future work.
  • Limitations and Future Work: The authors position COVT as a foundation for self-contained, interpretable multimodal systems that think across language and vision.
Loading 2511.19418v3…