PaperlayerView exact revision on arXiv

Source-linked AI summary

Distilling Visual Reasoning into Text Space

Wenhan Yang, Nilay Naharas, Ali Payani, Baharan Mirzasoleiman

arXiv:2609.34408v1cs.CV

TL;DR

LVLMs can struggle with tasks requiring visual concepts beyond what is directly observable, while intermediate visual representations can introduce errors and interfere with textual reasoning. V2T distills visual grounding from an interleaved-image teacher into a text-only student, which consistently outperforms its teacher and existing baselines across multimodal benchmarks.

  • Problem

    LVLMs often struggle with reasoning that requires concepts beyond the directly observable image, and intermediate visual representations can be inaccurate or interfere with textual reasoning.

    Source: §1 · pp. 1–2

  • Method

    V2T trains a teacher on interleaved visual and textual reasoning, then distills its logits and ground-truth textual CoT into a student that reasons in text space.

    Source: §3.2 · p. 4, §3.2.1 · p. 5

  • Results

    Across multimodal benchmarks, V2T consistently outperforms its teacher and existing baselines, including a 14.3% improvement over its teacher on the held-out set.

    Source: §1 · p. 2

  • Takeaways & Limitations

    At inference, V2T reasons entirely in text at the same 1× cost as text-only models, without intermediate visual representations or their error propagation.

    Source: §6 · p. 10

    V2T requires paired interleaved multimodal reasoning data that is expensive to collect and limited in scale.

    Source: IMPACT STATEMENT · p. 14

Abstract

from arXiv · show

Large Vision-Language Models (LVLMs) have shown strong promise for multimodal reasoning, yet often struggle with tasks requiring concepts beyond what is directly observable in the input image. Existing methods generate intermediate images or latent visual tokens to guide reasoning, but these representations can introduce errors and increasingly interfere with textual reasoning as reasoning progresses. We propose Visual-to-Text Chain-of-Thought Distillation (V2T), a framework that enables LVLMs to internalize visual reasoning without generating intermediate visual representations at inference time. V2T first trains a teacher LVLM using interleaved visual and textual chains of thought, and then uses knowledge distillation to train a student LVLM using the teacher's logits and cross-entropy supervision from ground-truth textual reasoning. When reasoning images can be mapped to the original image, V2T can additionally distill the teacher's attention to corresponding regions, while ground-truth bounding boxes can further guide a subsequent reinforcement learning stage. Experiments across multiple multimodal reasoning benchmarks show that V2T consistently outperforms the teacher and existing baselines, improving average accuracy by 14.3% on a held-out set and 2.7% on the broader visual evaluation suite. Moreover, lightweight SFT and substantially reduced RL make V2T up to 42x faster to train than state-of-the-art baselines.

1 INTRODUCTION

LVLMs can struggle when tasks require implicit visual states, while interleaved images may introduce errors and increasingly interfere with textual reasoning. V2T distills visual reasoning into a text-only student and outperforms existing methods across evaluated benchmarks.

Motivation

  • Many visual tasks require intermediate states absent from the raw image, such as auxiliary geometric lines or points that reveal implicit relationships.LVLMs trained primarily in text space often struggle to reason over these representations.Source: §1 · p. 1
  • Generated intermediate images or latent visual tokens can be inaccurate or incomplete, while interleaved images increasingly interfere with textual reasoning as it progresses.Irrelevant visual content further increases the loss gap between reasoning with and without interleaved images.Source: §1 · p. 2, Figure 2 · p. 2

Main results

  • 75.5% average accuracy on the held-out set, 14.3% above the teacher and 14.6% above the strongest existing baseline.V2T also achieves a 61.8% overall score on general benchmarks, 2.7% above the strongest baseline.Source: §1 · p. 2

Contribution

  • V2T consistently outperforms both external visual-tool and latent visual-reasoning methods across in-domain and broad multimodal reasoning benchmarks.The framework distills a teacher trained with interleaved reasoning images into a student that reasons entirely in text, with optional attention distillation and reinforcement learning when regions are available.Source: §1 · p. 2

2 RELATED WORK

Visual chain-of-thought work includes text-only, tool-augmented, and interleaved latent approaches. V2T instead transfers visual grounding from interleaved teacher reasoning into a student that reasons in text space.

Text-Only Visual Chain of Thoughts

  • Text-only visual CoT trains models to verbalize perception and reason, but long text sequences can overshadow the underlying visual information.This approach pairs images with detailed textual reasoning steps.Source: §2 · pp. 2–3

Tool-Augmented Visual Chain of Thoughts

  • Tool-augmented methods feed generated images or annotations back into reasoning, but tool-policy demands, tool limits, and image inaccuracies constrain them.These methods can use bounding boxes, sketches, crops, or zooming.Source: §2 · p. 3

Interleaved Latent Visual Chain of Thoughts

  • Interleaved latent methods represent visual evidence with tokens or updated visual states, but these inputs can increasingly interfere with textual reasoning.Examples include localized region tokens, iterative latent states, and representations decoded into approximate image regions.Source: §2 · p. 3

V2T

  • V2T distills reasoning from a teacher trained with interleaved images into a student that reasons entirely in text space.The training framework uses teacher logits and textual CoT supervision, with attention distillation to transfer visual grounding.Source: §2 · p. 3, Figure 3 · p. 3

3 DISTILLING VISUAL CHAINS OF THOUGHT INTO TEXT SPACE

V2T transfers visual reasoning learned from interleaved image-text trajectories into a student that generates textual CoT using the original problem image alone. Optional attention distillation and grounding-reward RL add region-level supervision when mappings or boxes are available.

3.1 REASONING WITH LARGE VISION-LANGUAGE MODELS

  • Standard VLMs lack a reliable mechanism to generate faithful intermediate visual states, while interleaved visual inputs can increasingly interfere with textual reasoning and propagate errors.These limitations motivate reasoning in text space while retaining intermediate visual information.Source: §3.1 · p. 4

3.2 V2T: DISTILLING VISUAL CHAINS OF THOUGHT INTO TEXT SPACE

  • V2T trains a teacher on CoT steps paired with reasoning images, then distills its visual reasoning into a student that reasons entirely in text space.The student uses teacher logits and ground-truth textual CoT.Source: §3.2 · p. 4

3.2.1 STAGE 1: TRAINING THE TEACHER MODEL ON MULTIMODAL REASONING DATA

  • The teacher is trained with cross-entropy on textual tokens conditioned on preceding text-image pairs; reasoning images are conditioning inputs, not prediction targets.The loss is computed token by token under teacher forcing.Source: §3.2.1 · pp. 4–5
  • 75.5%: V2T achieves the best overall held-out Zebra-CoT result, outperforming the interleaved teacher by 14.3%.Table 1 covers the held-out set.Source: Table 1 · p. 5

3.2.2 STAGE 2: DISTILLING MULTIMODAL REASONING INTO THE TEXT SPACE

  • The student receives teacher next-token logits and ground-truth text supervision while seeing only the prompt and original image; the teacher uses ground-truth interleaved trajectories to prevent generated-visual-state errors from propagating.The student receives no intermediate reasoning images during training or inference.Source: §3.2.2 · p. 5

3.2.3 AUXILIARY: DISTILLING VISUAL GROUNDING VIA ATTENTION

  • When reasoning images map to regions of the problem image, V2T projects teacher attention onto those regions and aligns student attention with a KL loss.Attention is extracted at the first ground-truth text token following each reasoning image.Source: §3.2.3 · p. 6

3.2.4 AUXILIARY: REINFORCEMENT VISUAL GROUNDING

  • After distillation, optional short GRPO fine-tuning can improve generalization, using task rewards and, when ground-truth boxes are available, a visual grounding reward.The grounding reward encourages attention to task-relevant image regions.Source: §3.2.4 · p. 6

4 EXPERIMENTS

V2T outperforms its teacher and existing baselines on held-out and broader visual benchmarks. Ablations show additional gains from attention distillation and visual grounding rewards.

4.1 MAIN RESULTS

  • 61.8% average score: V2T beats the strongest existing visual reasoning models by 2.7% on the broader suite; Table 2 also reports 63.4 for V2T + GRPO.V2T without GRPO also exceeds Qwen2.5-VL-7B by 4.0%.Source: Table 2 · p. 7, §4.1 · p. 8

4 EXPERIMENTS

  • V2T uses 20 H100 GPU-hours for teacher SFT and student distillation, or 80 with GRPO, and matches text-only inference cost at 1×.Even with GRPO, it uses 17.8×, 19.2×, and 42.4× fewer GPU-hours than Thyme, DeepEyes, and Vision-R1-7B, respectively.Source: §4 · p. 7
  • Evaluation covers held-out Zebra-CoT tasks and broader visual benchmarks; experiments compare text-based, latent, interleaved, and tool-based methods.The Zebra-CoT training set is filtered to about 23K examples, and the held-out tasks are Graph Algorithm, RPM, Tetris, and Visual Search.Source: §4 · pp. 6–7

4.1 MAIN RESULTS

  • 75.5% average accuracy: V2T leads the Zebra-CoT held-out set, beating the Interleaved SFT teacher by 14.3% and the strongest baseline by 14.6%.It performs best in all four held-out task categories.Source: §4.1 · p. 8

4.2 ABLATION STUDIES

  • Attention distillation raises the average from 57.1% to 61.8%, while the visual grounding reward raises the GRPO average from 62.0% to 63.4%.Gains vary by task: attention distillation helps most on HR-Bench 4K/8K and V*Bench; the reward particularly improves BLINK Functional Correspondence.Source: §4.2 · p. 8

5 ERROR ANALYSIS

Attention analysis finds stronger grounding to the original problem image with V2T, while error analysis shows its gains are concentrated in spatial and task-consistency errors.

Attention Analysis

  • Attention IoU uses top-k attended pixels against ground-truth boxes, averaging middle-layer attention and examining the first text token after image insertion.The analysis covers 100 random Zebra-Visual Search examples.Source: §5 · p. 9
  • 0.269 problem-image IoU: V2T scores highest, while the interleaved teacher's problem-image IoU falls to 0.168 when reasoning images are provided.The teacher's attention shifts to the reasoning image (IoU 0.234).Source: §5 · p. 9, Table 4 · p. 9

Error Analysis

  • V2T reduces total errors by 7.4% relative to Interleaved SFT, with the largest reductions in Spatial and Global Context (−16.1%) and Task and Answer Consistency errors (−14.4%).Perception and Grounding errors remain similar, while Reasoning and Knowledge errors still dominate.Source: §5 · p. 9, Table 5 · p. 9

6 CONCLUSION

V2T distills training-time interleaved visual information into text-based chain-of-thought reasoning. The conclusion reports stronger benchmark performance and text-only inference without intermediate visual representations.

6 CONCLUSION

  • V2T uses interleaved visual information as training-time supervision and distills visual grounding into text-based chain-of-thought reasoning.Source: §6 · p. 10
  • The paper reports that V2T outperforms text-only, latent visual reasoning, and interleaved models, including its teacher, on held-out and broader visual benchmarks.Source: §6 · p. 10
  • At inference, V2T reasons entirely in text at the same 1× cost as text-only models, without intermediate visual representations or their error propagation.Source: §6 · p. 10

IMPACT STATEMENT

The impact statement identifies limitations in training data, teacher dependence, error types, and evaluation scale. The appendices describe training, evaluation, baselines, and dataset composition.

IMPACT STATEMENT

  • The authors report no societal consequences they feel must be specifically highlighted.Source: IMPACT STATEMENT · p. 14
  • V2T requires costly, limited-scale paired intermediate-image reasoning data, inherits teacher biases or errors, and has smaller gains on perception and domain-knowledge errors.Experiments use a single training run; multi-seed evaluation and larger model scales remain future work.Source: IMPACT STATEMENT · p. 14

A.1 SFT HYPERPARAMETERS

  • SFT uses LoRA with rank 32 and α 64 on four H100 80GB GPUs, with training sequences constrained to 512 × 28 × 28–1280 × 28 × 28 image pixels.Source: §A.1 · p. 14, Table 6 · p. 14

A.2 GRPO HYPERPARAMETERS

  • The optional GRPO stage starts from the distilled student, trains 160 steps on TreeVGR-RL-37K, and takes approximately 60 GPU-hours.Each prompt uses five rollouts; the reward combines answer correctness, format, and visual grounding.Source: §A.2 · p. 14, Table 7 · p. 15

A.3 EVALUATION AND BASELINE DETAILS

  • Evaluation uses 800 held-out Zebra-CoT examples and broader visual benchmarks, with VLMEvalKit and Qwen3-235B as judge; baselines are grouped by reasoning approach.Broader tests include high-resolution perception, visual search, correspondence, localization, multi-view reasoning, and hallucination.Source: §A.3 · p. 15
  • Tool-based baselines use more expensive RL; DeepEyes, for example, uses 1536 GPU-hours and 16 rollouts per prompt, while V2T uses five text-only rollouts in GRPO.Thyme and DeepEyes train on V* data, which shares its distribution with V*Bench; V2T lacks comparable V* training examples.Source: §A.3 · p. 15

A.4 FILTERED ZEBRA-COT COMPOSITION

A.5 HYPERPARAMETER ABLATIONS

The ablations vary V2T’s logit-distillation and attention-supervision weights, and V2T+GRPO’s visual-grounding reward weight. The best tested settings are λ = 0.1, β = 0.5, and η = 0.5, respectively.

A.5 HYPERPARAMETER ABLATIONS

  • The ablations vary logit distillation weight λ, attention supervision weight β, and visual grounding reward weight η.Source: §A.5 · p. 16
  • 61.8% average accuracy is V2T’s best tested result, with λ = 0.1 and β = 0.5.The other tested settings yield 56.2%, 56.7%, and 56.5%.Source: Figure 6 · p. 16, §A.5 · p. 16
  • 63.4% average accuracy is V2T+GRPO’s best tested result, with η = 0.5 versus 63.0% for η = 0.1.The paper uses η = 0.5 in its final configuration.Source: Figure 6 · p. 16, §A.5 · p. 16

B CONTROLLED ANALYSIS OF INTERLEAVED VISUAL REASONING

The controlled analyses find that interleaved images and visually unmatched content can disrupt subsequent textual reasoning. Generation results show that images can help when few are present, but removing them matches or outperforms interleaving as their number grows.

B CONTROLLED ANALYSIS OF INTERLEAVED VISUAL REASONING

  • At 100% of the CoT, loss gaps reach 4.53 for Checkers, 2.29 for Connect Four, and 3.91 for Object Counting.Across those categories, the gaps grow toward the end of the reasoning trajectory; mean loss gaps also rise from +0.38 with three images to +5.99 with nine.Source: B CONTROLLED ANALYSIS OF INTERLEAVED… · p. 17
  • Increasing irrelevant visual content by 50% raises loss gaps to 5.12 for Checkers, 9.90 for Object Counting, and 3.50 for Connect Four.The useful reasoning content is kept unchanged by placing the original image on a larger gray canvas.Source: B CONTROLLED ANALYSIS OF INTERLEAVED… · p. 17
  • With few images, visual states can help; as their number grows, removing them matches or beats interleaving, while gray backgrounds consistently reduce accuracy.The generation-level comparison tests retaining images, removing them, or adding 50% gray background.Source: B CONTROLLED ANALYSIS OF INTERLEAVED… · p. 17
  • The analyses motivate using intermediate images as training-time supervision and distilling their useful reasoning signal into text space for inference.This avoids repeated visual conditioning and the additional interference from irrelevant visual tokens.Source: B CONTROLLED ANALYSIS OF INTERLEAVED… · p. 17

C.1 MATH-FOCUSED REASONING MODELS

VLAA-Thinker-7B outperforms the Qwen2.5-VL-7B base model on the four math-focused benchmarks examined. Its math-focused training improves mathematical reasoning but transfers less strongly to the broader visual tasks evaluated.

C.1 MATH-FOCUSED REASONING MODELS

  • VLAA-Thinker-7B consistently exceeds Qwen2.5-VL-7B on MathVerse, MathVista, MM-Math, and DynaMath.The table reports performance on these math-focused reasoning benchmarks.Source: §C.1 · p. 17, Table 9 · p. 18
  • The results suggest math-focused training improves mathematical reasoning while transferring less strongly to the broader visual tasks evaluated.Source: §C.1 · p. 18
  • The comparison covers Qwen2.5-VL-7B and VLAA-Thinker-7B on math-focused reasoning benchmarks.Source: Table 9 · p. 18

D JUDGE CONSISTENCY ANALYSIS

On the selected evaluated subset, two judge models produce nearly identical scores. The accompanying examples contrast model answers on traffic-light recognition and flower counting and show interleaved reasoning examples for chemistry and chess.

D JUDGE CONSISTENCY ANALYSIS

  • The two judges’ scores differ by at most 0.5 percentage points across HR-Bench 4K, HR-Bench 8K, and V*Bench.GPT-4o is used only for this cross-check; Qwen3-235B-A22B-Instruct-2507-FP8 remains the main-evaluation judge.Source: D JUDGE CONSISTENCY ANALYSIS · p. 18, Table 10 · p. 18

E.1 TRAFFIC-LIGHT RECOGNITION

  • On the traffic-light example, V2T predicts the ground-truth Red by focusing on distant lights, while the base and interleaved SFT models predict Green.V2T reasons over the original image; no intermediate image is generated or provided.Source: Figure 7 · p. 19

E.2 FLOWER COUNTING

  • On the flower-counting example, V2T identifies two white flowers, while the base and interleaved SFT models under-count and predict one.V2T focuses on the central floral arrangement and predicts the ground-truth answer.Source: Figure 8 · p. 20

F INTERLEAVED COT EXAMPLES

Related papers

Loading 2609.34408v1…