PaperlayerView exact revision on arXiv

Source-linked AI summary

Render-of-Thought: Rendering Textual Chain-of-Thought as Images for Visual Latent Reasoning

Yifan Wang, Shiyu Li, Peiming Li, Xiaochen Yang, Yang Tang, Zheng Wei

arXiv:2601.14750v4cs.CLcs.CV

TL;DR

CoT improves LLM reasoning but its verbosity creates computational overhead, while outcome-focused latent methods lack supervision of intermediate reasoning. RoT renders textual reasoning steps into images and uses pre-trained vision encoders as semantic anchors for visual latent reasoning. It reports 3-4× token compression and inference acceleration compared with explicit CoT while maintaining competitive accuracy across mathematical and logical benchmarks.

  • Problem

    Verbose CoT imposes computational overhead, while prior latent methods lack intermediate-process supervision and make reasoning chains difficult to analyze.

  • Method

    RoT renders textual reasoning steps into images and uses pre-trained vision encoders as semantic anchors to align visual embeddings with textual space.

  • Results

    3-4× token compression and significant inference acceleration are reported compared with explicit CoT, with competitive accuracy across mathematical and logical benchmarks.

  • Takeaways & Limitations

    Visual rendering is presented as a viable paradigm for efficient and analyzable latent reasoning within standard VLM architectures.

  • Takeaways & Limitations

    Evaluation is limited to English-language mathematical and logical reasoning tasks, leaving other reasoning domains and non-English languages unexplored.

Abstract

from arXiv · show

Chain-of-Thought (CoT) prompting has achieved remarkable success in unlocking the reasoning capabilities of Large Language Models (LLMs). Although CoT prompting enhances reasoning, its verbosity imposes substantial computational overhead. Recent works often focus exclusively on outcome alignment and lack supervision on the intermediate reasoning process. These deficiencies obscure the analyzability of the latent reasoning chain. To address these challenges, we introduce Render-of-Thought (RoT), the first framework to reify the reasoning chain by rendering textual steps into images, making the latent rationale explicit and traceable. Specifically, we leverage the vision encoders of existing Vision Language Models (VLMs) as semantic anchors to align the vision embeddings with the textual space. This design ensures plug-and-play implementation without incurring additional pre-training overhead. Extensive experiments on mathematical and logical reasoning benchmarks demonstrate that our method achieves 3-4x token compression and substantial inference acceleration compared to explicit CoT. Furthermore, it maintains competitive performance against other methods, validating the feasibility of this paradigm. Our code is available at https://github.com/TencentBAC/RoT

1 Introduction

CoT improves LLM reasoning but incurs substantial efficiency costs, while prior latent approaches can obscure intermediate rationale. Render-of-Thought addresses both issues by rendering reasoning steps into images and anchoring latent states to visual embeddings.

  • Verbose CoT increases inference latency and memory consumption, limiting efficiency and scalability.
  • Prior latent-reasoning methods often prioritize outcome alignment without supervising intermediate reasoning, obscuring rationale and logical errors.
  • Render-of-Thought renders textual reasoning steps into images, keeping the rationale explicit while compressing the reasoning process.
  • A two-stage strategy first aligns LLM latent representations with rendered-CoT visual embeddings, then enables autoregressive visual reasoning without explicit text decoding.
  • 3-4× token compression and marked inference acceleration are reported relative to explicit CoT in experiments using Qwen3-VL-4B-Instruct.
  • Pre-trained vision encoders serve as semantic anchors aligning visual embeddings with textual space, enabling plug-and-play implementation without additional pre-training.

2 Related Work

Related work has compressed textual reasoning or expanded context through visual rendering, but RoT applies visual rendering specifically to compress VLM reasoning steps.

  • Existing CoT-compression methods select pivotal tokens, remove redundant content, chunk reasoning, or use reinforcement learning to shorten inference paths.
  • Text-as-Image research has mainly addressed input compression rather than compression of reasoning steps.
  • RoT is presented as the first framework to apply visual rendering to compress VLM reasoning steps.

3 Method

RoT compresses textual reasoning into visual latent representations through a progressive two-stage training and decoding framework. The method aligns language states with rendered-CoT visual embeddings, then trains latent reasoning generation and answer decoding.

  • Framework: RoT converts textual CoT into compact visual representations whose encoder features supervise the LLM’s latent reasoning.A pretrained vision encoder extracts features from rendered reasoning images, while the LLM generates continuous latent representations instead of verbose textual steps.
  • Two-Stage Training: The progressive training paradigm first aligns LLM hidden states with visual embeddings before enabling autoregressive latent reasoning.This reverses the usual MLLM direction by mapping linguistic states into visual embedding space at the output side.
  • Stage I: Visual Alignment: Stage I jointly models visual alignment with cross-entropy objectives for the image-end token and final answer.The Stage I objective includes prediction of the special token and answer, alongside the alignment loss.
  • Stage II: Latent Reasoning: Stage II freezes the vision encoder and projection head, then fine-tunes the LLM backbone with LoRA to generate latent visual tokens followed by the answer.The frozen projection head constrains generated hidden states to map to meaningful visual representations, while training optimizes answer and termination-token likelihood.
  • Inference and Decoding Strategies: Inference supports dynamic termination through a special token or static termination through a fixed latent-chain budget.Dynamic decoding stops when the termination token is most probable, whereas static decoding manually appends the image-end token at the budget threshold.
  • Experimental Results: The framework’s experimental comparison reports significant token compression while maintaining competitive accuracy against explicit CoT.Table 1 evaluates four grade-school reasoning datasets across three VLM architectures using accuracy-to-token trade-offs.

4 Experiments

Experiments evaluate RoT across mathematical reasoning datasets, model architectures, rendering configurations, training stages, decoding strategies, and latent-token behavior. The results show substantial token and inference-time efficiency, competitive reasoning performance, and sensitivity to training and rendering choices.

  • Experiment Settings: Experiments span GSM8k-Aug-NL, GSM-Hard, SVAMP, MultiArith, and MATH, measuring Pass@1 accuracy and average reasoning-chain length.The study uses five random seeds and reports mean values with 95% confidence intervals.
  • Main Results: 55.4% average accuracy with 32 latent tokens compares with 79.3% accuracy using 108.4 tokens for explicit CoT on Qwen3-VL-4B-Instruct.On MultiArith, RoT reaches near-parity performance with a 1.8× reduction in token consumption.
  • Main Results: 8.55s to 1.84s reduces GSM-Hard per-sample inference time for RoT relative to explicit CoT on a single NVIDIA H20 GPU.The passage attributes the latency reduction to compressing textual thoughts into visual latent embeddings.
  • Ablation Study & Analysis: Removing Stage I drops GSM8k-Aug accuracy from 37.8% to 24.8%, while removing Stage II lowers MATH performance from 33.2% to 26.2%.The ablation identifies visual alignment and latent-space navigation as necessary components of the two-stage framework.
  • Ablation Study & Analysis: 37.8% Pass@1 accuracy makes single-line rendering outperform the fixed-size square baseline on GSM8k-Aug.Dynamic width removes blank regions, preserves complete text, and represents reasoning as a continuous visual sequence.
  • Ablation Study & Analysis: Fixed token budgets consistently outperform dynamic termination via special tokens because self-regulated stopping is unstable in continuous latent spaces.The optimal budget varies by dataset: 32 tokens for GSM8k-Aug and 64 for MATH.

5 Conclusion

Render-of-Thought compresses textual Chain-of-Thought into visual latent representations using pre-trained vision encoders as semantic anchors. It reports 3-4× token compression and significant inference acceleration while maintaining competitive accuracy across mathematical and logical benchmarks.

  • Render-of-Thought compresses Chain-of-Thought reasoning by rendering textual steps into visual latent representations.
  • Pre-trained vision encoders serve as semantic anchors that bridge the modality gap within standard Vision Language Model architectures.
  • 3-4× token compression and significant inference acceleration are achieved compared to explicit Chain-of-Thought reasoning.
  • Competitive accuracy is maintained across mathematical and logical reasoning benchmarks.

Limitations

The evaluation is limited to English-language mathematical and logical reasoning tasks, while token budgets require task-specific calibration and training incurs additional rendering and vision-encoder overhead.

  • Evaluation is currently limited to English-language mathematical and logical reasoning tasks.Effectiveness on commonsense reasoning, causal inference, and non-English languages remains unexplored.
  • Optimal latent token budgets require task-specific calibration, which may be impractical when task complexity is unknown beforehand.The authors report different optimal budgets for GSM8k-Aug and MATH.
  • Dynamic termination via special tokens exhibits instability in continuous latent spaces.The authors identify this as a phenomenon requiring future investigation.
  • Training incurs additional computational overhead from rendering textual Chain-of-Thought into images and processing them through the vision encoder.The authors state that this cost is eliminated during inference.

Ethics Statement

The work uses public mathematical and logical reasoning datasets, open-source vision-language models under research-permitting licenses, and rendered content containing mathematical reasoning without personal data or offensive material.

  • The evaluation uses publicly available datasets for grade-school and challenging mathematical reasoning.The listed datasets include GSM8k-Aug, GSM8k, GSM-Hard, SVAMP, MultiArith, and MATH.
  • The datasets were assessed as containing generic or fictional names rather than information identifying real-world individuals.Consequently, no additional anonymization or de-identification was required beyond standard public-benchmark use.
  • Open-source vision-language models were used under licenses permitting research use, with their terms of use reviewed for compliance.
  • Rendered images contain mathematical reasoning steps and problem solutions without personal data or offensive content.A sample of rendered images was manually inspected for inappropriate material.

A More Implementation Details

Implementation uses LoRA-fine-tuned vision-language backbones, special latent-state tokens, and a visual projection head. Ablations examine projection-head architecture, rendering configurations, and training choices.

  • Experiments use Qwen3-VL-2B/4B-Instruct, LLaVa-V1.6-Mistral-7B, and Qwen3-4B-Instruct backbones with LoRA fine-tuning.The LoRA configuration uses α = 32, r = 16, and dropout 0.05.
  • Stage I alignment uses loss weight λ = 10.0, with training conducted on two NVIDIA H20 GPUs and reproducibility seeds fixed to 0.
  • Special image-boundary tokens are initialized from normalized random vectors scaled to norm √hd.The scaling is intended to match the typical norm of pre-trained embeddings for numerical compatibility and training stability.
  • The method is evaluated on GSM8K-Aug, GSM8K-Hard, SVAMP, MultiArith, and MATH, with a 10% validation allocation for MATH.
  • The visual projection head maps LLM hidden states into visual embedding space and is evaluated through activation-function and hidden-dimension ablations.
  • 4096 is the default hidden dimension, while reducing it to 2048 causes noticeable performance degradation, particularly on MATH.

C Ablation Study on Visual Rendering Configurations

The ablation evaluates image height, font size, and padding for visual rendering, identifying 32 px height, 20 px font size, and 4 px padding as the best configuration. Image height is especially important because insufficient resolution degrades semantic encoding.

  • 32 px height, 20 px font size, and 4 px padding achieve the optimal accuracy in the rendering ablation.The study varies image height, font size, and padding using Qwen3-VL-4B-Instruct.
  • 16 px height causes significant performance degradation, while 64 px height does not consistently improve performance.The authors associate the degradation at 16 px with blurred character details and regard 32 px as generally sufficient for legibility.
  • Font sizes deviating from 20 px negatively affect results, potentially by distorting character strokes or changing spatial density.The proposed explanation relates the optimum to the spatial features expected by the pretrained encoder.
  • 4 px padding avoids boundary artifacts and preserves character features within the visual receptive field.

D Training Cost

RoT requires more training time than text-only SFT-CoT because of online rendering and visual-encoder computation, but the authors report that its one-time overhead is offset by faster inference. A same-backbone comparison further reports that RoT outperforms adapted latent-reasoning baselines.

  • Training Cost: RoT training takes approximately 1.5× as long as text-only SFT-CoT because of online rendering and visual-encoder forward propagation.The comparison uses consistent training configurations on GSM8k-Aug, including batch size 16 and one NVIDIA H20 GPU.
  • Training Cost: 3-4× inference speedup during deployment significantly offsets RoT’s one-time training overhead.The authors also suggest offline rendering as a way to further reduce training cost.
  • Fair Comparison on Same VLM Backbone: RoT outperforms adapted latent-reasoning baselines on the same Qwen3-VL-4B-Instruct backbone.This comparison is presented as evidence that the improvements stem from RoT rather than VLM pre-training alone.

F Generalization to Non-Mathematical Tasks

RoT extends beyond mathematics to logical and scientific reasoning benchmarks, where it reports strong Pass@1 results and superior performance to prior latent-space approaches. Qualitative analyses also connect structured latent-token patterns with successful reasoning and disordered, repetitive patterns with failures on out-of-distribution tasks.

  • F Generalization to Non-Mathematical Tasks: RoT achieves Pass@1 accuracies of 60.5 on GPQA and 100.0 on ProsQA.The results are reported for logical and scientific reasoning datasets beyond mathematics.
  • F Generalization to Non-Mathematical Tasks: RoT outperforms previous latent-space reasoning approaches on both GPQA and ProsQA.The reported findings support generalization of the visual rendering paradigm to logical reasoning tasks.
  • Qualitative Analysis: On GSM8k-Aug, successful reasoning compresses the path into 32 latent embeddings with diagonal, locally coherent similarity patterns.These patterns suggest adjacent latent tokens remain semantically related while carrying distinct information.
  • Qualitative Analysis: On MATH, successful cases use a 64-token budget and show block-diagonal similarity structures that potentially correspond to different solving stages.Rendered images containing complex LaTeX expressions are reported to align successfully with latent states.
  • Qualitative Analysis: Out-of-distribution failures show large, highly similar blocks and higher variance, indicating repetitive or indistinguishable latent tokens.The analyzed datasets include GSM-Hard, SVAMP, and MultiArith; the authors associate these patterns with unfamiliar problem structures and incorrect decoding.
Loading 2601.14750v4…