PaperlayerView exact revision on arXiv

Source-linked AI summary

VTC-R1: Vision-Text Compression for Efficient Long-Context Reasoning

Yibo Wang, Yongcheng Jing, Shunyu Liu, Hao Guan, Rong-cheng Tu, Chengyu Wang, Jun Huang, Dacheng Tao

arXiv:2601.22069v2cs.CL

TL;DR

Long-context reasoning faces efficiency bottlenecks, while existing compression approaches can require extra training or external models. VTC-R1 renders prior reasoning into compact visual representations and consistently improves accuracy across benchmarks while achieving up to 3.4× token compression and 2.7× end-to-end speedup.

  • Problem

    Long-context reasoning faces rapidly increasing computation and memory costs, while efficient approaches may require additional training or external compression models.

  • Method

    VTC-R1 iteratively renders completed reasoning segments into images and feeds accumulated images back to vision-language models as optical memory.

  • Results

    VTC-R1 consistently improves reasoning accuracy across multiple benchmarks, with up to 3.4× token compression and 2.7× end-to-end inference speedup.

  • Takeaways & Limitations

    VTC-R1 provides an effective alternative representation for scalable long-context reasoning.

Abstract

from arXiv · show

Long-context reasoning has significantly empowered large language models (LLMs) to tackle complex tasks, yet it introduces severe efficiency bottlenecks due to the computational complexity. Existing efficient approaches often rely on complex additional training or external models for compression, which limits scalability and discards critical fine-grained information. In this paper, we propose VTC-R1, a new efficient reasoning paradigm that integrates vision-text compression into the reasoning process. Instead of processing lengthy textual traces, VTC-R1 renders intermediate reasoning segments into compact images, which are iteratively fed back into vision-language models as "optical memory." We construct a training dataset based on OpenR1-Math-220K achieving 3.4x token compression and fine-tune representative VLMs-Glyph and Qwen3-VL. Extensive experiments on benchmarks such as MATH500, AIME25, AMC23 and GPQA-D demonstrate that VTC-R1 consistently outperforms standard long-context reasoning. Furthermore, our approach significantly improves inference efficiency, achieving 2.7x speedup in end-to-end latency, highlighting its potential as a scalable solution for reasoning-intensive applications. Our code is available at https://github.com/w-yibo/VTC-R1.

1. Introduction

VTC-R1 addresses the quadratic efficiency bottlenecks of long-context reasoning by iteratively replacing preceding textual reasoning with compact visual representations. It provides a lightweight, model-free compression paradigm, trains VLMs on rendered reasoning traces, and reports up to 3.4× token compression.

  • Long-context reasoning suffers severe computation and memory bottlenecks because transformer complexity grows quadratically with sequence length.
  • Existing efficient methods add training or sampling stages, or rely on external strong models, increasing training and inference costs.Examples include 16 sampled responses per problem and external summarization models such as Llama-3.3-70B-Instruct.
  • Vision-text compression renders textual content into visual forms, enabling VLMs to encode rich semantic information with substantially fewer vision tokens.The approach uses lightweight rendering and is described as both lightweight and model-free, with 3-4× compression in Figure 1.
  • VTC-R1 iteratively integrates vision-text compression into long-context reasoning by rendering preceding reasoning segments into compact images.The reasoning process is decomposed into a sequence of steps, with preceding steps treated as long-context visual input.
  • 3.4× token compression is achieved by segmenting long reasoning traces, rendering preceding segments into images, and forming paired image–text reasoning data.The dataset is based on OpenR1-Math-220K and is used to fine-tune Glyph and Qwen3-VL under the iterative framework.

2. Related Work

Prior work has advanced LLM reasoning and explored mechanisms for making long-context reasoning more efficient, but existing approaches incur training or information-preservation trade-offs. Vision-text compression offers a promising alternative, although its ability to support mathematically intensive, multi-step reasoning remains unclear.

  • Reasoning in Large Language Models: LLM reasoning supports rigorous domains such as mathematics and code generation, evolving from structured prompting toward reinforcement-learning-based approaches.
  • Efficient Reasoning: Long-context reasoning faces computational bottlenecks from the quadratic complexity of Transformer architectures, motivating diverse efficiency mechanisms.
  • Efficient Reasoning: Existing efficient reasoning methods can require complex multi-stage training or extensive offline sampling, substantially increasing pre-deployment costs.
  • Vision-Text Compression: Vision-text compression reduces long-sequence processing costs by transforming text into compact visual representations, but prior work does not establish faithful support for complex reasoning.The unresolved question is especially relevant to mathematically intensive and multi-step reasoning tasks.
  • Vision-Text Compression: Concurrent work applies VTC to agent histories or latent reasoning, while lacking explicit long-context reasoning coverage or systematic evaluation on challenging benchmarks.AgentOCR compresses tool-invocation histories into rendered images, whereas RoT uses rendered visual tokens as latent tokens.

3. Preliminaries

The section defines long-context reasoning as generating an answer alongside a long sequence of intermediate steps, and formalizes vision-text compression as rendering text into images that VLMs encode with fewer vision tokens. It specifies the rendering and tokenization pipeline and interprets the compression ratio as a measure of encoding efficiency.

  • Long-context reasoning produces a final answer A from question Q alongside a long sequence of intermediate reasoning steps.
  • Vision-text compression renders text into an image so a vision-language model can encode the content using fewer vision tokens.
  • The rendering pipeline maps input text T to multiple PNG images I through I = R_θ(T), with configuration controlling typography, layout, and visual style.Configuration includes parameters such as dpi, page size, font, spacing, scale, colors, and borders.
  • The rendered images are processed by the model’s image processor and vision encoder, with M_vision denoting the vision tokenizer that produces vision tokens V.
  • The original text is also tokenized by M_txt into T = {t_1, . . . , t_Lt}, where L_t is the number of text tokens.
  • A compression ratio ρ > 1 indicates that vision tokenization encodes the same content with fewer tokens, and larger ρ means greater compression efficiency.

4. Methodology

VTC-R1 reformulates long-context reasoning as iterative generation, rendering completed reasoning segments into images that serve as compact optical memory for subsequent steps. It uses adaptive iteration, lightweight rendering, and supervised fine-tuning on image–text data to reduce reasoning-context costs.

  • Iterative Reasoning: Long-context reasoning is decomposed into sequential segments, and iterative generation conditions each segment on the question and previously generated segments.The complete trace is formed by concatenating the segments.
  • Iterative Reasoning: Iterative and one-pass autoregressive generation yield the same answer distribution under the same conditionals and answer-extraction function.This equivalence follows from factorizing the full trace by the chain rule.
  • VTC-R1 Reasoning Paradigm: At each iteration, completed reasoning segments are rendered into images and supplied as vision-token optical memory for generating the next segment.The stored image set grows across iterations, while the model continues reasoning rather than restarting.
  • VTC-R1 Reasoning Paradigm: VTC-R1 selects the number of reasoning iterations dynamically by problem difficulty and imposes a maximum limit T to prevent unbounded generation.Inference continues until a final answer is produced or the iteration limit is reached, after which the answer is extracted.
  • VTC-R1 Reasoning Paradigm: 3–4 is the approximate compression ratio achieved by representing prior textual reasoning as rendered images under the stated rendering configuration.The images provide a compact representation through vision tokens.
  • VTC-R1 Reasoning Paradigm: No additional training, extra sampling stages, or external models are introduced for the rendering mechanism.The method therefore relies on a lightweight rendering process.
  • Training Data Construction: 3.4× compression reduces 181M original reasoning-trace text tokens to 54M vision tokens in the supervised fine-tuning dataset.The training corpus contains 61K question–answer pairs and uses approximately 105K rendered PNG images across 106K instances.

5. Experiments

Experiments compare VTC-R1 with long-context SFT and TokenSkip across mathematical and out-of-distribution benchmarks using accuracy, token count, and latency. VTC-R1 improves accuracy, supports multi-iteration reasoning, and reduces inference cost across Glyph and Qwen3-VL architectures.

  • Experimental Setup: Training uses OpenR1-Math-Inf, a subset of OpenR1-Math-220K, with 2K, 4K, and 6K-token reasoning segments; 4K is the default.Evaluation covers GSM8K, MATH500, AIME25, AMC23, and GPQA-Diamond.
  • Experimental Setup: VTC-R1 is compared with Base SFT, SFT, and TokenSkip on Glyph and Qwen3-VL-8B using accuracy, average generated tokens, and inference latency.Accuracy uses pass@1 for GSM8K, MATH500, and GPQA-Diamond, and avg@16 for AIME25 and AMC23.
  • Performance Gains: 5.6% on MATH500 and 3.4% on AMC23 are Glyph gains over the baselines, while Qwen3-VL shows consistent improvements or competitive accuracy.VTC-R1 consistently outperforms Base SFT, SFT, and TokenSkip across all four Glyph benchmarks.
  • Performance Gains: 7.6% and 11.1% accuracy improvements on GPQA-Diamond indicate that VTC-R1 generalizes beyond in-distribution mathematical benchmarks.The reported out-of-distribution results follow similar trends to the main evaluation.
  • Efficient Inference: At least 1.4× speedup is observed across Glyph benchmarks, reaching 1.7× and 1.6× on challenging benchmarks, while Qwen3-VL reaches up to 6.6×.For Glyph AMC23, approximately 1.3× token reduction accompanies 1.6× latency improvement, exceeding token-count savings.
  • Iteration Epochs: Accuracy consistently improves as the maximum iteration epoch increases, eventually surpassing the 8,192-token long-context baseline across benchmarks.Different problem difficulties produce different effective token lengths, with GSM8K using fewer tokens and AIME25 using longer sequences and more epochs.
  • Ablation and Qualitative Analysis: Removing image inputs drops accuracy by 11.1% on AIME25, 7.5% on AMC23, and 25.4% on GPQA-D, supporting rendered images as memory for prior reasoning.Accuracy remains partly intact because many problems can be solved within a single reasoning iteration.

6. Conclusion … A.2. Rendering Example

VTC-R1 integrates vision-text compression into iterative reasoning by rendering prior reasoning segments as compact visual representations. The paper concludes that this model-free approach improves benchmark reasoning accuracy while achieving 3.4× token compression and 2.7× end-to-end inference speedup.

  • 6. Conclusion: VTC-R1 replaces lengthy textual reasoning contexts with compact visual representations that are iteratively reused as vision tokens.This provides a lightweight, model-free compression mechanism for long-context reasoning.
  • 6. Conclusion: 3.4× token compression is achieved while preserving the iterative reasoning paradigm.The conclusion reports up to 3.4× compression from replacing long textual contexts with fewer vision tokens.
  • 6. Conclusion: 2.7× end-to-end inference speedup accompanies consistently improved reasoning accuracy across multiple benchmarks.The reported efficiency and accuracy gains support VTC-R1 as an efficient long-context reasoning approach.
  • A. Image Rendering: The image-rendering component determines how previous reasoning segments are converted into visual representations for subsequent reasoning.The rendering pipeline is described as part of the vision-text compression mechanism.
  • A.1. Rendering Configuration: Rendering configuration factors control the final layout, visual clarity, and typography of generated images.These factors are organized in a configuration vector and summarized with their sampling strategies in Table 7.
  • A.1. Rendering Configuration: The implementation follows Glyph’s default settings but replaces its font with DejaVuSans.ttf for mathematical-symbol rendering reliability.The replacement addresses incorrect glyphs produced by the default Glyph font for certain mathematical symbols.
  • A.2. Rendering Example: Figure 6 shows an image rendered using the default configuration specified in Figure 5.This example illustrates the resulting rendering under the experimental default settings.

B. Details about Experiments · B.1. Implementation Details.

The experiments use standardized supervised fine-tuning and controlled inference settings across model architectures, with TokenSkip evaluated at a fixed compression ratio and VTC-R1 using bounded iterative generation. The implementation also specifies rendering defaults and provides a rendered-page example.

  • B.1. Implementation Details.: Supervised fine-tuning uses a 1 × 10−5 learning rate, 0.1 warmup ratio, cosine schedule, one epoch, batch size 64, and 32,768-token sequences.Training uses 8 NVIDIA H20 GPUs with 96 GB of memory.
  • B.1. Implementation Details.: TokenSkip supports compression ratios from 0.6 to 0.9, but experiments use 0.8 because 0.6 causes unstable, collapsing training.The official TokenSkip implementation is adopted.
  • B.1. Implementation Details.: Evaluation runs on one NVIDIA H20 GPU with 96 GB memory using vLLM, temperature 0.6, and top-p 0.95.Standard SFT allows 32,768 max new tokens, while VTC-R1 allows 8,192 tokens per iteration for up to 8 iterations.
  • B.1. Implementation Details.: The rendering configuration uses 595 × 842 dpi pages, 72 dpi, 10-unit horizontal and vertical margins, DejaVuSans.ttf, and 9-point text.Additional defaults include 10-unit line height, left alignment, and automatic cropping.
  • B.1. Implementation Details.: Figure 5 documents the default rendering configuration used in the experiments.Its caption identifies the figure as the experiment-wide rendering default.
  • B.1. Implementation Details.: Figure 6 provides an example of a rendered page produced under the experimental rendering setup.The passage identifies it specifically as an example rendered page.

B.2. Benchmark

The evaluation covers multi-step grade-school math, competition-level mathematics, and graduate-level scientific reasoning, including an out-of-distribution benchmark.

  • Benchmark: GSM8K contains 8,500 grade school math word problems, with a canonical test set of 1,319 problems.It is used for multi-step reasoning evaluation.
  • Benchmark: MATH500 comprises 500 problems from high school math competitions.It evaluates challenging mathematical reasoning.
  • Benchmark: AIME25 consists of 30 challenging problems from the 2025 American Invitational Mathematics Examination.The dataset targets competition-level mathematical problem solving.
  • Benchmark: AMC23 comprises 50 problems from the 2023 American Mathematics Competitions.It serves as a benchmark for competition-level mathematical reasoning.
  • Benchmark: GPQA-Diamond contains 198 complex graduate-level multiple-choice questions across scientific domains and serves as the out-of-distribution benchmark.It is a high-quality subset of the GPQA benchmark.

B.3. Training Dataset · B.4. Batch Inference

VTC-R1 builds its training set by cleaning and segmenting verified mathematical reasoning traces, then performs iterative batch inference that removes finished requests while rendering ongoing reasoning into image history. The pipeline supports structured answer extraction and returns a final answer for each question.

  • B.3. Training Dataset: 220k math problems with two to four verified DeepSeek R1 reasoning traces form the source dataset for OpenR1-Math-Inf.The problems originate from NuminaMath 1.5, and all traces were verified using Math Verify.
  • B.3. Training Dataset: 60,688 valid instances yield 106K training instances and approximately 105K rendered images after cleaning and segmenting traces by η.η controls the maximum token length of each reasoning segment.
  • B.3. Training Dataset: Final answers use <answer> A </answer>, while later reasoning steps use <think>Got it, let’s continue. {step text}</think>.The answer tags facilitate extraction and explicitly indicate reasoning termination; the intermediate format applies when step > 1.
  • B.4. Batch Inference: Batch inference initializes an active request set, current image sets from initial images, and empty final-answer storage for a batch of questions.The algorithm accepts questions Q = {Q1, . . . , QB}, a vision-language model M, system prompt ⟨Sv⟩, rendering operator Rθ, and maximum iteration T.
  • B.4. Batch Inference: Each iteration constructs batch prompts from the system prompt, question, and current images, then obtains batch outputs via vLLM.Only requests remaining in the active set are included in batch generation.
  • B.4. Batch Inference: Finished requests have answers extracted and are removed from the active set; unfinished requests have reasoning progress rendered into new images appended to image history.This update repeats across iterations while active requests remain.
  • B.4. Batch Inference: The procedure extracts final answers and outputs A = {A1, . . . , AB} for the entire batch.A final extraction step handles the remaining outputs before returning the answer set.

B.5. Case Study

The case study shows how image-conditioned reasoning supports verification, summarization, error correction, and reflection across successive reasoning segments. Gray-shaded regions mark steps conditioned on images rendered from prior segments.

  • B.5. Case Study: Image-conditioned steps are shown in gray-shaded regions, using rendered images of previous reasoning segments as conditioning context.The case study presents four examples of this process.
  • B.5. Case Study: Example 1 demonstrates further verification of a previously obtained solution.
  • B.5. Case Study: Example 2 derives the final answer by summarizing completed prior reasoning.
  • B.5. Case Study: Example 3 performs error correction and reflection based on contradictions identified in earlier reasoning, eventually reaching the correct answer.
  • B.5. Case Study: Example 4 continues the reasoning process after the preceding examples.The supplied passage truncates the description of Example 4.
Loading 2601.22069v2…