Source-linked AI summary
Glyph: Scaling Context Windows via Visual-Text Compression
Jiale Cheng, Yusen Liu, Xinyu Zhang, Yulin Fei, Wenyi Hong, Ruiliang Lyu, Weihan Wang, Zhe Su, Xiaotao Gu, Xiao Liu, Yushi Bai, Jie Tang, Hongning Wang, Minlie Huang
TL;DR
Long-context LLMs face prohibitive computation and memory costs as inputs reach hundreds of thousands or millions of tokens. Glyph renders long texts into compact images for VLM processing and uses genetic search to optimize rendering, achieving strong compression with competitive accuracy and improved efficiency.
Problem
Scaling LLM context windows to hundreds of thousands or millions of tokens remains costly in computation and memory, limiting practical long-context modeling.
Method
Glyph renders long texts into images for VLM processing, then uses LLM-driven genetic search and targeted post-training to optimize visual-text compression.
Results
Across long-context benchmarks, Glyph achieves 3–4× token compression while maintaining accuracy comparable to Qwen3-8B and improving prefilling, decoding, and SFT efficiency.
Takeaways & Limitations
Visual-text compression offers an alternative route to scaling long-context models and also benefits multimodal tasks such as document understanding.
Takeaways & Limitations
Performance is noticeably affected by rendering configurations such as resolution, font, and spacing, and robustness across settings remains open.
Abstract
from arXiv · showhide
Large language models (LLMs) increasingly rely on long-context modeling for tasks such as document understanding, code analysis, and multi-step reasoning. However, scaling context windows to the million-token level brings prohibitive computational and memory costs, limiting the practicality of long-context LLMs. In this work, we take a different perspective-visual context scaling-to tackle this challenge. Instead of extending token-based sequences, we propose Glyph, a framework that renders long texts into images and processes them with vision-language models (VLMs). This approach substantially compresses textual input while preserving semantic information, and we further design an LLM-driven genetic search to identify optimal visual rendering configurations for balancing accuracy and compression. Through extensive experiments, we demonstrate that our method achieves 3-4x token compression while maintaining accuracy comparable to leading LLMs such as Qwen3-8B on various long-context benchmarks. This compression also leads to around 4x faster prefilling and decoding, and approximately 2x faster SFT training. Furthermore, under extreme compression, a 128K-context VLM could scale to handle 1M-token-level text tasks. In addition, the rendered text data benefits real-world multimodal tasks, such as document understanding. Our code and model are released at https://github.com/thu-coai/Glyph.
1 Introduction
Long-context modeling faces prohibitive costs at very large sequence lengths, so Glyph renders text as compact images for VLM processing and searches rendering configurations to balance compression and accuracy. Experiments report competitive long-context performance alongside substantial compression and efficiency gains.
- 1 Introduction: Scaling context windows to hundreds of thousands or millions of tokens incurs prohibitive computation and memory costs, limiting practical deployment.These costs affect applications including document understanding, code analysis, and multi-hop reasoning.
- 1 Introduction: Glyph renders plain text into compact images so VLMs can process more textual content within a fixed context window without relying on external retrieval.Visual tokens act as compact carriers of multiple textual tokens while preserving semantic information.
- 1 Introduction: An LLM-driven genetic search explores font size, layout, and resolution to maximize compression while preserving downstream performance.The selected configuration is then used for post-training with supervised fine-tuning and reinforcement learning.
- 1 Introduction: Glyph’s framework combines continual pre-training on rendered long-context data with post-training and auxiliary OCR alignment for visualized inputs.The OCR task improves recognition of textual content and aligns visual and textual representations.
- 1 Introduction: 3–4× token compression preserves accuracy comparable to leading LLMs such as Qwen3-8B, with up to 4.8× faster prefilling, 4.4× faster decoding, and about 2× faster SFT training.The reported gains apply to long-context modeling and include both inference and training efficiency.
2 Related Work
Prior long-context work mainly modifies architectures, positional representations, or training methods, while multimodal models provide evidence that visual tokens can represent multiple characters or words. Glyph builds on this opportunity by using visual representations for context compression.
- 2 Related Work: Long-context research has explored sparse or hierarchical attention, positional interpolation or extrapolation, content-aware encodings, and specialized training strategies.Examples include LongAlign for sequences up to 100k tokens and LongLoRA combining shifted sparse attention with parameter-efficient fine-tuning.
- 2 Related Work: Multimodal large language models extend LLMs to jointly process and reason over text and visual inputs, with progress in architectures, language backbones, and vision-language pretraining.Their perception and OCR abilities provide the multimodal foundation relevant to Glyph.
- 2 Related Work: Multimodal models’ OCR capabilities show that multiple characters or words can be represented by a single visual token, motivating visual context compression.This observation supports using vision-language models to increase information density for long-context inputs.
3 Method
Glyph scales long-context understanding by rendering text as compact visual pages for a VLM, then optimizing rendering and post-training to balance compression, readability, and performance.
- Visual Compression: Glyph renders ultra-long text into visual pages so a VLM processes compressed inputs instead of million-token text sequences.Each page contains glyphs from multiple text segments, and training instances become (I, V, R).
- Rendering Pipeline: The rendering configuration controls typography, layout, and visual density, allowing compression and readability to be adjusted for the VLM.Parameters include dpi, page size, font family, font size, line height, alignment, indentation, and spacing.
- Continual Pre-Training: Glyph uses continual pre-training with diverse rendered texts and OCR, interleaved language modeling, and generation tasks to transfer long-context comprehension into the visual modality.The resulting model, Glyph-Base, understands rendered text and handles long contexts.
- Rendering Search: An LLM-driven genetic search evaluates candidate renderings for accuracy and compression, then uses critique, mutation, crossover, and search history to select θ*.The selected configuration is fixed for post-training.
- Post-Training: Post-training combines supervised fine-tuning, reinforcement learning, and OCR alignment to improve reasoning over compressed visual inputs and recognition of textual details.OCR alignment addresses faithful recovery of fine-grained text from rendered images.
4 Experiments
Glyph is evaluated across long-context understanding, compression, scaling, and efficiency, with results compared against leading text-only LLMs. The experiments show competitive accuracy, improved context utilization, and better scaling as sequence lengths increase.
- Glyph achieves performance on par with or surpassing state-of-the-art text-only LLMs of similar size, including Qwen3-8B and GLM-4-9B-Chat-1M.Tables 1 and 2 summarize the overall comparison on long-context tasks.
- 3.3× average effective compression on LongBench and 3.0× on MRCR let Glyph use several times more original context within the same token budget.Some LongBench tasks reach around 5× compression.
- Glyph’s context advantage scales with sequence length: a 3× compression ratio turns a 32K-token window expansion into about 96K tokens of original text.The paper reports faster improvement in usable context as nominal context length increases.
- Higher inference-time rendering resolution substantially improves performance and can surpass strong text-only baselines.This result demonstrates test-time scaling through increased DPI.
Model SP CP UA Acc F1
Additional experiments examine efficiency, multimodal transfer, rendering choices, OCR supervision, and extreme compression. These analyses report increasing efficiency gains with sequence length, improved document understanding, and successful operation at 8× compression.
- Efficiency: Glyph provides speedups in prefill, decoding, and SFT training, with inference speedups growing as sequence length increases from 8K to 128K.The evaluation compares Glyph with its text backbone model.
- Cross-modal generalization: Glyph improves performance over GLM-4.1V-9B-Base on MMLongBench-Doc, supporting generalization to multimodal long-document understanding.The benchmark contains 130 long PDF documents with diverse layouts and embedded images.
- Configuration Search: The searched rendering configuration consistently outperforms randomly sampled and manually designed settings despite comparable compression ratios.The comparison covers average performance and most individual tasks under SFT.
- OCR Auxiliary Tasks: Adding OCR auxiliary objectives during SFT and RL yields consistent gains across benchmarks.The analysis attributes these gains to reinforced low-level text recognition and stronger representations.
- Extreme Compression Exploration: 8× effective compression extends MRCR evaluation from 128K to 1024K while achieving performance on par with GLM-4-9B-Chat-1M and Qwen2.5-1M.This demonstrates performance retention under a more aggressive compression setting.
5 Conclusion
Glyph presents visual compression as an efficient paradigm for scaling long-context modeling. Its framework combines image rendering, genetic configuration search, and post-training to retain competitive performance while improving efficiency and multimodal document understanding.
- Glyph renders long texts into compact images and processes them with vision-language models to achieve 3–4× context compression.The framework combines continual pretraining, LLM-driven rendering search, and targeted post-training.
- The experiments report competitive performance with similar-size leading LLMs, substantial inference-speed and memory-efficiency gains, and cross-modal benefits for document understanding.The conclusion characterizes token information density as a promising direction for long-context scaling.
Limitations and Future Work
Glyph’s current limitations concern rendering sensitivity, OCR fidelity, and the narrow task diversity of its evaluation. Future work targets adaptive rendering, stronger visual-text alignment, broader evaluation, and agentic applications.
- Limitations: Rendering configurations such as resolution, font, and spacing can noticeably affect performance, although the search procedure identifies configurations that work well downstream.Robustness across rendering settings remains open.
- Limitations: UUID recognition remains particularly challenging because rare alphanumeric sequences can be misordered or misclassified by visual models.The authors attribute this to possible training-data sparsity or visual-encoder limitations.
- Limitations: The evaluation mainly covers long-context understanding, while visual-text models generalize less effectively across tasks than textual models.The authors call for broader task evaluation and training to assess robustness and generality.
- Future Work: Future directions include task-conditioned rendering, finer-grained visual text recognition, stronger visual-text alignment, broader applications, and agent memory systems.The paper specifically suggests knowledge distillation or cross-modal supervision for alignment.
A Rendering Parameters
The rendering search selects a best-performing text-to-image configuration, whose parameters and resulting visual output are shown together.
- A Rendering Parameters: Figure 6 presents the optimal rendering configuration identified by the LLM-driven genetic search.The configuration is described as the best result from the search.
B Implementation Details
The implementation combines rendered long-context data, benchmark evaluation, efficiency measurements, and comparisons with similarly sized language models. Rendering factors and benchmark layouts define the experimental settings.
- B Implementation Details: Continual pre-training uses a GLM-4.1V-9B-Base backbone with rendered long-context data and vision-language corpora within a 128k context length.The stated global batch size is 170 and training runs for around 4000 steps.
- B Implementation Details: Post-training includes supervised fine-tuning and reinforcement learning, with SFT using 1.5k steps and GRPO used for reinforcement learning.The SFT learning rate decays from 5e-6 to 2e-6, while RL samples 16 candidate responses per group.
- B Implementation Details: Rendering factors include DPI, page size, font family, layout, and spacing, sampled through mixtures designed to provide broad typography and layout coverage.Table 8 describes these factors as controlling the compression ratio ρ(θ).
- B Implementation Details: Glyph is compared with leading open-source models of similar size, including Qwen3-8B, Qwen2.5-7B-Instruct-1M, LLaMA-3.1-8B-Instruct, and GLM-4-9B-Chat-1M.
- B Implementation Details: Evaluation covers LongBench, MRCR, and Ruler, spanning long-context tasks such as single- and multi-document question answering, summarization, few-shot learning, and synthetic tasks.MRCR uses context-length intervals from 0k–8k through 64k–128k in the described table.
- B Implementation Details: Efficiency is measured using prefill latency and per-sample inference time, while KV-cache testing is omitted because compression translates almost directly into memory savings.The efficiency setup uses an H100 and sets output length to 256 tokens.