PaperlayerView exact revision on arXiv

Source-linked AI summary

See the Text: From Tokenization to Visual Reading

Ling Xing, Rui Yan, Alex Jinpeng Wang, Zechao Li, Jinhui Tang

arXiv:2510.18840v2cs.CVcs.CL

TL;DR

Subword tokenization can miss visual and morphological cues, complicating typo robustness and multilingual coverage. SeeTok encodes rendered text images with pretrained vision encoders; across three NLU tasks, it matches text-tokenization performance while using 4.43× fewer visual tokens and reducing FLOPs by 70.5%.

  • Problem

    Modern LLMs [11], [12], use subword tokenization that misses visual and morphological cues, complicating typo robustness and low-resource multilingual coverage [15],.

    Source: §1 · p. 1

  • Method

    SeeTok renders text as images and encodes them with pretrained MLLM vision encoders in place of standard text tokenization.

    Source: §1 · p. 1, §3.1 · p. 3

  • Results

    Across three NLU tasks, SeeTok matches text-tokenization performance while using 4.43× fewer visual tokens and reducing FLOPs by 70.5%.

    Source: §1 · p. 1

  • Takeaways & Limitations

    The results support visual tokenization as a promising general alternative to prevailing subword tokenization.

    Source: §6 · p. 12

    Vision-centric detokenization remains outside this work’s scope because orthographic fidelity, pixel-generation costs, and adaptive layouts pose non-trivial challenges.

    Source: §3.4 · p. 4

Abstract

from arXiv · show

People see text. Humans read by recognizing words as visual objects, including their shapes, layouts, and patterns, before connecting them to meaning, which enables us to handle typos, distorted fonts, and various scripts effectively. Modern large language models (LLMs), however, rely on subword tokenization, fragmenting text into pieces from a fixed vocabulary. While effective for high-resource languages, this approach over-segments low-resource languages, yielding long, linguistically meaningless sequences and inflating computation. In this work, we challenge this entrenched paradigm and move toward a vision-centric alternative. Our method, SeeTok, renders text as images (visual-text) and leverages pretrained multimodal LLMs to interpret them, reusing strong OCR and text-vision alignment abilities learned from large-scale multimodal training. Across three different language tasks, SeeTok matches or surpasses subword tokenizers while requiring 4.43 times fewer tokens and reducing FLOPs by 70.5%, with additional gains in cross-lingual generalization, robustness to typographic noise, and linguistic hierarchy. SeeTok signals a shift from symbolic tokenization to human-like visual reading, and takes a step toward more natural and cognitively inspired language models.

1 INTRODUCTION

SeeTok replaces subword tokenization with visual-text processing, drawing on pretrained multimodal models to interpret rendered text. It targets the segmentation and perturbation problems of subword tokenization while taking inspiration from human visual reading.

Motivation

  • Humans recognize familiar words from visual shapes and tolerate scrambled or noisy inputs, whereas subword tokenization can be disrupted by typos and minor perturbations and over-segment low-resource languages.The paper links human visual word recognition to the Visual Word Form Area and describes tokenization’s vocabulary coverage trade-off in multilingual settings.Source: §1 · p. 1

Method

  • SEETOK renders text as images for pretrained MLLM visual encoders, then uses visual-text instruction tuning with lightweight LoRA so models interpret visual instructions without training from scratch or architectural changes.The encoders reuse OCR ability and text–vision alignment learned through large-scale vision-language pretraining.Source: §1 · p. 1

Results

  • Across three NLU tasks, SEETOK matches text-tokenization baselines with 4.43× fewer visual tokens and 70.5% fewer FLOPs; in 13-language translation, it yields 86% lower fertility and +3.87 COMET-22.The translation results are relative to the text-tokenization counterpart.Source: §1 · pp. 1–2

Robustness and structural awareness

  • SEETOK shows smaller performance drops than text-tokenization models under character-, word-, and visual-level attacks, and its preserved intra-token structure supports character counting and word unscrambling.The paper also reports improved modeling of character-level composition.Source: §1 · p. 2

Efficiency and cross-lingual generalization

  • Across 14 languages, visual tokenization reduces token counts (4.43× for English; 13.05× for Georgian) and improves translation quality over text tokenization for high- and low-resource languages.The paper attributes the efficiency advantage to a language-agnostic design that avoids text tokenization’s bias toward high-resource languages.Source: §1 · p. 2

Input flexibility

  • Rendering configurations such as font size and resolution allow a cost–performance trade-off, while bolding, highlighting, or italics can emphasize content in ways static token IDs cannot.Source: §1 · p. 2

2 RELATED WORK

The related work contrasts tokenizers’ tradeoffs and low-resource over-segmentation with visual-text approaches that still leave subword tokenization as the primary interface.

2.1 Text Tokenization

  • Character-level tokenizers use small vocabularies but create long sequences [28], word-level methods risk OOV and large multilingual vocabularies, while subword methods can break morphology and remain noise-sensitive.Fixed multilingual vocabularies leave low-resource languages with limited coverage and over-segmented words; the paper proposes treating text as images to promote multilingual fairness and low token fertility.Source: §2.1 · p. 2

2.2 Enhancing Text Understanding through Visual Inputs

  • OCR-free multimodal models can understand text in raw images without transcription,, [47],,, [50],,,, but visual text generally remains auxiliary to subword tokenization, [56],.Unlike those approaches, this method bypasses subword tokenization entirely.Source: §2.2 · p. 3

3 METHODOLOGY

SeeTok replaces discrete input tokenization with rendered text images processed by pretrained multimodal LLMs. Its methodology combines patch-based visual tokenization and vision-centric instruction tuning with conventional text generation.

3.1 Overall Pipeline

  • SeeTok renders input text as images, encodes them with a vision encoder and MLP projector, then feeds visual features to an LLM for reasoning and generation.The pipeline is evaluated with Qwen2.5-VL 3B/7B [11], Llava-next, and JanusPro [21]; Fig. 2 depicts the pipeline and LoRA integration.Source: §3.1 · p. 3, Figure 2 · p. 3

3.2 Visual Text Tokenization

  • Patch-based visual tokenization reduces token sequence length by 4× and fertility by 86% on average across 13 languages, without enlarging a language-specific vocabulary.In the training setup, text is rendered in one RGB image; a 14 × 14 patch encodes roughly 1.1 Qwen tokens in English, and the projector aggregates four neighboring patches.Source: §3.2 · p. 3

3.3 Vision-centric Finetuning

  • Because pretrained MLLMs read visual text but follow visual-text instructions less reliably, SeeTok instruction-tunes LoRA adapters on both the vision encoder and LLM.Instructions are rendered as images while target answers remain text; keeping the projector frozen is essential for stable tuning and preserves pretrained cross-modal alignment.Source: §3.3 · pp. 3–4, §3.1 · p. 3

3.4 Why not Vision-centric Detokenization?

  • SeeTok reduces input token length but retains pretrained text detokenization, preserving linguistic knowledge and architecture while avoiding external OCR at output.Vision-centric text generation is outside this work’s scope because it poses challenges in orthographic fidelity, pixel-generation cost, and adaptive layouts that avoid blank-space padding.Source: §3.4 · p. 4

4 EXPERIMENT · 4.1 Experimental Setup

The experimental setup uses OpenHermes 2.5 for instruction tuning and evaluates SEETOK across 13 languages and four dimensions, from general reasoning to fine-grained lexical perception.

4.1 Experimental Setup

  • The instruction-tuning corpus is OpenHermes 2.5, filtered to 658k instances after excluding excessively long samples to avoid out-of-memory issues.Source: §4.1 · p. 4
  • SEETOK’s multilingual evaluation covers 13 languages: five high-resource and eight low-resource languages spanning Turkic, South Caucasian, Baltic, Slavic, and Austronesian families.Source: §4.1 · p. 4
  • For high-resource languages, SEETOK is finetuned on ALMA, comprising 58K examples from WMT’17–WMT’20 and Flores-200 [60].Source: §4.1 · p. 4
  • For low-resource languages, training uses the Flores-200 dev set [60], following X-ALMA to ensure high data quality.Source: §4.1 · p. 4
  • General-purpose proficiency is assessed on five representative downstream benchmarks measuring knowledge and reasoning capabilities.Source: §4.1 · p. 4
  • The evaluation also measures multilingual transfer across 13 languages and resilience to character-, word-, and visual-level input noise.Source: §4.1 · p. 4
  • Intra-token structural fidelity is tested through subword composition, character counting, and word unscrambling tasks.Source: §4.1 · p. 4

4.2 Implementation Details

Section 4.2 describes evaluation across several multimodal LLMs, the instruction-tuning and optimization setup, JanusPro image configuration, and benchmark-specific evaluation protocols.

4.2 Implementation Details

  • Evaluation spans Qwen2.5-VL 3B [11], Qwen2.5-VL 7B [11], LLaVA-NeXT 8B, and JanusPro [21].The implementation uses DeepSpeed ZeRO stage-2 and float16 precision to reduce computational overhead.Source: §4.2 · p. 4
  • For instruction tuning, LoRA injects low-rank adapters at rank r = 8, scaling factor α = 32, and 10% dropout; all bias parameters are kept frozen during training.Optimization uses AdamW [63] with a peak learning rate of 2×10−5 and weight decay 0.1; the schedule linearly warms up from 1 × 10−7 over 1,000 steps, then decays exponentially to zero, with global gradient clipping at 1.0.Source: §4.2 · pp. 4–5
  • JanusPro [21] validation uses 384 × 384 images configured as M = 1, H = 16, W = 9216, and C = 3 for vision-encoder compatibility.Source: §4.2 · p. 5
  • Evaluation uses zero-shot MMLU, 5-shot SST5, and Contriever top-k Wikipedia retrieval for TriviaQA, NQ, and PopQA, following CEPE.The retrieved passages are prioritized for the decoder.Source: §4.2 · p. 5

4.3 Performance on Comprehensive Benchmarks

SEETOK applies vision-centric tokenization to benchmark tasks, matching or surpassing text tokenization on average while reducing sequence length and computational costs. Its results vary by task, and it also scales to longer inputs under a fixed memory constraint.

4.3.1 Effectiveness

  • 37.77 versus 37.32 average across five datasets, SEETOK matches or surpasses Qwen2.5-VL 3B, including +1.61 on TriviaQA and +15.60 on SST5.TriviaQA and SST5 rely on surface-form cues; visual-text preserves character-level fidelity, while subword tokenization can fragment or obscure cues in rare words and entities, [23].Source: §4.3.1 · p. 5
  • MMLU remains a gap because it depends heavily on world knowledge from large-scale textual pretraining, which the vision pathway has not received at a comparable scale.The paper suggests visual-text pretraining could narrow this gap and notes a related trend in scaling experiments.Source: §4.3.1 · p. 5

4.3.2 Efficiency

  • 4.43× shorter sequence length, 70.5% lower FLOPs, and 33.5% faster latency are achieved with visual-text input on TriviaQA while maintaining comparable performance.Latency is total wall-clock time from input reception through generation of 64 output tokens.Source: §4.3.2 · p. 5
  • At 50k text tokens, SEETOK reduces TFLOPs 58.0% (308.59 to 129.55) and peak GPU memory 33.9% (23.6 GB to 15.6 GB); under 23.6 GB, it supports 74k tokens, 48% more.At 74k tokens, SEETOK uses 183.97 TFLOPs, below the baseline’s 308.59 TFLOPs at 50k tokens.Source: §4.3.2 · pp. 5–6, Table 3 · p. 5

4.4 Multilingual Translation Evaluation

SeeTok is evaluated on translation from five high-resource and eight low-resource languages into English, with COMET-22 measuring translation quality and Fertility measuring tokens per word. It improves translation quality while representing text with fewer tokens, particularly in low-resource settings.

4.4 Multilingual Translation Evaluation

  • The evaluation covers German, Czech, Icelandic, Chinese, and Russian, plus Kyrgyz, Uzbek, Georgian, Lithuanian, Latvian, Bulgarian, Macedonian, and Malagasy, translating each language into English.COMET-22 (COM) measures translation quality, while Fertility (FET) is the average number of tokens per word; Jieba is used for Chinese segmentation and whitespace splitting for other languages.Source: §4.4 · p. 6

4.4.1 High-resource Languages

  • Finetuned on ALMA for fair comparison, SEETOK achieves 65.17 COM versus 61.45 for Qwen2.5-VL 3B†, and improves visual-text Qwen2.5-VL’s average COM by +31.45 across five languages.Source: §4.4.1 · p. 6

4.4.2 Low-resource Languages

  • 58.12 vs. 54.84 average COMET: SEETOK visual-text finetuning outperforms pure-text finetuning across eight low-resource languages.The models are finetuned using X-ALMA parallel data and evaluated on FLORES; Figure 4 also reports a 7.85× average token compression ratio.Source: §4.4.2 · p. 7, Figure 4 · p. 7

4.4.3 Fertility

  • 0.48 vs. 3.88 average FET in low-resource languages and 0.37 vs. 2.21 in high-resource languages show consistently lower fertility for SEETOK than standard text tokenization.The corresponding average compression ratios are 7.85× for low-resource and 5.71× for high-resource languages; Georgian reaches 8.33 FET under standard tokenization.Source: §4.4.3 · pp. 7–8, §4.4.2 · p. 7

4.5 Perturbation Probing

In zero-shot MMLU probing, vision-centric tokenization is more robust than subword tokenization to character-level, visual, and word-level perturbations. Its advantage grows with perturbation intensity or n-gram granularity, consistent with visual representations limiting changes to local details.

4.5 Perturbation Probing

  • The robustness assessment compares Qwen2.5-VL 3B text tokenization with SEETOK vision tokenization on MMLU in a zero-shot setting across three perturbation types.The types are character-level noise, visually similar glyph substitutions, and word-level corruption.Source: §4.5 · p. 8

4.5.1 Perturbation Implementation Details

  • Character-level noise shuffles within-word n-grams and edits characters; word-level perturbations use synonym substitution, internal reordering, or deletion; visual attacks replace letters with similar glyphs.Across word- and character-level perturbations, vision-centric tokenization also yields higher similarity scores than text tokenization.Source: §4.5.1 · p. 8

4.5.2 Results

  • Across character-level, visual, and word-level noise, Visual-Text Input suffers smaller MMLU accuracy drops than Pure-Text Input, with the gap widening as perturbation intensity or n-gram granularity increases.The paper attributes this to subword sequences shifting sharply after character changes, while vision tokenization represents character shapes and spatial layout, so noise affects mostly local visual details.Source: §4.5.2 · p. 9

4.6 Probing Intra-token Structural Fidelity

SeeTok probes whether visual-text representations preserve word-internal structure that subword tokenization obscures. Across compositionality, character counting, and word unscrambling, the reported results favor vision-centric representations for fine-grained lexical perception.

4.6.1 Subword Compositionality

  • Across English, French, and Russian, vision tokenization achieves cosine similarity close to 1.0 and much smaller angles than text tokenization for composed word embeddings.The results suggest visual representations preserve compositional structure, whereas independent subword units do not explicitly model character-to-word hierarchy.Source: §4.6.1 · pp. 9–10

4.6.2 Character Counting

  • In zero-shot character counting, vision tokenization raises frozen Qwen2.5-VL 3B accuracy by +5.84% (57.99% to 63.83%), while SEETOK reaches 64.98%.SEETOK was not exposed to character-counting tasks or related datasets during training.Source: §4.6.2 · p. 10

4.6.3 Word Unscrambling

  • On zero-shot word unscrambling, vision-centric tokenization improves the frozen Qwen2.5-VL-3B baseline by +0.93%, and SEETOK achieves the best accuracy at 12.50%.Overall accuracy is low across models, underscoring the task’s difficulty; SEETOK was not explicitly trained on word unscrambling.Source: §4.6.3 · p. 10

4.7 Ablation Study

The ablations show that SEETOK improves visual-text instruction following across additional MLLMs and with more training data, while retaining vision-native performance and handling unseen fonts. Performance is strongest when the vision encoder and LLM are adapted while the projector remains frozen.

4.7.1 Extension to More MLLMs

  • On TriviaQA, SEETOK improves visual-text instruction following across JanusPro 1B, Qwen2.5-VL 7B, and Llava-next-8b, recovering or exceeding pure-text performance.Without SEETOK, both backbones show degraded performance on visual-text inputs, which the authors attribute to lacking exposure to such instructions during pretraining.Source: §4.7.1 · p. 11

4.7.2 Scaling Study on Visual-Text Instruction Data

  • MMLU accuracy rises from 32.31% to 52.52% as visual-text instruction data scales from 9k to 658k examples, a 20.21% absolute gain.Performance improves consistently across all benchmarks as the training data increases, including the 145k-example setting.Source: §4.7.2 · p. 11

4.7.3 Effect of Font Type

  • After training with Noto Sans, SEETOK’s performance on unseen Arial and Georgia fonts remains comparable or slightly improves.Evaluation uses the same protocol as training, changing only the font family at inference.Source: §4.7.3 · p. 11

4.7.4 Ablation on Fine-tuning Scope

  • Adapting the vision encoder and LLM while freezing the projector scores 43.53, 5.98 points above the 37.55 all-frozen baseline.Vision-encoder-only adaptation improves the score by 2.57, while adapting the LLM or projector alone reduces performance; the authors attribute the frozen projector’s benefit to preserving pretrained cross-modal alignment.Source: §4.7.4 · p. 11

4.7.5 Preservation of Vision-native Capabilities

  • SEETOK remains on par with the Qwen2.5-VL 3B baseline across VQAv2, DocVQA, and TextVQA, with TextVQA at 80.1% vs. 79.3%.VQAv2 and DocVQA accuracy are nearly identical to the baseline.Source: §4.7.5 · p. 11

5 DISCUSSION

Discussion finds that visual-text instruction tuning gains primarily reflect improved ability to follow visual-text instructions rather than access to new knowledge, and reports stronger pure-text performance and closer alignment between visual-text and pure-text representations.

5.1 Do Visual-Text Instruction Improvements Only Stem from Additional Knowledge from the New Data?

  • Visual-text finetuning improved over baseline by +16.70 versus +0.30 for filtered pure-text finetuning; under equal compute, training on more examples scored 52.52 vs. 49.00.The authors attribute the gains primarily to improved visual-text instruction following, with token efficiency enabling more examples under the same compute constraints.Source: §5.1 · p. 11

5.2 Enhanced Text-Only Performance after Visual-Text Instruction Tuning

  • Across five benchmarks, visual-text instruction tuning improved pure-text inference more than pure-text tuning, while using far fewer input tokens and lower computational cost.The authors interpret this as cross-format generalization: tuning on visual-text inputs also benefits pure-text performance.Source: §5.2 · p. 12

5.3 Layerwise Effect of Instruction Tuning on Cross-Modal Alignment

  • Procrustes analysis on 1k out-of-distribution ALPAGASUS samples found instruction-tuned models’ residuals declined in deeper layers, while frozen models remained high in layers 31–35.Residual norm measures the Frobenius distance between transformed visual-text embeddings and corresponding pure-text embeddings; lower residuals indicate stronger structural similarity.Source: §5.3 · p. 12

6 CONCLUSION

SEETOK replaces conventional text tokenization with rendered images encoded by pretrained vision encoders. It achieves competitive or superior performance while improving multilingual efficiency, compositionality, and robustness to noise; future work will explore vision encoders as a shared interface for multimodal reasoning.

6 CONCLUSION

  • SEETOK replaces conventional text tokenization by encoding rendered images through pretrained vision encoders.The paper presents it as a simple, vision-centric tokenization method.Source: §6 · p. 12
  • SEETOK achieves competitive or superior performance to conventional text tokenization, with advantages in multilingual efficiency, compositionality, and noise robustness.The conclusion describes visual tokenization as a potential general alternative to prevailing subword tokenization.Source: §6 · p. 12
  • Future work will use vision encoders as a unifying interface across modalities toward more general multimodal reasoning.Source: §6 · p. 12
Loading 2510.18840v2…