PaperlayerView exact revision on arXiv

Source-linked AI summary

Seeing the Unseen: Visual Similarity for Pixel Language Model Adaptation

Ran Zhang, Miryam de Lhoneux, Wessel Poelman

arXiv:2608.30541v1cs.CL

TL;DR

Pixel-based language models process rendered text images, but their adaptation to low-resource languages with complex morphology and unseen scripts remains underexplored. Using Tibetan, this paper introduces rendering-level visual-similarity metrics and studies continued pre-training across data scales and scripts. It finds that greater orthographic proximity tends to improve semantic transfer, while adaptation gains depend on the starting model and remain observable with small data amounts.

  • Problem

    The paper addresses limited evidence about fine-grained visual similarity and adapting pixel-based models to low-resource languages written in new scripts.

  • Method

    Using Tibetan as a case study, the paper introduces four rendering-level metrics and evaluates continued pre-training with language, script, and data-scale comparisons.

  • Results

    Higher orthographic proximity tends to yield better semantic transfer, and small-scale continued-pre-training data produces noticeable gains on semantic tasks.

  • Takeaways & Limitations

    Visual script similarity and pre-training starting point are relevant considerations when adapting pixel-based models in low-resource settings.

  • Takeaways & Limitations

    The study focuses on Tibetan, three other low-resource Brahmic-script languages, Khmer, and Chinese, prioritizing analytical depth over language breadth.

Abstract

from arXiv · show

Pixel-based language models (LMs) replace traditional tokenizers by processing rendered images of text, making cross-lingual transfer heavily dependent on the visual and structural properties of writing systems. However, the dynamics of adapting these models to low-resource languages with complex morphology and written in unique scripts are not yet explored. Using Tibetan as a case study, we analyze how continued pre-training of pixel-based LMs is influenced by data scale, initial script exposure, and cross-lingual transfer from languages written in other Brahmic scripts. We introduce four rendering-level metrics to quantify visual script similarity. We evaluate downstream performance across three tasks. Our results show that higher orthographic proximity enhances semantic transfer, even under severe data constraints. Additionally, we find a performance asymmetry based on the pre-training starting point: while multilingual pre-training PIXEL-M4 has stronger initial performance, its capacity for subsequent adaptation seems to be constrained, whereas adapting a monolingual model PIXEL with mixed scripts yields more gains on sentence-level tasks. Our metrics and case study offer empirical observations that could help inform data selection and script adaptation choices when working with pixel-based models in similar low-resource settings.

1 Introduction

Pixel-based language models process rendered text images rather than tokens, motivating study of their adaptation to low-resource languages and unseen scripts. This paper uses Tibetan to examine visual similarity, continued pre-training, and data requirements.

  • Pixel-based models process rendered image patches, avoiding some limitations of token-based multilingual modeling.They are proposed as a tokenizer-free approach for multilingual language modeling.
  • Existing pixel-model research primarily evaluates high-resource languages, making visual-similarity effects difficult to isolate.Prior multilingual language choices were selected with visual diversity or existing script overlap, complicating attribution.
  • Tibetan combines scarce resources, complex agglutinative morphology, and dense syllable-cluster glyphs in a distinctive Brahmic script.Its writing system uses consonant roots with attached superscripts, subscripts, and vowels, separated by tsheg rather than spaces.
  • The paper proposes four patch-level metrics to quantify visual similarity using units that pixel models actually process.The metrics are intended to analyze cross-script behavior and guide language selection for transfer.
  • Continued pre-training is evaluated as a way to adapt pretrained pixel models to Tibetan and other low-resource languages written in new scripts.The study explicitly examines feasibility rather than assuming that existing pixel models already support such adaptation.
  • Data ablations measure how little continued-pre-training data can support meaningful adaptation to an unseen script.

2 Related Work

Related work frames pixel models as tokenizer-free systems whose cross-lingual transfer may depend on visual rather than token-level similarity. Prior studies establish transfer across scripts but leave fine-grained similarity and Tibetan adaptation underexplored.

  • Subword tokenizers can create structural bottlenecks for low-resource and non-Latin languages through fixed vocabularies, limited data, and more severe OOV issues.
  • Pixel models represent text as image-patch sequences, removing the finite vocabulary and offering a tokenizer-free alternative for multilingual modeling.The related models include English-only, multilingual, decoder-only, and broader-coverage variants.
  • Cross-lingual transfer in pixel models is likely influenced by visual similarity because their inputs are visual patches rather than tokens.
  • The paper's visual-similarity setup compares scripts at patch level rather than relying only on broad script-family labels.Figure 1 illustrates four similarity settings and distinguishes a single visual patch from metrics computed over complete input blocks.
  • Existing work often treats script similarity as a binary category, leaving its finer-grained transfer mechanism insufficiently explored.Prior pixel models nevertheless showed cross-script transfer to unseen languages.
  • Tibetan is selected because it combines limited resources and complex morphology with a Brahmic script related to Hindi's Devanagari.This makes it both an under-studied language and a test case for visual-similarity mechanisms.

3 Methodology

The methodology defines patch-level visual-similarity metrics, validates them across languages and scripts, and uses them to select transfer languages for Tibetan continued pre-training. It then standardizes multilingual data and evaluates pixel-based models against reference baselines across three downstream tasks.

  • 3.1 Visual Similarity Metrics: Four metrics characterize orthographic similarity between languages using the rendered patches processed by pixel-based models.The metrics are designed to analyze cross-script behavior and guide language selection for cross-lingual transfer.
  • 3.1 Visual Similarity Metrics: Pixel density measures nonbackground pixel coverage as a proxy for ink coverage and glyph complexity, using τ = 250 as the background threshold.The threshold retains a small margin for anti-aliasing artifacts.
  • 3.1 Visual Similarity Metrics: Average grapheme clusters per block measures human-perceived characters, average code-points per block measures Unicode encoding length, and their ratio reflects glyph compositionality.Higher code-points-to-grapheme ratios indicate more stacking or conjunct forms.
  • 3.1 Visual Similarity Metrics: Across 41 languages in the Parallel Bible Corpus, Khmer is closest to Tibetan and Burmese is farthest among the evaluated languages.Both languages are in the Brahmic family, showing that within-family visual distances can vary substantially.
  • 3.2 Language Selection: The transfer-language sample includes same-script Dzongkha, Devanagari Nepali, and Myanmar Burmese, with Khmer and Chinese added as opposite-distance follow-up cases.Dzongkha represents a slight variation of Tibetan in the same script, whereas Nepali and Burmese use other Brahmic scripts.
  • 3.2 Language Selection: Metric rankings place Dzongkha at the high-similarity end, Nepali in the middle, and Burmese at the low-similarity end relative to Tibetan.Pixel density is 0.194 for Tibetan and 0.191 for Dzongkha, compared with 0.160 for Nepali and 0.138 for Burmese; grapheme clusters are 678.97, 636.83, 502.89, and 378.02 respectively.
  • 3.2 Language Selection: Absolute metric values vary with text domain, although Nepali and Burmese retain the same similarity ranking relative to Tibetan across corpora.Dzongkha cannot be compared across corpora because it is absent from the Parallel Bible Corpus.
  • 3.3 Data: The continued-pretraining data comprise approximately 714k Tibetan sentences, 97k Dzongkha sentences, and 330k sentences each for Nepali and Burmese, with Chinese and Khmer used in a follow-up study.Corpora are normalized, rendered into 16×16 patches, and scaled to comparable rendering-block counts; data scale is measured in rendering blocks.

4 Results and Analysis

Continued pre-training benefits Tibetan semantic tasks, but gains depend on the model’s starting point, auxiliary-script exposure, and data scale. Multilingual PIXEL-M4 starts stronger, whereas monolingual PIXEL often gains more from mixed-script adaptation; POS tagging changes little across conditions.

  • 4.1 Continued Pre-training Results: PIXEL-M4 outperforms PIXEL without CPT on NER (69.08 vs 59.49), POS (70.57 vs 70.46), and Text Classification (44.34 vs 41.73).The advantage may partly reflect Hindi exposure, whose Devanagari script shares orthographic proximity with Tibetan.
  • 4.1 Continued Pre-training Results: Under Tibetan-only CPT, NER improves by +1.11 and Text Classification by +3.38, while POS tagging remains largely unchanged.PIXEL-M4’s absolute gains are generally smaller than PIXEL’s across tasks.
  • 4.1 Continued Pre-training Results: PIXEL-MONO with allmixed CPT reaches 54.57 on Text Classification, above PIXEL-M4’s 48.61, but trails it on NER, 68.26 versus 70.55.The interaction between multilingual pre-training and mixed CPT data differs by task.
  • 4.2 Visual Similarity: On Text Classification, Dzongkha reaches 53.41 versus 50.56 for Tibetan self-CPT, while Nepali reaches 47.67 and Burmese 46.15.NER follows the same ordering: Dzongkha 67.59, Nepali 64.14, and Burmese 63.68; POS varies minimally.
  • 4.2 Visual Similarity: Khmer counters the proximity trend, scoring 60.74 on NER despite being close to Tibetan, below Burmese at 63.68.Chinese scores 50.85, 61.63, and 69.77 on Text Classification, NER, and POS, respectively.
  • 4.3 Data Size Ablation: Text Classification rises from 41.73 without CPT to 52.75 with 35k blocks, with the largest single-step gain—4.1 points—occurring by 1k blocks.The curve has not clearly plateaued at 35k, suggesting further gains may remain possible at the current scale.
  • 4.3 Data Size Ablation: NER increases from 59.49 to 65.59 across no CPT to 35k blocks, while its gains change negligibly after 24k, suggesting near-saturation there.A small drop occurs at 8k before recovery.
  • 4.3 Data Size Ablation: POS tagging increases only 1.1 points, from 70.46 without CPT to 71.53 at 35k, unlike the stronger scale sensitivity of semantic tasks.CANINE-S reaches 77.10 on POS, suggesting character-level models may better exploit Tibetan morphological cues.

5 Conclusion

The study evaluates Tibetan adaptation in pixel-based models using rendering-level visual metrics and continued pretraining. It finds stronger transfer with greater orthographic proximity, gains from small-scale data, and greater CPT gains for pixel-mono than pixel-m4.

  • Four rendering-level metrics quantify visual similarity and support analysis of cross-script transfer and language selection.
  • Higher orthographic proximity between auxiliary languages and Tibetan tends to yield better semantic transfer.
  • Small-scale continued pretraining data produces noticeable gains on semantic tasks during adaptation to new scripts.
  • Pixel-mono benefits more from auxiliary-language pretraining than pixel-m4, revealing a trade-off between initial capability and later CPT gains.

Limitations

The study prioritizes depth over breadth by examining Tibetan and a limited set of related or contrasting script settings. Its scope covers Tibetan, three other low-resource Brahmic-script languages, Khmer, and Chinese.

  • The analysis focuses on Tibetan and three other low-resource languages written in Brahmic-family scripts.
  • The study additionally includes Khmer in a Brahmic script and Chinese in Han script.
  • The authors chose depth instead of breadth in language selection and analysis.

AI Usage

The authors used LLMs for coding assistance, grammar help, and draft review, while retaining responsibility for the verified content.

  • LLMs assisted with coding, grammar, and reviewing a draft.
  • The authors state that all content was verified and reviewed by them.
  • The authors take full responsibility and state that all writing is their own.

A CPT Details and Architecture

The appendix reports data statistics and shared hyperparameter configurations for the continued-pretraining experiments. It identifies the relevant tables and gives an overall computational budget of approximately 170–190 GPU hours on NVIDIA H100 GPUs.

  • Data statistics and hyperparameter configurations are provided for all continued-pretraining experiments.
  • Table 4 provides data sources and sizes for the four languages used in continued pretraining.
  • Table 5 lists hyperparameter settings shared across all continued-pretraining conditions.
  • Approximately 170–190 GPU hours on NVIDIA H100 GPUs comprise the overall computational budget.

B Fine-tuning Details and Architecture

The paper reports fine-tuning dataset, split, metric, and hyperparameter details for three downstream tasks. Models are initialized from each continued-pretraining checkpoint, with checkpoint selection based on validation macro-F1.

  • Fine-tuning setup: Three downstream tasks use documented dataset resources, splits, and evaluation metrics summarized in Table 7.The datasets are TNCC, Tibetan-Mongolian NER, and TDTreebank v1.1.
  • Checkpoint selection: All models are fine-tuned from each CPT checkpoint, and the best checkpoint is selected using validation macro-F1.
  • Evaluation: POS tagging uses adjusted macro-F1 that excludes extremely low-frequency tags, including CONJ, INTJ, SCONJ, and SYM.

C Text Rendering Details

Text is rendered into grayscale 16×16-pixel patches using PangorCairoTextRenderer, with continuous sentence-level rendering for CPT. Task-specific rendering preserves word-to-patch alignment for NER and POS tagging.

  • Rendering pipeline: All corpora are rendered with PangorCairoTextRenderer, Noto Sans and fallback fonts, producing grayscale images with 16×16-pixel patches.
  • CPT rendering: CPT concatenates multiple sentences into rendering blocks containing between 23 and 511 patches.The paper uses continuous sentence-level rendering for all four languages.
  • Fine-tuning rendering: NER and POS render words as lists to enable word-to-patch alignment, while text classification renders sentences continuously.Maximum lengths are 196 patches for NER and 256 patches for both text classification and POS tagging.

D Full Experiment Results

Table 8 reports complete downstream results across all continued-pretraining conditions. The experiments compare pixel-m4 initialization, pixel-mono data-scale ablations, and pixel-mono auxiliary-language adaptation.

  • Experiment results: Table 8 reports downstream results for three tasks across all CPT conditions, with means and standard deviations over three seeds.
  • pixel-m4 initialization: The first experiment group starts continued pre-training from pixel-m4.
  • pixel-mono experiments: The second group tests Tibetan data-scale ablations from pixel-mono, while the third tests auxiliary-language experiments from pixel-mono using 24k rendered blocks per language.

E 41 Language Validation

The 41-language validation tests whether rendering-level metrics recover script-group structure while retaining finer-grained orthographic variation. Results show moderate clustering agreement, strong contributions from GC and CG, and better performance when all four metrics are combined.

  • Validation design: The validation samples 41 languages across six script groups from the Parallel Bible Corpus, using verse-aligned common sets to control corpus-content confounds.The groups are Brahmic, Latin, Cyrillic, Han, Hangul, and Others; preprocessing and rendering follow the paper’s standard pipeline.
  • Observed metric structure: Latin and Cyrillic cluster together, whereas Brahmic scripts are more dispersed; Latin and Cyrillic also show higher grapheme-cluster counts per rendered block.The code-point metric is consistently 1 for both Latin and Cyrillic, partly because of Unicode encoding.
  • Clustering validation: Purity = 0.68, ARI = 0.33, and NMI = 0.55 align Ward clusters with six script groups better than randomization at p < 0.0001.The clustering uses Euclidean distance and Ward linkage with k = 6; B-cubed F1 is 0.581.
  • Interpretation: The moderate rather than near-perfect alignment suggests that the metrics capture finer-grained orthographic similarity beyond script-group labels.
  • Metric ablation: GC (F = 35.12, p < 0.0001) and CG (F = 39.83, p < 0.0001) discriminate significantly among the six groups, unlike PD and CP individually.PD has F = 0.57, p = 0.637, while CP has F = 0.07, p = 0.977.
  • Metric ablation: Combining all four metrics yields the best clustering quality across all three measures, indicating that PD and CP contribute through interactions with other metrics.
Loading 2608.30541v1…