PaperlayerView exact revision on arXiv

Source-linked AI summary

All-in-One Multilingual Scene Text Recognition with Script-aware Mixture-of-Experts

Xingsong Ye, Yongkun Du, Jiaxin Zhang, Zhixian Li, Chong Sun, Chen Li, Jing Lyu, Lianwen Jin, Zhineng Chen

arXiv:2609.24058v1cs.CV

TL;DR

Multilingual STR faces sparse training data for many languages and the challenge of serving diverse scripts in one model. The paper builds balanced synthetic data and proposes ScriptMoE, a script-aware all-in-one recognizer. ScriptMoE performs strongly on TextMuSS-Bench and improves end-to-end CC-OCR results when substituted for PP-OCRv5's recognizer.

  • Problem

    Multilingual STR has sparse training data for many languages and must serve diverse scripts within a single model.

    Source: §1 · pp. 1–2

  • Method

    The paper constructs TextMuSS-10M and proposes ScriptMoE, which shares a visual encoder and routes each image to script-aligned experts plus a shared expert.

    Source: §1 · pp. 2–3

  • Results

    ScriptMoE achieves state-of-the-art average accuracy on TextMuSS-Bench and raises CC-OCR multilingual-task F1 from 65.71% to 80.89%, slightly surpassing the best VLM at 80.73%.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The results support an all-in-one recognizer that is simpler than per-language experts and lighter than VLMs.

    Source: §1 · p. 2

    A domain gap remains between synthetic TextMuSS-10M and real-world imagery, and Latin-Cyrillic homoglyph confusion is not fully solved.

    Source: Limitations and Discussion · p. 22

Abstract

from arXiv · show

Multilingual scene text recognition (STR) remains challenging due to the scarcity of training data for most languages and the difficulty of serving diverse scripts within a single model. Existing solutions either deploy one recognizer per language, inflating cost and introducing error accumulation, or rely on massive vision-language models (VLMs) that are expensive and still inaccurate on many scripts. In this work, we pursue an all-in-one multilingual recognizer that is simpler than per-language experts, lighter than VLMs, and more accurate than both. First, we construct TextMuSS-10M, a large-scale synthetic scene text dataset spanning 10 scripts and 229 languages. It provides balanced and sufficient supervision where real data is unavailable. Second, we propose ScriptMoE, a script-aware Mixture-of-Experts (MoE) architecture. It shares a single visual encoder and replaces the dense decoder with a sparse MoE block, which consists of an image-level router dispatches each image to the top-2 script-aligned experts and a shared expert absorbs cross-script knowledge. Extensive experiments on our assembled TextMuSS-Bench (10 scripts, 10,899 images) show that ScriptMoE achieves the highest accuracy of 82.06%, outperforming the strongest STR baseline by 1.31%. On the CC-OCR end-to-end multilingual task, replacing only the recognizer in PP-OCRv5 with ScriptMoE lifts F1 score from 65.71% to 80.89%, slightly surpassing the best VLM (80.73%) at a fraction of the parameter count.

1 Introduction

Multilingual STR must serve many scripts despite sparse real training data and conflicting script-specific demands on a shared recognizer. The paper pairs balanced synthetic data with ScriptMoE, an all-in-one recognizer that performs strongly across benchmark and end-to-end multilingual OCR evaluations.

1 Introduction

  • Per-language expert systems add training and deployment cost and can accumulate routing errors, while VLMs have parameter and inference costs that hinder edge deployment.The paper seeks an all-in-one recognizer that is simpler than per-language experts and lighter than VLMs.Source: §1 · pp. 1–2
  • TextMuSS-Bench combines the MLT2019 test set for seven scripts with additional Tibetan, Russian, and Thai images, and joint training already surpasses multilingual OCR experts and strong VLMs.The assembled real-image benchmark covers all ten scripts.Source: §1 · pp. 2–3
  • A dense recognizer must share its parameter budget across conflicting script priors, leaving low-resource scripts with too little capacity and common scripts insufficiently specialized.The paper highlights Thai and Tibetan as low-resource examples.Source: §1 · p. 3
  • ScriptMoE shares one visual encoder and uses an image-level router to activate the Top-2 script-aligned experts alongside an always-on shared expert.The shared expert absorbs cross-script knowledge, while script-aligned experts specialize recognition without separate models for each language.Source: §1 · p. 3
  • ScriptMoE achieves state-of-the-art average accuracy on TextMuSS-Bench, outperforming the jointly trained baseline by 1.31%, and raises CC-OCR F1 from 65.71% to 80.89%, slightly exceeding the best VLM at 80.73%.The recognizer-only replacement in PP-OCRv5 also performs on par with VLMs in end-to-end multilingual OCR while remaining lightweight.Source: §1 · p. 3
  • TextMuSS-10M supplies balanced synthetic scene-text supervision across 10 scripts and 229 languages, including scripts with little or no large-scale real STR data.The dataset synthesizes 1M samples per script to provide coverage where real data is unavailable.Source: §1 · pp. 2–3

2 Related Work

Prior multilingual OCR work explores script identification, script-aware recognition, and language-domain routing, while deployed systems and VLMs offer alternatives to unified STR. The paper frames script-aware decoder capacity as a response to multilingual models' limits and trade-offs.

Scene Text Recognition

  • Autoregressive methods lead in accuracy but decode slowly, while nonautoregressive methods decode quickly but lack expressive language priors and can degrade on morphologically rich scripts.Both families were designed mainly for monolingual English or Chinese and are rarely adapted to other scripts.Source: §2 · p. 3
  • The paper keeps an autoregressive backbone but reallocates decoder capacity to script-aware experts, activating only the parameters for the current script.This responds to dense decoders forcing high- and low-resource languages to contend for one parameter set.Source: §2 · p. 4

Multilingual OCR

  • Prior multilingual systems include word-level script routing, script-informed character features, language-domain routing, separate per-script models, and generalist or OCR-specific VLMs.The paper reports that VLM performance remains behind specialized recognizers on one or more scripts.Source: §2 · p. 4

3 Method

The method addresses data and capacity gaps with balanced synthetic supervision and a script-aware sparse decoder, guided by an auxiliary script signal.

3.1 Overview

  • TextMuSS-10M supplies balanced multilingual synthetic supervision, while ScriptMoE replaces dense decoder FFNs with image-level routing and an always-on shared expert.The overview frames data coverage and shared decoder capacity as the two obstacles to an all-in-one recognizer.Source: §3.1 · p. 4

3.2 Balanced Multilingual Data through Synthesis

  • TextMuSS-10M addresses limited multilingual STR data; its synthesis pipeline creates 1M samples per script using language corpora, rendered text, and scene backgrounds.The corpus combines collected words, generated text, and phrases or sentences from newspapers; layouts and visual effects are applied over selected backgrounds.Source: §3.2 · pp. 4–5

3.3 Script-aware Mixture-of-Experts

  • The image-level router sends each image to top-2 script-family experts; all output tokens share that route alongside an always-on shared expert.The ten scripts are grouped into four families: alphabet, CJK, Arabic, and Others.Source: Figure 3 · p. 6, §3.3 · p. 6

3.4 Script-aware Supervision

  • A separate four-way classifier uses Unicode-derived script-group labels to guide specialization without controlling routing; its auxiliary loss is weighted 0.1.The classifier and router share pooled visual input but not weights, leaving the router free to form useful within-script sub-populations.Source: §3.1 · p. 4, §3.4 · pp. 6–7

4 Experiments

Experiments evaluate ScriptMoE on TextMuSS-Bench and the CC-OCR multilingual task, then ablate its training data and model design. ScriptMoE leads STR baselines and improves end-to-end OCR when used as the recognizer in PP-OCRv5.

4.2 Main Results

  • 80.89% F1 on CC-OCR follows replacing only the PP-OCRv5 MLT recognizer, up from 65.71% and above Qwen3.5-9B at 80.73%.The comparison also edges past Qianfan-OCR (76.70%) and GoogleOCR (71.78%).Source: Table 3 · p. 8, §4.2 · pp. 8–9

4.1 Evaluation and Implementation Details

  • Evaluation covers 10,899 TextMuSS-Bench images across ten scripts and CC-OCR; STR baselines share ScriptMoE’s training data and epoch count.For CC-OCR, the PP-OCRv5 detector is fixed and only the recognizer is replaced.Source: §4.1 · p. 8

4.2 Main Results

  • 82.06% Avg on TextMuSS-Bench puts ScriptMoE 1.31% above SVTRv2-AR, with 41.13M of 45.85M parameters activated per image.The gains are concentrated on Arabic (+2.98%), Thai (+2.40%) and Tibetan (+1.96%).Source: §4.2 · p. 8, Figure 4 · p. 9

4.3 Ablation Study

  • Adding TextMuSS-10M raises the seven-script MLT2019 average by 8.40% over real-only training; synthetic-only training exceeds real data by 2.49%.Real-only training collapses on the three low-resource scripts it does not cover; adding synthetic data lowers Latin while adding real data lowers Russian amid Latin–Cyrillic confusion.Source: Table 4 · p. 9, §4.3 · p. 9
  • Four experts achieve the best average of 82.06%; ten experts lose the gain, while Top-2 routing best balances shared patterns and activated parameters.Top-1 misses shared sub-patterns, and Top-4 yields negligible return while activating more parameters; token-level routing is essentially a wash against image-level routing.Source: Table 5 · p. 10, §4.3 · p. 10
  • Latin–Cyrillic confusion remains difficult to close because Latin benefits from abundant real data while Cyrillic relies on synthetic substitutes.Visually identical glyphs can map to different characters across the scripts.Source: §4.3 · pp. 9–10
  • Removing the script-classification signal costs 0.43% average, while removing the shared expert costs 0.71%, especially on Arabic and Tibetan.The shared expert’s removal drops Arabic by 3.62% and Tibetan by 3.65%; the routing case study shows each script group dominated by a distinct expert.Source: §4.2 · p. 9, §4.3 · p. 10, Figure 5 · p. 10

5 Conclusion

The paper studies all-in-one multilingual scene text recognition through dataset construction and model design. It presents TextMuSS-10M and TextMuSS-Bench alongside ScriptMoE, and reports gains on TextMuSS-Bench and CC-OCR.

Data contribution

  • TextMuSS-10M supplies balanced synthetic training signal across 10 scripts and 229 languages, while TextMuSS-Bench provides real evaluation data across ten scripts.The dataset targets scripts for which real training data is scarce.Source: §5 · p. 11

Model contribution

  • ScriptMoE keeps a shared visual encoder and uses an image-level router to activate script-aligned experts, with a shared expert for cross-script transfer.A lightweight script-classification signal guides expert specialization.Source: §5 · p. 11

Results and future directions

  • ScriptMoE achieves competitive average accuracy on TextMuSS-Bench with notable low-resource-script gains, and surpasses expert OCR systems and strong VLMs on CC-OCR.The conclusion identifies continual learning for adding scripts and stronger text detectors as future directions.Source: §5 · p. 11

Data Card of TextMuSS-10M

TextMuSS-10M balances lexical realism with broad character coverage through a fixed mixture of text sources. Its rendering effects and unified character vocabulary support varied scene text synthesis.

Text-source composition

  • Each script’s text strings use a fixed mix: 40% real words, 20% character-shuffled words, 20% rare-character expansion strings, and 20% News Crawl sentences.For Latin, lexicons are sampled across Latin-based languages to balance the intra-script language distribution.Source: Data Card of TextMuSS-10M · p. 15

Rendering effects

  • Curved, multidirectional or vertical, and perspective-distorted layouts are each applied independently with 20% probability and may be stacked.Backgrounds come from SynthText’s 8k background images.Source: Data Card of TextMuSS-10M · p. 15

Character set

  • The unified character set contains 19,684 characters; adding BOS, EOS, and PAD gives the decoder vocabulary 19,687 entries.The unified set merges deduplicated per-script character tables and adds the space symbol.Source: Data Card of TextMuSS-10M · p. 15, Table 6 · p. 15

Collection and Annotation of TextMuSS-Bench

TextMuSS-Bench extends the MLT2019 test split with Russian, Thai, and Tibetan images collected and annotated for the benchmark. The added material uses verified text boxes, expert transcriptions, and a separate quality check.

Benchmark extension

  • TextMuSS-Bench extends the MLT2019 test split with three collected and annotated scripts: Russian, Thai, and Tibetan.The collection and annotation pipeline is specified for these added scripts.Source: Collection and Annotation of… · p. 15

Image collection

Detection and box verification

Transcription annotation

Quality control

  • A separate reviewer checks that each transcription visually matches the image text at the morphological level.The reviewer is not specialized in these low-resource languages.Source: Collection and Annotation of… · p. 15

Full Language-to-Script Mapping

The paper maps 229 languages to ten representative scripts and groups those scripts into four router expert families. Its routing uses sample-level visual features to select experts, alongside a shared expert.

Full Language-to-Script Mapping

  • The ten representative scripts cover 229 languages that share the released character space.The character set contains 19,684 characters, including the space symbol.Source: Full Language-to-Script Mapping · pp. 15, 17, Table 7 · p. 16
  • The router organizes scripts into alphabet, CJK, Arabic-family, and Others expert groups; Greek is folded into the alphabet group.Languages written in more than one script are listed under each script in which they appear.Source: Table 7 · p. 16, Full Language-to-Script Mapping · p. 17
  • The forward pass averages visual features for sample-level routing, selects the Top-2 experts, and also uses a shared expert with a learnable per-token gate.During training, the sample-level router input receives multiplicative Gaussian jitter.Source: Full Language-to-Script Mapping · p. 17

Implementation Details of ScriptMoE

The supplementary implementation description gives the exact network configuration and training objective, with listed values serving as defaults unless otherwise stated.

Implementation Details of ScriptMoE

  • The supplementary implementation details specify the exact network configuration and training objective so the model can be reproduced without ambiguity.Unless stated otherwise, the listed values are the defaults used in all experiments.Source: Implementation Details of ScriptMoE · p. 17

Network Configuration

ScriptMoE uses a shared hierarchical visual encoder and a script-aware Transformer decoder, with supplementary experiments detailing its training, efficiency, evaluation, benchmark performance, and limitations. Reference benchmarks and ablations report results alongside qualitative comparisons and implementation details.

Network Configuration

  • The shared SVTRv2 encoder has three stages with widths [128, 256, 384], while the decoder uses two Transformer blocks with 12 attention heads and script-aware MoE-FFNs.Greedy autoregressive decoding uses maximum lengths of 25 for word-level STR and 100 for the line-level end-to-end variant.Source: Network Configuration · p. 17

Main Training Objective Details

  • The autoregressive objective is token-level cross-entropy over the target transcription conditioned on visual memory, using label-smoothed targets with ε=0.1.Source: Main Training Objective Details · p. 17

Parameter and Efficiency Breakdown

Additional Implementation Details

  • All 15 STR baselines are retrained with identical data, schedule, and hardware; word accuracy requires an exact whole-string match.For the line-level end-to-end variant, the maximum text length is 100, with detection threshold 0.0 and recognition threshold 0.7.Source: Additional Implementation Details · p. 19

VLM Evaluation Protocol

  • Figure 6 marks character mismatches in red and unsupported languages as <Null>; the qualitative comparison says ScriptMoE consistently produces correct results while competing methods often fail.Source: Figure 6 · p. 20, VLM Evaluation Protocol · p. 20

Monolingual Reference Benchmarks

  • 86.85% Avg on BCTR (+0.92% over SVTRv2-AR) and 88.95% Avg on Union14M-Benchmark (+0.28%) show ScriptMoE remains competitive on Chinese and English benchmarks.The authors state these results indicate multilingual gains without trading off high-resource scripts.Source: Table 10 · p. 19, Monolingual Reference Benchmarks · p. 21

Extended Ablations

  • 81.44% and 81.35% averages at script-loss weights 0.2 and 0.5, respectively, accompany the finding that over-weighting the signal can over-constrain routing.The best setting is λscls=0.1; the low-resource Tibetan result falls to 84.55% at λscls=0.5.Source: Extended Ablations · p. 21, Table 11 · p. 22

Limitations and Discussion

  • The paper identifies a synthetic-to-real domain gap, residual Latin-Cyrillic homoglyph confusion, and reliance on a detector whose multilingual quality may bound end-to-end performance.The current end-to-end pipeline reuses the PP-OCRv5 detector.Source: Limitations and Discussion · p. 22

Related papers

Loading 2609.24058v1…