Source-linked AI summary
OmniOCR: Generalist OCR for Ethnic Minority Languages
Bonan Liu, Zeyu Zhang, Bingbing Meng, Han Wang, Hanshuo Zhang, Chengping Wang, Daji Ergu, Ying Cai
TL;DR
OCR for ethnic minority languages remains limited by complex scripts, scarce annotations, and historical variation. OmniOCR adapts a vision-language OCR foundation model with Dynamic LoRA and sparsity regularization, and reports improved accuracy and parameter efficiency across four datasets. The authors also identify limited script coverage and nontrivial training-resource requirements as boundaries of the evaluation.
Problem
Ethnic minority OCR is underexplored because complex writing systems, scarce annotations, and diverse historical and modern forms make low-resource or zero-shot generalization challenging.
Method
OmniOCR uses Dynamic LoRA to allocate adaptation capacity across layers and scripts, while sparsity regularization prunes redundant updates without extra inference cost.
Results
OmniOCR consistently surpasses zero-shot foundation models and conventional post-training baselines across TibetanMNIST, Shui, Ancient Yi, and Dongba, with improved accuracy and lower parameter overhead.
Takeaways & Limitations
The framework is presented as a parameter-efficient OCR solution for heterogeneous minority scripts in low-resource and resource-constrained settings.
Takeaways & Limitations
Evaluation covers only four curated datasets, and training still requires noticeable GPU resources and nontrivial memory usage.
Abstract
from arXiv · showhide
Optical character recognition (OCR) has advanced rapidly with deep learning and multimodal models, yet most methods focus on well-resourced scripts such as Latin and Chinese. Ethnic minority languages remain underexplored due to complex writing systems, scarce annotations, and diverse historical and modern forms, making generalization in low-resource or zero-shot settings challenging. To address these challenges, we present OmniOCR, a universal framework for ethnic minority scripts. OmniOCR introduces Dynamic Low-Rank Adaptation (Dynamic LoRA) to allocate model capacity across layers and scripts, enabling effective adaptation while preserving knowledge.A sparsity regularization prunes redundant updates, ensuring compact and efficient adaptation without extra inference cost. Evaluations on TibetanMNIST, Shui, ancient Yi, and Dongba show that OmniOCR outperforms zero-shot foundation models and standard post training, achieving state-of-the-art accuracy with superior parameter efficiency, and compared with the state-of-the-art baseline models, it improves accuracy by 39%-66% on these four datasets. Code: https://github.com/AIGeeksGroup/OmniOCR.
1. Introduction
OmniOCR addresses the underexplored OCR of ethnic minority scripts, whose complex writing systems, scarce annotations, and historical variation challenge generalization. Its Dynamic LoRA and sparsity regularization support efficient adaptation, with reported gains across four datasets.
- Ethnic minority OCR remains underexplored because complex writing systems, limited annotations, and historical–modern variation challenge conventional methods.
- Existing OCR progressed from handcrafted, script-specific systems to segmentation-free deep learning and multimodal models, but generalization to minority scripts remains difficult.
- OmniOCR is a universal framework that applies Dynamic LoRA to adapt model capacity across layers and scripts.
- Sparsity regularization prunes redundant updates, producing compact adaptation without extra inference cost.
- The paper presents OmniOCR as the first universal OCR framework for heterogeneous ethnic minority scripts.
2. Related Work
OCR research has shifted from task-specific and segmentation-dependent systems toward deep learning and general-purpose multimodal models. However, ethnic minority scripts remain difficult targets for existing approaches.
- Transformer-based OCR models such as TrOCR combined pretrained image and text Transformers for end-to-end recognition.
- Vision-language and large multimodal models provide strong contextual and visual capabilities but still struggle to generalize to ethnic minority scripts.
- Earlier ethnic minority OCR systems relied on handcrafted features and script-specific segmentation strategies for scripts such as Mongolian and Tibetan.
- Segmentation bottlenecks motivated a shift toward segmentation-free CNN-based and attention-based deep learning models.
3. Datasets
OmniOCR is trained and evaluated on four datasets spanning handwritten numerals, ancient characters, and historical or contemporary minority scripts. The datasets provide varied recognition settings, with some experiments using representative subsets.
- The study curates four publicly available datasets covering diverse ethnic minority scripts, historical and contemporary writing systems, and handwritten numerals.
- TibetanMNIST contains 17,768 handwritten Tibetan-numeral images from multiple researchers and supports evaluation across character-morphology and writing-style variation.
- The Shui Dataset contains 5,280 ancient Shui-character images across 12 representative classes, including pictographic cultural symbols.
- The Ancient Yi dataset originally contains over 427,000 samples across 2,922 classes, but the experiment selects 30 classes using frequency and stroke-structure criteria.
- The Dongba dataset originally contains 445,273 images across 1,404 classes and 2,546 variant forms, while the experiment selects 30 classes for feasibility and representativeness.
4. Method
OmniOCR adapts a pretrained OCR backbone to heterogeneous minority scripts using Dynamic LoRA, which allocates update capacity across layers and tasks. Sparsity regularization prunes redundant directions to produce compact adaptation without extra inference cost.
- Framework: OmniOCR builds on a vision-language foundation model to adapt one OCR backbone across multiple low-resource minority scripts.The framework is designed for structural diversity across Tibetan, Ancient Yi, Shui, and Dongba.
- Dynamic LoRA Adaptation: Dynamic LoRA adaptively determines update rank for each layer and task instead of using a fixed-rank update.This avoids underfitting complex scripts and overfitting scarce data.
- Dynamic LoRA Adaptation: Learnable importance weights combine candidate low-rank matrices, allowing more capacity for complex scripts and fewer ranks for simpler ones.Dongba and Ancient Yi are given as complex examples, while Tibetan digits are given as a simpler example.
- Sparse Adaptation: ℓ1 sparsity regularization encourages retention of critical update directions while pruning redundant ones.The supervised loss, number of updated matrices, and sparsity coefficient determine the regularized objective.
- Training Procedure: The training procedure freezes the pretrained backbone, updates only LoRA factors and importance weights, then prunes directions with small weights.The resulting model returns the original backbone with compact LoRA modules.
5. Experiments
OmniOCR is evaluated across four ethnic-minority OCR datasets using standard metrics, baseline comparisons, and ablations of optimization and Dynamic LoRA components. The full configuration delivers strong cross-dataset performance while retaining parameter and inference efficiency.
- Evaluation: Experiments evaluate OmniOCR on four representative datasets spanning heterogeneous scripts and writing systems.The evaluation includes TibetanMNIST, Shui, Ancient Yi, and Dongba.
- Evaluation: The study reports Accuracy, Recall, and F1-score, with Recall and F1-score emphasized for heterogeneous, potentially imbalanced data.Accuracy measures overall correctness, while Recall and F1-score provide complementary assessments.
- Main Results: Zero-shot vision-language foundation models perform poorly, whereas supervised adaptation of RolmOCR produces substantial gains across the four benchmarks.LoRA and full post-training exceed 80% accuracy on Tibetan and Ancient Yi and 90% on Shui and Dongba.
- Main Results: OmniOCR surpasses full RolmOCR post-training on Tibetan, Shui, and Dongba, reaching 90.37%, 95.95%, and 95.32%, respectively.On Ancient Yi, OmniOCR records 89.62% versus 90.53% for full fine-tuning, while maintaining lower memory use and parameter efficiency.
- Ablation Study: A learning rate of 5e-6 with batch size 2 consistently achieves the best accuracy, recall, and F1 score across datasets.Both larger and smaller learning rates, and batch sizes of 1 or 4, perform worse.
- Ablation Study: The complete Dynamic LoRA configuration achieves the best accuracy, recall, and F1 scores across all four datasets.Removing dynamic rank adaptation, MLP adaptation, attention adaptation, or sparsity regularization weakens efficiency, adaptation, alignment, or knowledge retention.
6. Limitation and Future Work
OmniOCR’s evaluation is limited to four curated scripts and benchmark recognition settings, while training still requires substantial computational resources. Future work targets broader script coverage, lighter continual learning, and multimodal robustness.
- Limitations: The four curated datasets do not cover the full diversity of minority writing systems, including decorative glyphs, mixed phonetic–logographic properties, and context-dependent ligatures.Additional scripts and historical documents may present challenges beyond the current evaluation.
- Limitations: Training still requires noticeable GPU resources and nontrivial memory, potentially restricting deployment in resource-constrained environments.This boundary is especially relevant to community-level digitization projects requiring lightweight solutions.
- Limitations: The study emphasizes benchmark recognition accuracy, while practical OCR must also handle degradation, background noise, and complex document layouts.These real-world factors are only partially addressed in the current work.
- Future Work: Future work will broaden script and historical-document coverage, add lightweight continual learning, and explore cross-modal pre-training with speech, text, and visual data.The stated goals are improved adaptability, robustness, and generalization.
7. Conclusion
OmniOCR is presented as a universal OCR framework for heterogeneous ethnic minority scripts, combining adaptive capacity allocation with efficient adaptation. Across four representative datasets, it surpasses zero-shot and conventional post-training baselines, while future work targets broader script coverage and multimodal pre-training.
- OmniOCR is a universal OCR framework tailored to heterogeneous and complex ethnic minority scripts.
- Dynamic LoRA balances knowledge retention and efficient adaptation across layers and writing systems in low-resource scenarios.The design also reduces overfitting risk and maintains parameter efficiency.
- OmniOCR consistently surpasses zero-shot foundation models and conventional post-training baselines on TibetanMNIST, Shui, ancient Yi, and Dongba.The experiments also report lower parameter overhead and memory consumption.
- Future work will expand OmniOCR to more minority scripts and historical writing systems and explore multilingual, cross-modal pre-training.