PaperlayerView exact revision on arXiv

Source-linked AI summary

ExpertHTR: Unified Handwritten Text Recognition with Multi-Task Learning and Sparse Mixture-of-Experts

Dang Hoai Nam, Nguyen Duy Hieu, Quang Huu Hieu, Vo Nguyen Le Duy

arXiv:2609.12705v1cs.CVcs.LG

TL;DR

Small, heterogeneous handwriting collections are difficult to combine for page-level training, and general multimodal pretraining does not ensure reliable handwriting recognition. ExpertHTR derives complementary supervision from existing structural annotations and adds representation-dependent sparse expert capacity. Across seven sources, the unified model improves on most evaluated benchmarks and reaches state-of-the-art IAM paragraph-level performance, though specialized HTR systems remain stronger on several challenging collections.

  • Problem

    Small HTR collections differ in language, script, document structure, and annotation organization, making joint page-level training difficult; general multimodal pretraining also does not ensure reliable handwriting recognition.

    Source: §1 · p. 2

  • Method

    ExpertHTR organizes existing structural annotations in a common Page–Region–Line representation for four complementary tasks, then adds a shared-expert sparse MoE with representation-dependent routing.

    Source: §1 · p. 3, §6 · p. 27

  • Results

    Complementary supervision improves page-transcription-only training across all seven single-source settings; multi-source training and Sparse MoE improve most sources, and the unified model reaches IAM paragraph-level state of the art.

    Source: §1 · p. 4, §6 · p. 27

  • Takeaways & Limitations

    ExpertHTR improves recognition across heterogeneous HTR resources within a single model, rather than replacing specialized recognizers in every setting.

    Source: §5.3 · p. 26

    Sparse MoE has more parameters than Dense MTL, so current ablations do not isolate conditional routing’s contribution from added model capacity.

    Source: §5.3 · p. 26

Abstract

from arXiv · show

Handwritten text recognition resources are often small and distributed across collections that differ in language, script, document structure, and annotation format, making joint page-level training difficult. We propose ExpertHTR, a unified vision-language framework that addresses this problem through complementary supervision and conditional model capacity. Structural annotations from heterogeneous datasets are first organized through a common Page-Region-Line representation and used to construct four related training tasks for complete transcription, physical-line coverage, text localization, and localized recognition, without requiring additional manual labels. Building on a jointly trained dense model, ExpertHTR introduces a sparse Mixture-of-Experts architecture with an always-active shared branch and conditionally routed full-MLP experts. Sparsegen allows the number of active routed experts to vary with the hidden representation, while routing regularization reduces persistent concentration on a small subset of experts. Experiments on seven heterogeneous handwriting benchmarks show that complementary supervision consistently improves training with page transcription alone, while joint multi-source training provides further gains on most datasets. The proposed sparse expert model further improves the dense baseline on six of the seven sources. The final unified model also substantially outperforms the evaluated general-purpose OCR and vision-language systems on most benchmarks and achieves state-of-the-art performance on the IAM paragraph-level benchmark, while specialized HTR systems remain stronger on several challenging collections.

1. Introduction

ExpertHTR addresses the difficulty of combining small, heterogeneous handwriting collections by deriving complementary tasks from their structural annotations and adding representation-dependent expert capacity. Across seven sources, multitask supervision improves page-transcription-only training, and the unified model improves on evaluated general-purpose systems on most benchmarks while reaching IAM paragraph-level state of the art.

1. Introduction

  • Heterogeneous collections vary in language, script, handwriting, document structure, and annotation organization, complicating joint page-level training, while multimodal models vary across languages and are less consistent on historical handwriting.Page-level recognition must cover multiple physical lines, preserve reading order, and handle longer output sequences.Source: §1 · p. 2
  • A common Page–Region–Line representation converts existing annotations into four tasks—Page Transcription, Physical Line Counting, Text-to-Line Localization, and Line-Span Transcription—without additional manual labeling.Page Transcription remains the primary objective; auxiliary tasks target line coverage, content–position correspondence, and localized recognition.Source: §1 · p. 3
  • Selected decoder feed-forward layers become shared-expert sparse MoE layers: an always-active shared branch accompanies routed full-MLP experts, with Sparsegen varying routed support by hidden representation.Routing regularization reduces persistent expert concentration; routing uses hidden representations without explicit language, script, dataset, or task labels.Source: §1 · p. 3
  • The framework combines broader use of existing HTR annotations with conditional capacity that retains a shared computation path.The official implementation and experimental configurations are stated to be made publicly available.Source: §1 · pp. 3–4
  • Multi-source Dense MTL improves on six of seven datasets, and Sparse MoE improves over the dense baseline on six of seven sources; the unified model beats evaluated general-purpose OCR/VLM systems on most benchmarks and reaches state of the art on IAM paragraph-level recognition.Specialized HTR systems remain stronger on several challenging collections.Source: §1 · pp. 3–4

2. Related Work

Related work spans page-level recognition, vision–language adaptation, and sparse expert routing. ExpertHTR draws on these directions while adapting them to unified page-level handwriting recognition across heterogeneous sources.

2.1. Paragraph- and Page-Level Handwritten Text Recognition

  • Page-level HTR includes segmentation-free systems such as DAN, Faster DAN, and Meta-DAN, while staged training approaches address limited data; ExpertHTR instead trains one page-level model across collections using a common representation.DANIEL targets layout recognition, HTR, and named entity recognition, whereas ExpertHTR’s auxiliary tasks support page transcription; line- and paragraph-level systems operate at different input levels.Source: §2.1 · pp. 5–6

2.2. Vision–Language Models and Multitask Adaptation

  • General VLMs do not guarantee reliable handwriting recognition; CHURRO illustrates historical-text adaptation, and Uni-MuMER illustrates complementary-task adaptation for a distinct handwriting-recognition problem.ExpertHTR derives auxiliary tasks from structural annotations and keeps Page Transcription as the final recognition objective.Source: §2.2 · pp. 6–7

2.3. Conditional Capacity with Sparse Mixture-of-Experts

  • Sparse MoEs commonly use fixed Top-k or Top-1 routing, while later methods relax fixed expert cardinality; ExpertHTR uses Sparsegen for representation-dependent routing of complete decoder FFNs alongside an always-active shared FFN.Its router uses only hidden states, without explicit language, script, dataset, or task labels; individual experts have no predefined roles.Source: §2.3 · p. 7, Figure 1 · p. 8

3. Proposed Method

ExpertHTR organizes heterogeneous annotations into complementary page-level supervision and adds representation-dependent conditional capacity to a jointly trained model. Its shared-expert sparse MoE retains a shared computation path while routing additional transformations according to hidden representations.

3. Proposed Method

  • ExpertHTR combines a common Page–Region–Line representation for complementary supervision with shared-expert sparse MoE capacity in selected decoder FFNs.The representation converts existing structural annotations into training targets; the MoE adds conditional capacity while retaining shared computation.Source: §3 · p. 8

3.1. Complementary Multitask Supervision

  • The four tasks supervise complete transcription, physical-line coverage, text-to-line localization, and localized line-span recognition from existing annotations without extra manual labels.Line-span transcription retains the complete page as visual context while restricting the output to a requested contiguous range.Source: §3.1 · pp. 9–10, Figure 2 · p. 9

3.2. Shared-Expert Sparse Mixture-of-Experts

  • Each modified decoder FFN combines an always-active shared expert with routed full-MLP experts, while other decoder FFNs remain dense.The shared and routed outputs are fused; the shared expert is evaluated for every token, and a routed expert only when its weight is positive.Source: §3.2 · pp. 10–12
  • Sparsegen makes routed-expert selection and support size depend on the hidden representation rather than fixing the number of active experts.Routing receives no explicit language, script, dataset, or task identifier as an additional input.Source: §3.2 · pp. 11–12

3.3. Training Objective

  • The autoregressive language-modeling loss trains tasks, routers, and experts jointly, while sparsity and load-balancing terms regulate routed computation.The sparsity term favors sparser routing without fixing support size; load balancing compares routed-expert use with a uniform-utilization reference.Source: §3.3 · pp. 12–13

4. Experiments

Experiments across seven heterogeneous handwriting benchmarks evaluate complementary supervision, joint training, and sparse expert routing. The final Sparse MoE outperforms evaluated general-purpose OCR/VLM systems on six datasets and achieves state-of-the-art performance on IAM paragraph-level recognition, while specialized HTR systems remain stronger on several collections.

4.1. Experimental Setup

  • Experiments evaluate ExpertHTR on seven HTR datasets—BRESSAY, Bentham, HWDB2.0, IAM, READ-2016, RIMES, and ScribbleLens—using CER as the primary metric and WER as a complementary metric; WER is not reported for HWDB2.0.For RIMES, 758 of 5,054 training pages were retained to reduce imbalance in the joint training mixture.Source: §4.1 · pp. 14–15

4.2. Main Recognition Results

  • Sparse MoE outperforms evaluated OCR/VLM checkpoints on six of seven sources, while IAM reaches 2.92% CER and state-of-the-art paragraph-level performance.Bentham is the exception: LightOnOCR obtains 7.24% CER versus 17.57% for Sparse MoE.Source: §4.2 · p. 16, Table 2 · p. 17

4.3. Ablation Studies

  • With matched optimizer updates, multitask supervision improves Page-Transcription-only training on all seven datasets, while multi-source Dense MTL improves six of seven single-source multitask results.The multi-source comparison has unmatched total training exposure and therefore reflects the adopted joint setting, not an isolated estimate of cross-source transfer.Source: §4.3 · p. 19, Table 4 · p. 19
  • On RIMES, the full multitask setting gives the lowest CER and WER, while removing PLC performs worse than Page Transcription alone.Removing LST produces the smallest CER increase; removing T2L increases CER to 7.10%.Source: §4.3 · pp. 19–20, Table 5 · p. 20
  • Regularized Sparsegen improves six of seven datasets over Dense MTL, whereas fixed Top-2 is inconsistent and unregularized Sparsegen improves only four.READ-2016 CER changes slightly in the opposite direction after regularization, from 51.82% to 52.23%.Source: §4.3 · pp. 20–21, Table 6 · p. 20

4.4. Routing Analysis

  • Sparsegen support varies across sources and decoder depth, generally shrinking in deeper layers before changing again near the final MoE layer.Support size is descriptive of routing behavior, not a consistent explanation for recognition gains.Source: §4.4 · pp. 21–22, Figure 4 · p. 22
  • Routing regularization keeps the load-balance statistic closer to 1.0 and spreads assignments more broadly, but routing remains non-uniform and balance does not predict CER directly.READ-2016 becomes less concentrated while its CER changes from 51.82% to 52.23%.Source: Figure 5 · p. 22, §4.4 · p. 23

4.5. Error Analysis

  • Error reductions arise from different components across datasets: fewer insertions and substitutions help READ-2016 and ScribbleLens, while Bentham also reduces substitutions and deletions.On READ-2016, Sparse MoE lowers CER from 80.35% to 52.23%; on Bentham, insertions increase from 8.45% to 9.86% despite lower CER.Source: §4.5 · pp. 23–24, Figure 6 · p. 24

5. Discussion

Complementary supervision improves page-level recognition, while joint training and sparse experts extend the model’s capacity across heterogeneous collections. The results are strong on most benchmarks, but training-exposure differences, added model capacity, and limited comparability constrain what the gains establish.

5.1. Complementary Supervision and Multi-Source Learning

  • Matched-update multitask training improves all seven datasets; on RIMES, the complete configuration performs best on CER and WER, but task contributions cannot be ranked independently.The auxiliary tasks provide focused signals for physical-line coverage, content–position correspondence, and localized recognition.Source: §5.1 · pp. 24–25
  • Joint training improves most datasets, but unmatched total training exposure means the comparison is not a controlled estimate of cross-source transfer.The reported advantage reflects the complete joint-training setting.Source: §5.1 · p. 25

5.2. Shared Computation and Adaptive Routed Capacity

  • The model combines an always-active shared path with routed experts that use hidden representations, not explicit language, script, dataset, or task labels.The experts have no predefined semantic roles.Source: §5.2 · p. 25
  • Sparsegen varies the active routed-expert support across data sources and network depth; routing strategy affects performance, but more balanced routing does not consistently lower CER.Routing regularization reduces persistent concentration on a small subset of experts.Source: §5.2 · pp. 25–26
  • Sparse MoE reduces substitution and insertion errors on several difficult datasets, but the pattern differs across sources rather than following one common error mechanism.Source: §5.2 · p. 26

5.3. Unified Recognition, Trade-offs, and Limitations

  • ExpertHTR outperforms evaluated OCR/VLM checkpoints on most benchmarks and reaches IAM paragraph-level state of the art, while specialized HTR systems remain stronger on several challenging collections.The IAM result uses the same checkpoint trained across all seven sources.Source: §5.3 · p. 26
  • Sparse MoE has more total parameters than Dense MTL, so the routing comparisons do not isolate conditional routing from added capacity; a capacity-matched dense baseline would be stronger control.Source: §5.3 · p. 26
  • Comparisons with specialized HTR systems are benchmark context, not fully controlled, because studies differ in input level, splits, training data, protocols, and checkpoint availability.Comparable page-level resources and published results remain limited.Source: §5.3 · p. 27

6. Conclusion

ExpertHTR unifies page-level HTR across heterogeneous datasets by deriving complementary supervision from existing structural annotations and adding shared-expert sparse conditional capacity. Multitask supervision and joint training improve results, and the sparse model advances performance on most sources while achieving IAM paragraph-level state of the art.

6. Conclusion

  • The multitask formulation improves Page-Transcription-only training, while joint multi-source training provides further gains on most datasets.Source: §6 · p. 27
  • Sparse MoE improves over Dense MTL on most sources, outperforms evaluated OCR/VLM checkpoints on most benchmarks, and achieves IAM paragraph-level state of the art; specialized HTR remains stronger on several challenging datasets.Sparsegen varies routed expert support with hidden representations, while routing regularization reduces persistent expert concentration.Source: §6 · p. 27

Related papers

Loading 2609.12705v1…