PaperlayerView exact revision on arXiv

Source-linked AI summary

Exploring In-Context Learning for Handwritten Text Recognition

Eric Ayllon, Abel Gandia, Jorge Calvo-Zaragoza

arXiv:2609.37195v1cs.CV

TL;DR

Historical text recognition can make manuscript contents easier to access and process, but adapting pretrained models for HTR remains challenging. This paper tests parameter-free ICL with pretrained VLMs and reports improvements over baseline in both ID and CD settings, with context size and selection shaping performance.

  • Problem

    Efficient adaptation of pretrained VLMs for HTR remains underexplored, despite domain shift and the costs of adapting large pretrained models.

    Source: §1 · p. 4

  • Method

    The study uses ICL with pretrained VLMs, supplying image–transcription demonstrations as context without updating model parameters.

    Source: §3.3 · p. 7, §6 · p. 12

  • Results

    The method visibly improves over baseline in both ID and CD; larger contexts stabilize performance, while much of the improvement appears with 2 or 4 demonstrations.

    Source: §6 · p. 13

  • Takeaways & Limitations

    Context selection matters, motivating further work on finding optimal demonstration-selection methods.

    Source: §6 · p. 13

    ICFHR16 remains a serious challenge, with considerably higher error ranges in both ID and CD settings.

    Source: §5 · pp. 9–10

Abstract

from arXiv · show

Handwritten Text Recognition (HTR) systems have become an indispensable tool for the digitization of historical documents. Not only do they cut down time and cost, but they also allow democratizing access and processing of their contents by generating their transcripts. However, literature in HTR currently focuses mostly on specialized models that require large amounts of annotated samples to achieve satisfactory performance. We explore the use of In-Context Learning with pre-trained Vision-Language Models (VLMs) to create a transcription pipeline without updating the model's parameters. We then evaluate this pipeline across multiple collections and models, and demonstrate that general-purpose VLMs can be effectively taught how to transcribe handwritten text from images. To assess how our observations may translate to practical applications, we evaluate the performance in a Cross-Domain (CD) scenario, where context examples are drawn from a different collection than the query image. Results in both the controlled In-Domain (ID) scenario and the realistic CD scenario follow the same patterns. First, as context size grows, the error range is expected to narrow towards the average performance. Thus, larger context sizes sacrifice the performance of the oracle-best sampling for lower expected error rates. The results obtained show that, without any parameter updates, this methodology has strong potential to compete with traditional HTR in the presence of domain shift. Moreover, we show and argue that some context samplings work better than others and suggest more effort should be put into finding an ideal sampling method in future work.

Highlights

The paper studies training-free In-Context Learning adaptation for HTR and evaluates it across models, collections, and In-Domain and Cross-Domain settings. The reported results are competitive with HTR literature.

  • ICL provides training-free transfer learning for handwritten text recognition.Source: Highlights · p. 2
  • The evaluation covers multiple models and HTR collections in both In-Domain and Cross-Domain settings.Source: Highlights · p. 2
  • The reported ICL results are competitive with the HTR literature.Source: Highlights · p. 2

1. Introduction

Historical text transcriptions support public access and machine processing, but domain shift and the costs of adapting pretrained models remain challenges. This study tests whether ICL can improve HTR with pretrained VLMs and examines how context choices affect performance.

1. Introduction

  • Historical document transcriptions can support indexing, searching, and processing, while digital images alone do not provide these text-based functions.Source: §1 · p. 3
  • Domain shift can degrade HTR quality, making adaptation before or during inference important to mitigate that effect.Source: §1 · p. 4
  • Adapting large pretrained models can be prohibitively expensive, motivating parameter-efficient alternatives such as ICL.Source: §1 · p. 4
  • The study addresses the underexplored use of efficient VLM adaptation for HTR by testing whether ICL improves transcription quality and examining context size and error ranges.Source: §1 · p. 4
  • The experiments compare multiple pretrained VLMs across HTR corpora, context samplings, and sizes, using ID contexts from the target corpus and CD contexts from other corpora.Source: §1 · p. 4
  • The authors frame their contribution as testing ICL's effect on VLM transcription quality and analyzing the conditions and trends under which it holds.Source: §1 · p. 4

2. Background

The study sits at the intersection of HTR, Vision-Language Modeling, and In-Context Learning, drawing on developments across all three areas.

2.1. Handwritten Text Recognition

  • Modern HTR systems extract visual features and decode sequences, using recurrent networks for lines and transformer decoders for more complex layouts.Source: §2.1 · p. 5

2.2. Vision-Language Models

  • VLMs integrate vision and text, while prior VLM work on HTR examined zero-shot performance rather than adaptation.Source: §2.2 · p. 5

2.3. In-Context Learning

  • ICL uses demonstrations of inputs and expected outputs to improve downstream task performance without explicit task-specific training.Source: §2.3 · p. 5
  • ICL can also be developed as a trainable skill through pretraining changes or fine-tuning, with fine-tuning more widely adopted partly because tailored data are lacking.Source: §2.3 · p. 5

3. Methodology

The methodology combines visual features with language-model decoding and uses in-context examples to guide transcription without updating model parameters.

3. Methodology

  • The methodology formulates language modeling, vision encoding, and ICL as the pipeline components used to enhance transcription without parameter updates.Source: §3 · pp. 5–6

3.1. Language Modeling

  • Language models generate a likely sentence from encoded context, with token embeddings representing text elements.Source: §3.1 · p. 6

3.2. Handwritten Text Recognition and Vision-Language Modeling

  • The HTR pipeline maps images to text by extracting visual features and decoding a sequence; VLMs condition language generation on visual features through cross-attention or a shared multimodal context.Source: §3.2 · pp. 6–7

3.3. In-Context Learning

  • In ICL, image–transcription demonstrations and a query image form the context, whose visual representations are combined with text embeddings for generation.Source: Figure 1 · p. 6, §3.3 · p. 7

4. Experimental setup

The experimental setup evaluates transcription across six varied collections using CER, with contexts drawn either from the target collection or from other collections. It varies context size and sampling to estimate performance under different context selections.

4.1. Corpora

  • Six collections cover English (IAM, Washington), French (Rimes-2011), German (ICFHR2016), Italian (LAM), and Latin (SaintGall) handwriting.The experiments use original data splits, including Washington cross-validation.Source: §4.1 · p. 7, Table 1 · p. 8

4.2. Metrics

  • CER measures Levenshtein edit distance divided by ground-truth length, estimating the effort an expert would need to correct a transcription.Source: §4.2 · p. 7

4.3. Experimental scenarios

  • ID draws contexts from the target collection's training split; CD draws them from other collections' training splits.CD is intended to assess a setting where demonstrations from an unannotated target collection are unlikely to be available.Source: §4.3 · p. 8

4.4. Implementation details

  • Experiments use Hugging Face models (Table 2), context sizes 0, 1, 2, 4, 8, and 16, and Python's vllm module.Each nonzero context size is sampled 30 times; min. CER estimates favorable context selection and max. CER unfavorable selection. The context-size cap followed preliminary marginal transcription-quality improvements.Source: §4.4 · pp. 8–9, Table 2 · p. 8

5. Results

In both ID and CD settings, in-context examples improve transcription over the zero-context baseline, while larger contexts tend to stabilize results without guaranteeing the best-case quality. The section also examines failure modes, model variation, and comparisons with specialized HTR systems.

5. Results

  • In ID, larger contexts generally improve average and worst-case CER, while minimum CER changes little; in CD, transcription improves over baseline but larger contexts trade best-case potential for stability.The reported values summarize global CER across the test set and are averaged across models; individual models differ in performance but follow the same trends.Source: §5 · p. 9
  • ICFHR16 has considerably higher error ranges than other collections in both ID and CD, suggesting this procedure may not perform well regardless of image characteristics or content.The authors identify ICFHR16 as a serious challenge for the methodology.Source: §5 · pp. 9–10

5.1. Qualitative analysis

  • The analysis identifies task failures that replace a transcription with an explanation, language failures that loop during generation, and transcription failures despite following the expected output structure.Task failures can vary with context for the same query image; transcription failure is described as the remaining error type when the context structure is replicated without verbosity or looping.Source: §5.1 · p. 10

5.2. Impact of model choice

  • Qwen3 leads consistently across collections and scenarios except IAM and ICFHR16, where the IAM gap is negligible and both models struggle on ICFHR16.The model comparison averages CER across context sizes greater than zero; ICFHR16 is excluded from the plot because its error range differs substantially from other collections.Source: §5.2 · pp. 10–11, Figure 2 · p. 11

5.3. Comparison to current literature

  • In most collections, average ID performance is similar to HTR literature, while CD transcription remains competitive with HTR-specific systems.The comparison uses Qwen3 and the best reported literature results; the authors identify ICFHR16 as a serious challenge and suggest linguistic divergence may contribute.Source: §5.3 · p. 12, Figure 3 · p. 12

6. Conclusions

The paper evaluates parameter-free ICL adaptation of pre-trained general-purpose VLMs for HTR across six collections and five languages, comparing it with zero-shot transcription. It finds improvements in both ID and CD settings, while noting that outcomes depend on the model, collection, and context selection.

6. Conclusions

  • ICL provides a promising parameter-free HTR adaptation mechanism, with visible improvements over the zero-shot baseline in both ID and CD scenarios.The study evaluates minimum, average, and maximum CER across context samplings and sizes.Source: §6 · pp. 12–13
  • Larger contexts make transcription more stable, but most improvement is already obtained with two or four demonstrations.The authors encourage future work on optimal demonstration selection.Source: §6 · p. 13
  • The framework is a good candidate for low-resource projects where training or fine-tuning specialized systems is not possible.The strength of its benefits depends on the VLM, target collection, and context selection.Source: §6 · p. 13
  • Future work should study language disparity across demonstrations, prompts, models, and target collections, as well as how context affects failure types.Source: §6 · p. 13

CRediT authorship contribution statement

The authors’ contributions span methodology, supervision, formal analysis, writing, and visualization, with roles varying by author.

CRediT authorship contribution statement

A. Complete results

The appendix provides complete ID and CD CER results by collection, model, and context size. Across models, average and maximum CER tend to improve with larger contexts, while minimum CER changes less consistently.

A. Complete results

  • The appendix reports CER for each collection, model, and context size, calculating minimum, average, and maximum edit distance across repetitions before dividing by total ground-truth label length.Tables 5 and 6 provide the per-model results for ID and CD; the appendix includes continuations of both tables.Source: A. Complete results · pp. 15, 17, 19
  • All models follow the summarized trend, but higher-CER models raise the average and can make the method appear worse than it is for some models and datasets.Some model-and-dataset combinations outperform specialized HTR models.Source: A. Complete results · p. 15
  • Average and maximum CER tend to improve as context size grows, while minimum CER stays similar but can worsen in some configurations.Average CER improves less with context size in CD. Tables 5 and 6 report the ID and CD configurations, respectively.Source: A. Complete results · p. 15, Table 5 · p. 16, Table 6 · p. 18
Loading 2609.37195v1…