PaperlayerView exact revision on arXiv

Source-linked AI summary

Handwritten Text Recognition Lives in the High-Pixel Variance Subspace

Carlos Garrido-Munoz, Jorge Calvo-Zaragoza

arXiv:2609.35473v1cs.CVcs.LG

TL;DR

HTR SSL lacks a clean, principled comparison of which objective works best and why. The paper compares six methods under matched conditions and tests whether preserving high-variance pixel content matters. Pixel-grounded methods lead across HTR measures, consistent with the proposed variance-geometry account.

  • Problem

    Existing HTR SSL work does not establish which method family performs best or provide a principled account, while comparisons vary in data and evaluation protocols.

    Source: §1 · p. 2

  • Method

    The paper evaluates six SSL methods across three families under matched encoder, data, and evaluation protocols, and tests a high-variance pixel-subspace account of HTR.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    Pixel-grounded SSL wins across HTR measures and alone benefits from real-data pretraining; full fine-tuning MAE reaches 4.5% mean CER and ranks first or second on every benchmark.

    Source: §5 · p. 6, §5.4 · p. 8, §6 · p. 10

  • Takeaways & Limitations

    The value of pixel reconstruction depends on where discriminative signal lies in the input; in HTR, it lies in the high-variance pixel subspace.

    Source: §6 · p. 10

    Experiments use one ViT-B scale and one seed, and cover only Latin-script benchmarks; non-Latin scripts and other transcription tasks remain untested.

    Source: §6 · p. 10

Abstract

from arXiv · show

In self-supervised pretraining for Handwritten Text Recognition (HTR), pixel reconstruction methods outperform contrastive methods, unlike in natural-image classification. We argue that this difference follows from where discriminative signal lies in pixel space: for HTR, it is concentrated in high-variance directions and largely absent from low-variance ones. This predicts that objectives preserving high-variance pixel content will transfer best. We test six SSL methods from three families (pixel-grounded MIM, JEPA, and contrastive) under matched encoder, data, and evaluation protocols on six handwriting benchmarks across five languages. With full labels, pixel-groundrounded SSL achieves the lowest CER on every benchmark and both frozen probes, exposes per-position character information that other families recover only through the readout, and is the only family to benefit from pretraining on real handwriting. Pixel-grounded representations are also more label efficient. Across datasets, encoder alignment with the high-variance pixel subspace predicts CER within every method. With a pretrained LLM decoder, a frozen pixel-grounded encoder is competitive with fully fine-tuned supervised baselines; full fine-tuning achieves the lowest mean CER and ranks first or second on every benchmark. These results show that the value of pixel reconstruction depends on where discriminative signal lies in the input.

1 Introduction

HTR remains dependent on costly real-handwriting labels, while synthetic data does not close the synthetic-to-real gap without real samples. The paper tests whether pixel-level detail, often treated as unhelpful for perception, is essential for transcription.

1 Introduction

  • Labeled real handwriting is costly and scarce, and synthetic-only training leaves a synthetic-to-real gap that persists without real samples.Handwriting variability also makes systems generalize poorly across writers and historical periods.Source: §1 · p. 1
  • The paper asks which SSL family best serves HTR and why, under matched comparisons, challenging the assumption that pixel reconstruction is unhelpful for perception.It argues that transcription depends on character-level detail that JEPA-style methods discard.Source: §1 · p. 2
  • Pixel-grounded methods outperform JEPA-style and contrastive methods across six benchmarks, while encoder alignment with the high-variance pixel subspace correlates with downstream CER.The introduction links this result to the claim that transcription signal lies in low-level pixel detail.Source: §1 · p. 2
  • The authors evaluate six methods across three SSL families under matched encoder, data, and evaluation protocols, including the first JEPA adaptation to text recognition.The study spans synthetic and real pretraining regimes and six benchmarks.Source: §1 · p. 2

2 Related Work

Related work traces HTR's dependence on labeled corpora and reviews competing self-supervised objectives. Prior text-recognition work leaves JEPA-style methods untested, motivating a matched comparison grounded in handwriting's variance structure.

2.1 Handwritten Text Recognition.

  • Transformer-based HTR still depends heavily on large labeled corpora, the bottleneck this paper addresses.Earlier systems include recurrent CTC, attention-based encoder-decoder, and Transformer approaches.Source: §2.1 · p. 2

2.2 Self-supervised pretraining

  • MIM reconstructs image content, JEPA predicts masked-region representations, and contrastive learning optimizes instance discrimination; their relative merits depend on the task.The paper argues that high-variance pixel directions contain HTR signal, opposite to the account for natural-image classification.Source: §2.2 · p. 3

2.3 Self-supervised pretraining for text recognition

  • Prior text-recognition SSL includes pixel-grounded and contrastive approaches, but no published work had adapted JEPA-style or other embedding-prediction objectives to text recognition.This study supplies a matched six-method comparison and the first study of JEPA-style methods on HTR.Source: §2.3 · p. 3

3 HTR signal lives in the top-variance subspace

The section argues that handwriting's discriminative signal lies in high-variance pixel directions, unlike natural-image classification. Projection probes and reconstructions support this account, which motivates testing SSL methods that preserve high-variance content.

3 HTR signal lives in the top-variance subspace

  • Handwriting's stroke configurations make high-variance directions discriminative, reversing the natural-image classification account that places signal in low-variance directions.The reported asymmetry spans six benchmarks, five languages, and several historical periods.Source: §3 · pp. 3–4
  • The projection-then-probe test fits pixel PCA on training images, reconstructs with matched-variance top-K and bot-K components, then evaluates identical BiLSTM–CTC probes.Top-K uses the fewest high-variance components for a variance budget; bot-K uses low-variance components to match it.Source: §3 · pp. 3–4
  • The structural account predicts better HTR transfer for SSL features that preserve high-variance pixel content, a prediction tested across six methods from three families.The paper uses this prediction to motivate its method comparisons.Source: §3 · p. 4
  • Top-variance projections approach the full-image probe baseline above a small variance threshold, while bottom-variance projections remain near ceiling across six benchmarks.Figure 1 plots CTC probe CER against the fraction of pixel principal components retained.Source: Figure 1 · p. 4, §3 · p. 4
  • Only top-K reconstructions preserve legible text when top- and bottom-variance PCA reconstructions use the same 80% variance budget.Figure 2 compares the two reconstructions per dataset.Source: Figure 2 · p. 4

4 Methods

The study compares six SSL methods across three families under two pretraining regimes and three evaluation protocols. A shared ViT-B encoder, benchmark data, and decoder setup support matched comparisons of representations and recognition.

4 Methods

  • Six SSL methods from three families are compared under two pretraining regimes and three evaluation protocols, with canonical instances chosen to characterize objectives rather than text-recognition-specific recipes.The study uses two canonical instances per family.Source: §4 · p. 4, §4.1 · p. 4

4.1 SSL families

  • Pixel-grounded MAE and SimMIM reconstruct masked pixels; I-JEPA and V-JEPA-2 predict latent representations, while MoCo-v3 and SigLIP provide image–image and image–text contrastive comparisons.SigLIP is the only compared method with transcription-level pretraining supervision.Source: §4.1 · pp. 4–5

4.2 Evaluation protocols

  • Frozen-encoder Linear–CTC and BiLSTM–CTC probes isolate encoder features, while a pretrained causal LM protocol is evaluated with the encoder frozen or fully fine-tuned.The two visual-only probes use real benchmark training splits and report test CER.Source: §4.2 · p. 5

4.3 Experimental setup

  • All six methods use a shared 113.55M-parameter ViT-B backbone with vertical-strip patch embedding, alongside canonical real benchmarks and a 12.5M-line synthetic corpus.The real regime uses the union of six benchmark training splits; the synthetic regime uses 2.5M lines per language across five languages.Source: §4.3 · pp. 5–6

5 Results

Across five complementary experiments, pixel-grounded SSL is strongest by every measure tested; the section examines encoder quality, per-position structure, alignment, supervised-system performance, and label efficiency.

5.1 Pixel-grounded SSL produces the most readable encoder

  • 5.5% mean CER for MAE and 8.7% for SimMIM lead real-pretrained frozen probes, with only pixel-grounded methods improving on real data.MAE improves by −2.9 pp and SimMIM by −3.8 pp; V-JEPA-2 (+1.9 pp), SigLIP (+1.7 pp), I-JEPA (+4.6 pp), and MoCo-v3 (+5.2 pp) worsen.Source: §5.1 · p. 6, Figure 3 · p. 7

5.2 Pixel-grounded encoders preserve sequential structure

  • 5.7 pp (MAE) and 10.3 pp (SimMIM) rescue gaps are smaller than JEPA's 26.6 pp and 39.0 pp, and contrastive methods' 29.5 pp and 59.1 pp.Small gaps indicate that the encoder already exposes per-position character information; larger gaps mean the readout must recover it.Source: Table 1 · p. 7, §5.2 · p. 7

5.3 Encoder–subspace alignment explains CER

  • Across all six methods, higher per-dataset R2-gap tracks lower frozen BiLSTM–CTC CER, including for encoders with poor overall HTR performance.Figure 4 plots the per-dataset relationship separately for each SSL method.Source: Figure 4 · p. 8, §5.3 · p. 8

5.4 SSL encoders compete with supervised SOTA HTR

  • 5.1% mean CER for encoder-frozen MAE is below CRNN (5.5%), DTrOCRB (5.6%), and no-SSL (5.7%); full fine-tuning reaches 4.5%.Full fine-tuning MAE is below TrOCRB's 4.7% and ranks first or second in every column; other SSL families gain 1.0–1.1 pp on the mean.Source: §5.4 · pp. 8–9, Table 2 · p. 9

5.5 Pixel-grounded SSL needs fewer labels

  • MAE has the lowest mean CER at every label budget; at 10% labels it reaches 8.8%, below the strongest JEPA or contrastive method's 10.6% even at 100%.The comparison uses frozen encoders and probes trained on nested labeled subsets.Source: Figure 5 · p. 9, §5.5 · p. 10

6 Conclusion

The conclusion links HTR performance to high-variance pixel information and reports strong results for pixel-grounded pretraining. The appendix details the pixel-PCA procedure used to analyze that relationship.

6 Conclusion

  • MAE-pretrained encoders with a pretrained LLM decoder achieve 4.5% mean CER across six benchmarks versus 4.7% for TrOCR-B, ranking first or second on every benchmark.The account predicts similar relevance for scene text, music notation, and mathematical expression recognition, which the paper leaves for future testing.Source: §6 · p. 10
  • HTR’s discriminative signal lies in the high-variance pixel subspace, and pixel-grounded methods alone benefit from SSL pretraining on real handwriting.Encoder alignment with that subspace predicts CER within every tested encoder.Source: §6 · p. 10
  • The study uses one ViT-B scale and one seed per experiment, and tests only Latin-script benchmarks.Transfer to non-Latin scripts and the account’s predictions for other transcription tasks remain untested.Source: §6 · p. 10

Appendix: Contents

A Pixel-PCA construction details

  • Training line images are grayscale, fixed-canvas vectors; the procedure centers them, fits PCA, projects images onto a selected subspace, restores the mean, and clips reconstructions to [0, 1].Fresh one-layer BiLSTM–CTC probes are trained on reconstructed images, and PCA is fit only on training images.Source: A Pixel-PCA construction details · pp. 15–16
  • Pixel-PCA selects top- and bottom-variance subspaces that each retain a target fraction of total pixel variance; the R²-gap averages thresholds p ∈ {0.10, 0.25, 0.50, 0.75, 0.90}.The top subspace uses far fewer components than the bottom subspace at every tested threshold, reflecting handwriting images’ heavy-tailed eigenvalue spectrum.Source: A Pixel-PCA construction details · p. 16

B SSL encoder pretraining

The six SSL encoders share a ViT backbone and a controlled training setup across pretraining regimes. MAE and SimMIM are the pixel-grounded methods, differing in their masked-patch prediction heads and reconstruction losses.

B SSL encoder pretraining

  • All six methods use a shared ViT encoder, image pipeline, and optimizer recipe, with family-specific losses and prediction heads.Images are resized to a 64 × 1024 canvas, and the encoder uses full-height vertical patches of width 4 pixels.Source: B SSL encoder pretraining · p. 16
  • Pretraining uses either balanced real handwriting or ten million synthetic lines across five languages, with a shared 500-epoch optimizer budget and training-time augmentation.Checkpoints are selected using an inline IAM CTC probe; downstream evaluation uses separate per-dataset probes and the vision-language pipeline.Source: B SSL encoder pretraining · pp. 16–17

B.1 MAE.

  • MAE predicts raw pixel intensities for masked patches using a four-block transformer decoder and mean squared error.Random masking leaves 25% of patches visible to the encoder.Source: §B.1 · p. 17

B.2 SimMIM.

  • SimMIM predicts masked-patch pixel intensities with a single linear projection and an L1 reconstruction loss.Unlike MAE, it keeps learnable mask tokens at the encoder input and uses a 60% masking rate.Source: §B.2 · p. 17

B.3 I-JEPA (and JEPA adaptation to HTR).

The paper adapts JEPA methods to left-to-right handwriting lines and describes the contrastive alternatives in the same pretraining setup. The methods differ in their targets, masking, and image-text or image-image objectives.

B.3 I-JEPA (and JEPA adaptation to HTR).

  • I-JEPA is adapted from 2D images to 1D line sequences using multiple contiguous target spans, a complementary context, and prediction of masked-patch representations.The adaptation removes the 2D block sampler and global-context crop that do not fit long, single-row lines.Source: §B.3 · pp. 17–18
  • Instance-normalising EMA target features prevents I-JEPA and V-JEPA-2 from collapsing to low-rank, nearly constant patch outputs during training.Without it, collapse occurs on both synthetic and real handwriting; all reported JEPA results use instance normalisation.Source: §B.3 · p. 18

B.4 V-JEPA-2.

  • V-JEPA-2 extends I-JEPA with concatenated targets from the EMA encoder’s last four layers, a deeper predictor, and an auxiliary loss at unmasked context positions.It retains the 1D inputs, four-span masking, instance-normalised targets, and L1 loss.Source: §B.4 · pp. 18–19

B.5 SigLIP.

  • SigLIP pairs each line image with its transcript and trains shared image-text embeddings using per-pair sigmoid cross-entropy.Positive pairs occupy the diagonal of the batch similarity matrix.Source: §B.5 · p. 19

B.6 MoCo-v3 (sequential variant).

  • MoCo-v3 uses matched horizontal windows from two independently augmented views as positives, with other batch windows serving as negatives.Its training objective is symmetrised InfoNCE between online-encoder queries and momentum-encoder keys.Source: §B.6 · p. 19

C Handwritten Text Recognition datasets

The evaluation uses canonical splits from six handwriting benchmarks and synthetic handwriting for pretraining. Frozen-encoder CTC probes range from a linear readout to a shallow sequential readout.

C Handwritten Text Recognition datasets

  • Test data is excluded from PCA fitting, encoder pretraining, probe training, and VLM fine-tuning across the six benchmarks.The benchmarks span five languages and several centuries.Source: C Handwritten Text Recognition datasets · p. 20
  • Synthetic pretraining uses roughly 12.5M lines rendered from Gutenberg text in five languages, with variation in handwriting appearance and layout.The corpus has about 2.5M lines per language and is used only for encoder pretraining and encoder-decoder alignment.Source: C Handwritten Text Recognition datasets · p. 20
  • Figure 6 shows two synthetic handwriting examples per language, rendered as 64 × 1024 grayscale line images at the resolution used for SSL pretraining.Rendering varies font, ink intensity, paper background, slant, baseline jitter, and inter-word spacing.Source: Figure 6 · p. 20

D CTC probes

  • Both frozen-feature CTC probes emit character-level logits per encoder patch using a shared 94-character vocabulary plus a blank symbol.The linear probe has no sequence-modeling capacity; the BiLSTM probe adds one bidirectional layer with hidden size 256.Source: D CTC probes · pp. 20–21
  • The Linear-CTC minus BiLSTM-CTC CER gap estimates how much sequential modeling capacity the readout adds: smaller gaps indicate more text-aligned encoder features.Both probes use the same official splits, image size, greedy character-level CTC decoding, and a single fixed random seed.Source: D CTC probes · p. 21

E Evaluation with limited labels

With limited labels, MAE has the lowest CER in nearly every benchmark-budget setting. The accompanying VLM pipeline combines a frozen SSL encoder with a pretrained language-model decoder, then evaluates encoder-frozen and full fine-tuning stages.

E Evaluation with limited labels

  • The limited-label evaluation freezes each real-handwriting-pretrained encoder and trains the same BiLSTM readout on 1%, 10%, 25%, 50%, or 100% of each benchmark’s labeled examples.Figure 7 reports test CER for each method, benchmark, and label budget.Source: E Evaluation with limited labels · p. 21, Figure 7 · p. 21
  • MAE has the lowest CER in 29 of 30 benchmark-budget settings; I-JEPA is better on Parzival at 1% labels, with 93.8% versus 95.6% CER for MAE.At 10% or more labels, MAE and SimMIM have the two lowest CER values on IAM, Rimes, Bentham, LAM, and Rodrigo; V-JEPA-2 ranks second on Parzival at 25%, 50%, and 100%.Source: E Evaluation with limited labels · p. 21, Figure 7 · p. 21

F VLM pipeline (Stages 1, 2, and 3)

  • The VLM pipeline pairs a frozen SSL encoder’s 256 patch features with a two-layer projector and a pretrained causal language-model decoder.The base encoder has 114M parameters, the projector about 7.3M, and the decoder about 218M.Source: F VLM pipeline (Stages 1, 2, and 3) · p. 22
  • Stage 1 aligns the visual features to the language-model embedding space using a trained projector and synthetic data, before per-dataset real-data tuning.Stage 2 freezes the encoder while unfreezing the decoder and projector; Stage 3 unfreezes the encoder for full fine-tuning.Source: F VLM pipeline (Stages 1, 2, and 3) · p. 22

G Evaluation protocol and baselines

The evaluation reports test CER using validation-selected checkpoints and greedy decoding. Supervised baselines differ in their initialization and pretraining exposure.

G Evaluation protocol and baselines

  • Test CER is reported for the checkpoint with minimum validation CER, using greedy CTC for probes and greedy autoregressive decoding without beam search for the VLM.Evaluation uses raw transcripts without case folding or punctuation stripping.Source: G Evaluation protocol and baselines · p. 22
  • Supervised baselines use differing initializations: TrOCR-B begins from a publicly released checkpoint, while DTrOCR and CRNN are trained from scratch.TrOCR-B was pretrained on roughly 684M synthetic printed-text lines; the paper describes the distinct pretraining exposure for transparency.Source: G Evaluation protocol and baselines · p. 22

H Compute and reproducibility

Experiments use single-GPU bfloat16 training on one RTX 5090. Reported aggregate compute is approximately 2,500 GPU-hours.

H Compute and reproducibility

  • Experiments ran on one RTX 5090 with 32 GB, using single-GPU training and bfloat16 mixed precision.No data-parallel or model-parallel training was used.Source: H Compute and reproducibility · p. 23
  • Total compute was approximately 2,500 GPU-hours across SSL pretraining, CTC probes, the VLM pipeline, and PCA and variance-sweep probes.The respective estimates are about 1,500, 300, 700, and 30 GPU-hours.Source: H Compute and reproducibility · p. 23
Loading 2609.35473v1…