Source-linked AI summary
Measuring Annotation Efficiency for Handwritten Devanagari Recognition: Sample-Complexity Curves for Four Pretraining Regimes
Manglesh Kumar Pandey, Sumit Kumar Banshal
TL;DR
Handwritten Devanagari recognition requires manual transcriptions, motivating the question of how much annotation is needed and how much pretraining can remove. The paper measures sample-complexity curves across matched budgets and initialisation regimes, finding substantial savings at CER 0.50 that diminish at stricter accuracy targets.
Problem
The paper asks how much transcription handwritten Devanagari recognition needs before becoming useful and how much of that cost pretraining can avoid.
Method
The study varies real fine-tuning labels from 10 to 4,000 across four initialisation regimes and converts the resulting curves into annotation-equivalent terms.
Results
At CER 0.50, supervised synthetic pretraining requires 81 words versus 355 for random initialisation, a 4.40× label multiplier [3.56, 4.99].
Takeaways & Limitations
Pretraining is worth approximately 136 transcribed words at zero shot, but its annotation saving decreases with stricter targets and is indistinguishable from zero at the most demanding measured target.
Takeaways & Limitations
The scarcity is constructed by subsampling a 95,430-word corpus, so the derived multipliers apply to this corpus under subsampling rather than to genuinely under-resourced scripts.
Abstract
from arXiv · showhide
To train handwritten text recognition systems we need word images and their corresponding transcriptions, and these transcriptions are produced manually. For a script that can be read by only a small number of specialists, this manual transcription is a limitation, because the trained models are supposed to save the time of those same specialists. A relevant question therefore arises: how many transcriptions are needed before a recogniser becomes useful, and how much of that cost can pretraining remove? In this study the answer is measured directly for handwritten Devanagari. We keep the recogniser, optimiser and evaluation protocol the same and change only the number of real transcribed words used for fine-tuning across nine budgets from 10 to 4,000 and four initialisation regimes, with six seeds at every point. The resulting curves are then converted into annotation-equivalent terms. A CER of 0.50 is reached by supervised synthetic pretraining using only 81 transcribed words, whereas random initialisation requires 355, which gives a label multiplier of 4.40 [3.56, 4.99]. There is a zero-shot reference point as well: with no real transcribed words at all, this pretraining is worth about 136 of them. This advantage gets smaller as the target accuracy improves, and at the most demanding target we measure, it cannot be distinguished from no saving at all. A fourth arm in which only the encoder is transferred separates the effect of the pretraining method from that of transfer scope, and masked image modelling is observed to transfer negatively over a bounded range of budgets. We emphasise that the scarcity in this study is constructed by subsampling a large corpus.
1 Introduction
The paper measures how much transcription handwritten Devanagari recognition needs and how pretraining changes that requirement. Across annotation budgets, it expresses pretraining benefits as equivalent human-transcription effort.
- The study varies real fine-tuning labels from 10 to 4,000 across four initialisation regimes while keeping the recogniser, optimiser, and evaluation protocol fixed.Six seeds are used at every budget.
- 81 words reach CER 0.50 with supervised synthetic pretraining, versus 355 from random initialisation, a 4.40× label multiplier [3.56, 4.99].This is the headline annotation-efficiency comparison.
- Supervised synthetic pretraining reaches the scratch model’s zero-shot CER 0.7322 at approximately 136 transcribed words without any real fine-tuning labels.
- The controlled encoder-only transfer arm separates pretraining-task effects from transfer scope, while masked image modelling shows negative transfer over a bounded budget range.The transferred encoder contribution remains detectable longer than the larger sequence-model contribution.
- The contribution is a fine-grained, uncertainty-aware measurement of existing methods in units of human annotation effort rather than a new training method.
2 Related Work
Prior work establishes synthetic and low-resource recognition methods but rarely measures annotation efficiency across budgets in human-effort units. This paper focuses on that annotation-starved regime while examining pretraining task and transfer scope.
- Existing scaling studies relate recognition accuracy to data volume, but generally do not express the relationship as the number of labels required for a target accuracy.
- Synthetic pretraining is established for scene text and handwritten recognition, while prior work also studies few-shot training and synthetic-to-real adaptation.
- Unlike work varying pretraining sources or stopping when scratch models fail, this study fixes the source, varies transferred network scope, and retains a from-scratch baseline.
- Layer-transfer research reports decreasing transferability with task-specific depth, motivating this paper’s CRNN-CTC analysis across annotation budgets in annotation-efficiency terms.
- Masked modelling has been effective but does not consistently outperform transfer from closely related domains, so it is included as a directly measured comparison arm.
- Devanagari handwriting is not inherently under-resourced here; scarcity is constructed by subsampling for systematic curve measurement.
3 Experimental Setup
The experiments use a corrected Devanagari word corpus, a common CRNN-CTC recogniser, and four regimes that vary pretraining and transfer scope. Synthetic and masked-image pretraining are trained under disclosed, non-identical settings.
- The dataset contains 95,430 Devanagari word images from 12 writers, using the corrected IIIT-HW-Dev v1 corpus.
- Scarcity is artificially created by subsampling the large corpus rather than treating Devanagari as intrinsically low-resource.
- All four arms use the same CRNN with CTC loss, while transfer analysis distinguishes the convolutional encoder, recurrent layers, and output head.
- The selected height-64, stride-4 encoder leaves 4.44 time steps per character at the first percentile and never runs out of CTC alignment positions.
- Greedy CTC decoding uses no language model or lexicon because the training vocabulary contains every test word, avoiding lookup-based error reduction.
- Synthetic pretraining uses 20,000 rendered words generated from the same 100-character inventory with an akshara-level grammar for broad character coverage.
- The four regimes compare random initialisation, masked image modelling, encoder-only supervised transfer, and full supervised transfer.
- Supervised and masked pretraining share 20 epochs and optimisation scheduling but differ in weight decay, and masked pretraining had not converged within the budget.
3.5 Annotation-Budget Protocol
The protocol evaluates four regimes across matched annotation budgets with repeated seeds and a fixed test metric. A smaller validation-budget experiment checks whether the large validation set drives the low-data results.
- All four arms are fine-tuned at nine matched training-label counts from 10 to 4,000, using six seeds per point and identical labelled subsets.The sweep comprises 216 fine-tuning runs.
- Corpus-level CER is measured on a constant 4,000-word test sample across every arm and label count.
- The fixed 1,500-word validation set selects checkpoints, but its labels are excluded from reported training budgets and represent additional real annotation cost.
- A robustness rerun uses validation size max(n, 10) at five budgets with three seeds, comparing each result seed-for-seed with the main sweep.
- The validation-budget check found no detectable effect: the largest mean test-CER change was +0.0125, within the corresponding 0.0129 seed standard deviation.
- Main results retain the fixed validation protocol so model selection remains constant across all four arms, although the check does not rule out every noise source.
3.7 Uncertainty
Annotation-equivalent estimates use matched seed-resampled comparisons, log-log interpolation, and monotonic observed segments only. Their uncertainty reflects random-seed variation, not test-set, writer, or interpolation uncertainty.
- Uncertainty estimation: Intervals use 2,000 bootstrap replicates that resample the same six seed indices across arms, producing matched comparisons and ratios.Ratios are formed within each replicate so flat-curve instability propagates into the interval.
- Uncertainty estimation: The intervals capture seed-choice variation only and omit test-set sampling, writer variation, and interpolation error, so six-seed estimates are indicative rather than precise.The three-writer test split prevents estimating writer variation from a single draw.
- Annotation-equivalent analysis: Annotation-equivalent counts compare real labelled words needed to reach a target CER, using log-log interpolation between measured points.Synthetic data is excluded because it requires no human transcription; scratch and pretrained counts define the saving and multiplier.
- Interpolation rules: Estimates are reported only within the measured range and on monotonically decreasing CER segments, avoiding extrapolation and ambiguous inversion.Non-monotonic segments cannot yield unique annotation-equivalent counts.
4 Annotation-Efficiency Curves
Annotation-efficiency curves compare CER across real-word budgets for four arms, showing the strongest supervised-pretraining advantage at scarce labels. The arms converge at high budgets, while masked pretraining has a non-monotone low-budget segment and no arm follows one power law.
- Ordering: At n = 10, supervised synthetic pretraining achieves CER 0.6558 versus 0.9529 for random initialisation, and remains best through n = 500.The encoder-only supervised arm is second across this range; masked image modelling is worst from n = 25 to n = 2000.
- High-budget convergence: At n = 4000, all four arms lie between CER 0.1924 and 0.1975, with differences comparable to seed standard deviations and no meaningful ordering.The arms also become close at n = 1000 and n = 2000, where encoder-only transfer is slightly better than full transfer.
- Non-monotonicity: The masked-pretraining curve rises from CER 0.9407 to 0.9436 between n = 10 and n = 25, so n = 25 is excluded from annotation-equivalent interpolation.The point remains in the table and figure; it is the only excluded point.
- Curve shape: No arm is well described by one power law: scratch slopes range from −0.0182 to −0.5285, while full-transfer slopes become steeper across most budgets.Adjacent-segment slopes are reported because curve shape changes substantially across the annotation range.
- Robustness: A smaller validation-budget experiment found no detectable effect: all 20 comparisons stayed within the corresponding seed standard deviations.The largest difference was a +0.0125 CER shift for synthsup at n = 10, versus a 0.0129 seed standard deviation.
5 What Pretraining Buys
Supervised synthetic pretraining substantially reduces the transcription needed to reach moderate CER targets, including a zero-shot-equivalent benefit, but savings shrink at higher accuracy.
- 81 words reach CER 0.50 with supervised synthetic pretraining versus 355 from scratch, yielding a 4.40× label multiplier [3.56, 4.99].The corresponding annotation saving is 0.7725 [0.7189, 0.7994], or roughly three quarters less transcription effort.
- The label multiplier falls from 4.40× at CER 0.50 to 1.68× at CER 0.40 and 1.16× at CER 0.30 as the target accuracy improves.At CER 0.20, the saving is 0.0616 [−0.0148, 0.1246], so it cannot be distinguished from no saving.
- CER 0.7322 is achieved with zero real labels after synthetic pretraining, matching the scratch curve at approximately 136 transcribed words.This provides a zero-shot annotation-equivalent reference point.
- The zero-shot result is essentially unaffected by memorisation because at most 6 of 4,000 test words could have been produced by memorised synthetic forms.The audit is limited by small counts and confounding between word length and synthetic-corpus overlap.
6 Transfer Analysis
Transfer analysis attributes most small-budget benefit to transferring the recurrent layers and output head alongside the encoder, while masked image modelling transfers negatively only over a bounded range.
- 6.1 Full-Model vs. Encoder-Only Transfer: Encoder-only transfer improves over scratch from n = 10 to n = 1000, but its CER advantage is small, ranging from 0.0165 to 0.0600.Full-model transfer reaches a difference of 0.3498 CER at n = 50 against scratch.
- 6.1 Full-Model vs. Encoder-Only Transfer: Full-model transfer improves more than encoder-only transfer at budgets n = 10–250, with the largest difference of 0.3237 CER at n = 50.At n = 500 and higher, the intervals do not establish that full-model transfer remains better.
- 6.2 Pretraining Task: Masked image modelling is worse than scratch at n = 50–500, with the largest difference of +0.0698 CER at n = 250.The study does not treat this as general masked-pretraining behavior because the effect disappears at larger budgets and the comparison disadvantages masked pretraining through convergence and weight-decay differences.
- 6.3 Transfer Components and Curve Shape: The sequence-model contribution is larger at small budgets, rising from 0.2729 CER at n = 10 to 0.3237 at n = 50 before falling to 0.1362 at n = 250.Its interval excludes zero only through n = 250, whereas the encoder contribution remains detectable through n = 1000.
- 6.3 Transfer Components and Curve Shape: The encoder contribution stays within 0.0165–0.0600 CER, while percentage shares become unstable as the total synthsup–scratch gap collapses.Absolute CER differences are therefore retained as the interpretable decomposition measure rather than percentage shares.
7 Discussion
The study’s contribution is a budget-wide, uncertainty-aware measurement of existing pretraining methods in human annotation units. Its curves show that benefits depend on target accuracy, curve shape, transfer scope, and statistically supported budget ranges.
- 7 Discussion: The paper evaluates pre-existing methods rather than introducing a new method, using full annotation-budget curves instead of a single fixed-budget comparison.This reframes pretraining comparison in units of human effort.
- 7 Discussion: 4.40× label multiplier at CER 0.50 falls to 1.07× at CER 0.20, so the supervised synthetic-pretraining benefit is target-dependent and eventually indistinguishable from zero.At CER 0.50, the multiplier is 4.40×; at CER 0.20, it is 1.07×.
- 7 Discussion: Curve shapes change across budgets: scratch improves rapidly in the middle range, whereas full transfer improves more steadily and steepens at high budgets.Because the arms are not parallel, a single fitted power law would obscure meaningful behavior.
- 7 Discussion: Negative transfer from masked image modelling and the component effects are range-limited, so claims require seed-resampled intervals that exclude zero.The encoder contribution remains detectable farther along the label axis than the larger sequence-model contribution.
- 7 Discussion: The encoder’s share of total benefit rises because total benefit shrinks, not because its absolute contribution increases.The study therefore reports absolute CER contributions and restricts share-based claims to intervals within [0, 1].
8 Limitations
The limitations constrain how broadly the reported annotation savings and transfer effects should be interpreted. They concern the artificially constructed scarcity, accounting choices, dataset and architecture scope, test-word overlap, synthetic rendering, and unresolved component comparisons.
- 8 Limitations: The scarcity is artificially created by subsampling 95,430 transcribed words, so the reported multipliers apply to this corpus and do not establish behavior for genuinely under-resourced scripts.Devanagari is explicitly not offered as a proxy for a real under-resourced script.
- 8 Limitations: Reported training-word savings understate total transcription cost because every arm also uses a fixed 1,500-word validation set, especially at the smallest budgets.A scaled-validation analysis is provided separately.
- 8 Limitations: The results come from one CRNN with CTC on one Devanagari dataset, leaving curve shape, decomposition, and negative transfer uncertain across architectures, corpora, and scripts.This limits architectural and cross-script generalisation.
- 8 Limitations: Because every distinct test word also appears in training, the error rates do not measure generalisation to unseen words, although the arm comparison remains unaffected.Performance would likely be worse on a test set containing unseen words.
- 8 Limitations: The synthetic corpus uses an akshara-level grammar selected for character coverage, so the study does not test linguistic realism or alternative synthetic-data designs.The synthetic corpus is not varied in this comparison.
- 8 Limitations: The study compares synthetic pretraining with random initialisation but includes no real-handwritten-data pretraining arm, so it cannot determine which source is better.This leaves the relative value of synthetic versus real handwritten pretraining unresolved.
- 8 Limitations: Component claims are range-restricted: the sequence-model contribution is supported only through n = 250, while the encoder contribution remains supported through n = 1000.Percentage-share interpretations are restricted further to intervals within [0, 1].
- 8 Limitations: The exact-match audit cannot separate memorisation from word-length effects because short words are both easier and more likely under the akshara-level grammar.The reported counts alone cannot distinguish these explanations.
9 Conclusion
The paper measures how human-transcribed training words relate to Devanagari recognition error across initialisation regimes and converts the resulting curves into annotation-equivalent terms. Supervised synthetic pretraining provides the largest early saving, while its advantage and transfer components narrow with stricter targets or larger budgets.
- 9 Conclusion: The study evaluates four initialisation regimes over nine annotation budgets with six seeds per point, then converts test CER curves into annotation-equivalent terms.The setup keeps the focus on human-transcription requirements.
- 9 Conclusion: Most small-budget benefit comes from the transferred sequence model and output head, but that contribution is short-lived; the smaller encoder contribution persists across a wider budget range.The sequence-model contribution is supported through n = 250, while the encoder remains detectable through n = 1000.
- 9 Conclusion: Masked image modelling shows negative transfer against random initialisation only from n = 50 to n = 500 under the stated compute budget.The conclusion explicitly treats this as a bounded observation rather than a general claim.
- 9 Conclusion: The study’s contribution is a fine-grained, uncertainty-aware measurement of existing methods in the human-annotation units relevant to planning transcription projects.It is an evaluation contribution rather than a new recognition method.
Code Availability
The paper provides code and generated artifacts for reproducing the synthetic corpus, pretraining and fine-tuning sweep, tables, and figures. The dataset itself is not redistributed and remains available from CVIT, IIIT Hyderabad under its own terms.
- Code Availability: Reproduction code covers the synthetic corpus, pretraining and fine-tuning sweep, and every table and figure.The repository’s results/ directory is the authoritative source for reported numbers.
- Code Availability: IIIT-HW-Dev is not redistributed; it is available from CVIT, IIIT Hyderabad under its own terms.The repository documents the expected dataset layout.