Source-linked AI summary
Equal CER, Unequal Damage: Severity-Weighted Character Error Rate for OCR
Nguyen Nguyen, Yongyi Zang
TL;DR
CER charges a typo as much as a wrong name, despite their unequal consequences for readers. semCER retains CER’s alignment and denominator while assigning judged severities to grapheme edits without word segmentation. Reader comparisons support semCER’s edit-level charges and ranking, while normalized CER remains a whole-transcription score and tracks readers as well as semCER or better.
Problem
CER charges each edit equally, while other meaning-sensitive metrics either score whole outputs or grade spans outside CER’s alignment.
Source: §1 · pp. 1–2Method
semCER keeps CER’s alignment and denominator, charges grapheme edits with severity from an LLM judge, and requires no word segmentation.
Source: §1 · p. 2, §3.1 · p. 4Results
On pairs where CER and semCER disagreed, English and Chinese readers favored semCER over normalized CER more often, while Vietnamese readers favored them equally often.
Source: §1 · p. 2Takeaways & Limitations
Normalized CER remains the score for whole transcriptions, while semCER breaks that score down into named edits and orders errors for checking.
Source: §1 · p. 2, §9 · p. 10Readers validated severity in only three languages; in thirteen others, severities rely on the judge, and resource level cannot be separated from script in the lowest-resource languages.
Source: §10 · p. 10
Abstract
from arXiv · showhide
Character Error Rate (CER), the standard metric for OCR and vision-language models (VLMs), charges a typo as much as a wrong name. We introduce semCER, which keeps CER's alignment and denominator and charges each grapheme edit a severity from an LLM judge, with no word segmentation. Normalized CER keeps the score of a whole transcription, where it tracks readers as well as semCER or better, and semCER breaks that score down into named edits. The charges track readers' ratings edit by edit in English, Vietnamese, and Chinese and put the edits readers rate most severe near the top of a sorted list. On pairs drawn where CER and semCER disagreed, readers side with semCER more often than with normalized CER in English and Chinese, and equally often in Vietnamese. Across 16 languages, the judge charges a wrong diacritic or tone mark almost nothing in Spanish but nearly a wrong letter's severity in Telugu. Where we can count real words, a general VLM's wrong letter forms one far above chance while a classical pipeline's stays near it. We release the 61 million judged edits, the aggregated reader ratings, and the distilled scorer at https://github.com/nguyenx2t/semcer.
1 Introduction
CER gives equal cost to edits that can have very different consequences for readers, and normalization still charges every surviving edit alike. semCER keeps CER’s alignment and denominator but judges each grapheme edit’s severity; reader comparisons support its use for ranking and grading errors while normalized CER remains a whole-transcription score.
1 Introduction
- CER charges a typo and a wrong agency name equally, although a reader may repair the typo but not the silent renaming.Normalization removes or folds some cosmetic differences, but still charges every surviving edit alike; fluent wrong words from VLMs are also charged no more than typos.Source: §1 · pp. 1–2
- semCER charges grapheme edits without word segmentation, keeps CER’s alignment and denominator, and returns CER when every edit receives a charge of one.The paper presents semCER as a tool for finding and grading errors, while normalized CER remains the score for a whole transcription.Source: §1 · p. 2
- Sorted by severity, semCER puts edits readers rate most severe near the top, and its charges track reader ratings beyond edit type.In English and Chinese, this relationship also holds beyond whether normalization would remove the edit.Source: §1 · p. 2
- 71% and 78%: English and Chinese readers sided with semCER over normalized CER on pairs where CER and semCER disagreed; Vietnamese readers sided with them equally often.On those same pairs, readers in all three languages sided with semCER significantly more often than with a three-class charge.Source: §1 · p. 2
- Across 16 languages, the charge for a wrong diacritic or tone mark ranges from almost nothing in Spanish to nearly a wrong letter’s severity in Telugu.Where CER and semCER disagreed, English and Chinese readers favored semCER over normalized CER more often, while Vietnamese readers favored the two equally often.Source: §1 · p. 2
- The paper releases 61 million judged edits, aggregated per-item reader ratings, and a distilled scorer whose agreement comes within 0.03 of the judge’s.The student is trained on the judge’s distributions without human ratings.Source: §1 · pp. 2–3
2 Related Work
Prior work addresses unequal error costs through semantic scores, span grading, normalization, and alternative edit costs. These approaches differ from semCER in what they score and how they align or transform text.
2 Related Work
- Speech metrics also replace WER’s unit charge with semantic, importance-based, or impact-based weights, while OCR benchmarks vary in their text normalization.In speech, stripping Indic vowel signs during normalization has been reported to flatter a model.Source: §2 · p. 3
- Semantic similarity metrics score whole outputs or aligned spans on scales unrelated to CER, while OCR metrics also decompose page errors by pipeline stage, reconstruction fidelity, or page-level checks.SCORE adjusts edit distance for representational variation and hallucinated or omitted tokens; semCER charges edits that remain after normalization.Source: §2 · pp. 3–4
- Span-based model judges grade marked errors, while CLEME2.0 aligns source, correction, and reference into word chunks and scores correction outcomes.semCER instead uses only a reference and transcription, scoring error recoverability on a fixed grapheme alignment without word segmentation.Source: §2 · p. 3
- Other document-transcription judges score whole outputs or count errors by class, unlike semCER’s edit-level severity charges.Source: §2 · p. 3
- Normalization rewrites strings before alignment, whereas learned or tuned cost tables let costs choose the alignment; semCER fixes a unit-cost alignment first and charges edits afterward.Source: §2 · p. 3
3 The semCER Metric
semCER replaces CER’s unit edit charge with a severity while retaining its alignment and denominator. It uses grapheme edits and an LLM judge’s probability distribution over a five-level rubric to assign each edit’s charge.
3 The semCER Metric
- semCER replaces CER’s unit charge with a severity for each edit, supplied by a judge.Source: §3 · p. 4
3.1 Definition
- The metric uses extended grapheme clusters and a fixed Levenshtein alignment, avoiding word segmentation while keeping CER’s reference-length denominator.A base letter with accents or tone marks counts as one unit; charging every edit one gives CER back, and semCER never exceeds CER.Source: §3.1 · p. 4
3.2 Severity from an LLM judge
- The judge rates how far an edit prevents a reader from noticing and repairing it, using a five-level rubric from surface change to no signal.It reads the full reference and output with the target edit marked, and no parameters are fit to human ratings.Source: §3.2 · pp. 4–5, Table 1 · p. 5
3.1 Definition
- In the illustrated line, six edits produce CER 0.171 over 35 graphemes, while severities totaling 1.90 produce semCER 0.054.The example assigns no charge to a lowercase-versus-uppercase difference or added space; a wrong word split into three edits carries most of the charge.Source: §3.1 · p. 4, Figure 2 · p. 5
3.2 Severity from an LLM judge
- Severity is the expected rubric rating under the judge’s probabilities, allowing edits at the same most-likely level to receive different charges.The rubric’s level ratings are ρk = k/4; if the judge returns no digit or puts less than half its probability on the five digits, the edit is charged one after one retry.Source: §3.2 · p. 5
4 Experimental Setup
The evaluation combines a multilingual OCR corpus, reader annotations in three languages, and comparisons against judge-free and judge-based baselines. Bootstrap intervals and corrected significance tests are used for comparisons.
4 Experimental Setup
- The judge scores 61 million edits from seven systems on 25 public datasets in 16 languages, covering classical pipelines and specialized and general VLMs.Source: §4 · p. 5
- Readers of English, Vietnamese, and Chinese annotate single edits, whole transcriptions, and preference pairs drawn where CER and semCER rank transcriptions oppositely.The single-edit pool samples low, intermediate, and high judge-severity bands; the pair-level pool tests summed scores.Source: §4 · pp. 5–6
- Figure 3 shows the share of all reader-rated damage caught after checking a fraction of the single-edit pool in each order, over the first half of the list.Readers’ own order is the ceiling, random order the diagonal, and the rule marks the 10% cut.Source: Figure 3 · p. 6
- Baselines include CER’s charge, filter and real-word labels, corpus-mean edit-type costs, coarser judge readouts, and normalized CER.Normalized CER applies the character filter and folds case before alignment.Source: §4 · p. 6
- Results use 95% percentile bootstrap intervals from 10,000 draws, with resampling within severity band for single-edit partial correlations and by source document elsewhere.Significance uses Holm correction within named test families at α = 0.05.Source: §4 · p. 6
5 Per-Edit Agreement with Readers
semCER’s edit ordering puts reader-rated damage near the top, while severity adds information beyond edit type and, in English and Chinese, beyond normalization. The section tests whether that ordering helps readers check errors and whether its finer charges track their ratings.
5.1 Which errors to check first
- 0.58 to 0.81 across the three languages is the score of the severity-sorted list, whose top 10% catches 81 to 92% of the damage caught by readers’ own top 10%.CER’s length-only per-edit charge scores about zero, while the best ordering without a judge lands about halfway; severity’s lead over it is significant in Chinese alone.Source: §5.1 · p. 7
- Severity correlates with readers’ mean rating in all three languages, and the highest-severity edits include the edits readers rate most severe.CER gives all six edits in the illustrated list the same charge and cannot order them.Source: §5.1 · p. 7
5.2 What the severity adds
- Controlling for the type table, severity still tracks readers’ ratings in all three languages, and it adds agreement beyond normalization in English and Chinese.For the example edit, the type table charges 0.59, while the judge charges 0.25 and readers rate it 0.20.Source: §5.2 · pp. 7–8
- Using the judge’s full severity distribution rather than its most likely level adds agreement with readers in Vietnamese and Chinese.Source: §5.2 · p. 8
6 Reader Preferences Where CER and semCER Disagree
Normalized CER tracks readers’ rankings of whole transcriptions as well as or better than semCER. On pairs where CER and semCER disagree, however, readers favor semCER over normalized CER in English and Chinese, especially when normalization leaves CER’s choice unchanged.
6 Reader Preferences Where CER and semCER Disagree
- Spearman’s ρ for normalized CER is 0.93, 0.90, and 0.76, versus 0.89, 0.87, and 0.76 for semCER.These correlations compare each score’s ranking of whole transcriptions with readers’ rankings.Source: §6 · p. 8
- On the disagreement pool, English and Chinese readers side with semCER on more than seven pairs in ten, while Vietnamese readers side with both scores equally often.The pool is drawn from pairs where CER and semCER ranked two transcriptions in opposite order; unnormalized CER therefore agrees with readers on one minus semCER’s share.Source: §6 · p. 8
- On near-miss pairs, semCER significantly leads normalized CER in English and Chinese; three-class charges trail semCER significantly in every language.A near-miss pair has a mostly correct output with an error normalization keeps.Source: §6 · p. 8
- On normalization-proof pairs, semCER agrees with most English and Chinese readers while normalized CER rarely agrees with English readers and agrees with fewer than half of Chinese readers.In Vietnamese, both scores agree with readers on about half of these pairs; the English and Chinese margins also hold with alternate severity scoring setups.Source: §6 · p. 8
7 Severity by Language and by System
The severity assigned to an edit varies with language and system, so equal edit counts can conceal different costs. Language-specific grapheme load and the form of a system’s letter errors help characterize those differences.
7 Severity by Language and by System
- Where general VLMs’ and EasyOCR’s edit counts tie, VLM edits cost more in Chinese and Vietnamese but less in English, French, and Spanish.At least three of four VLMs show the split in each language, which follows grapheme load; German shows no difference, and the gap is significant in 8 of 24 pairings after correction.Source: §7 · pp. 8–9
- 0.75 is the rank correlation between grapheme load and mean severity per edit across 15 languages with marks; values range from 0.26 in English to 0.74 in Telugu.The judge’s mean severity for a mark-only edit ranges from nearly nothing in Spanish to almost a wrong letter’s severity in Telugu; a wrong letter is charged 0.63 to 0.81 across scripts.Source: §7 · p. 9, Figure 5 · p. 9
- In the ten languages where real words can be counted, general VLM letter errors form real words more often than random errors in every language and domain but one.Across eight language-and-domain sets with enough errors from both system types, the VLM’s margin over random is larger; classical errors stay within 0.10 of the random rate.Source: §7 · pp. 9–10
- The judge charges a general VLM’s letter errors more than a classical pipeline’s in every measurable language except Spanish.Source: Figure 6 · p. 9, §7 · p. 10
8 A Distilled Scorer with Fixed Weights
The authors distill the judge into a fixed-weight student scorer so semCER can be computed locally without API scoring. The student nearly matches the judge’s agreement with readers and its edit ordering.
8 A Distilled Scorer with Fixed Weights
- The distilled student is Qwen3.5-0.8B-Base, trained to predict the judge’s severity distribution from an edit-marked pair, with rated-pool lines held out across systems.No human ratings are used for training or model selection, and the student is distributed as a Python package.Source: §8 · p. 10
- 0.79, 0.63, and 0.80 are the student’s correlations with reader ratings in English, Vietnamese, and Chinese, close to the judge’s 0.81, 0.66, and 0.82.The student orders nine in ten pairs as the judge does when judge severities differ by at least half a rubric step, with mean absolute severity difference 0.04 per edit.Source: §8 · p. 10
9 Conclusion
semCER supplements normalized CER by using judged grapheme-edit charges to break ties or disagreements and identify the edits behind a score. The paper recommends normalized CER for scoring whole transcriptions and semCER for comparing them.
9 Conclusion
- Normalized CER remains the recommended score for a transcription, while semCER helps choose between transcriptions when CER ties or the scores disagree.Source: §9 · p. 10
- semCER charges each grapheme edit by judged severity while retaining CER’s alignment and denominator, without requiring word segmentation.Source: §9 · p. 10
- When CER and semCER rank a pair differently, readers of English and Chinese mostly side with semCER, and Vietnamese readers side with both equally often.Source: §9 · p. 10
- The charges track readers’ edit-level ratings in three languages and place the errors they rate most severely near the top of a sorted list.Source: §9 · p. 10
10 Limitations
The validation and scope of semCER are limited by language coverage, the rubric’s construct, judge version changes, and how readers rated errors. The paper also cautions that low semCER does not remove the need to check a transcription.
10 Limitations
- Reader validation covers three languages; the other thirteen rely on judge severities that may vary with language resource level, which is confounded with non-Latin script in this corpus.The authors have not tested graphemes as the right unit for Thai or abjads.Source: §10 · p. 10
- Human studies test whether the judge applies the rubric as readers do, but leave open whether recoverability is the right construct.Source: §10 · p. 10
- Because rubric levels depend on valid-word status, part of the higher charge for general VLM letter errors reflects the rubric; semCER’s downstream-task value remains untested.Source: §10 · p. 11
- Severity comparisons are valid only within one judge checkpoint; corpus severities use the pre-replacement checkpoint, while the fixed-weight student avoids this boundary by approximating the judge.Source: §10 · p. 11
- Readers rated single edits in isolation, so their ratings do not establish severity amid multiple errors; an aligned wrong word split into edits pays once per edit.Source: §10 · p. 11
- A low semCER indicates errors a reader could recover, but the transcription still needs checking, and an undercharged error class could be hidden.Source: §10 · p. 11
Use of AI Assistants
The appendix documents the judge, corpus, reader studies, and analyses behind semCER. It reports that severity depends largely on line context, while whole-transcription agreement also depends on normalization.
Use of AI Assistants
- The authors report using language models for per-edit severity judgments, re-judging rated transcriptions, and scorer distillation, as well as for copy editing and research assistance.They state that the authors remain responsible for the scientific content, study design, experiments, and conclusions.Source: Use of AI Assistants · p. 11
A.1 Judge configuration
- The evaluator compares REF and HYP with edit positions marked, reads sentence context, and scores semantic damage on an integer 0–4 scale according to whether context signals error and the original is recoverable.Source: §A.1 · p. 17
A.3 Human studies
- Native or fluent English, Vietnamese, and Chinese readers rated errors and preferences without seeing system names, judge outputs, or the study hypothesis.Single-edit and pair-level items presented text blocks, while preference items showed two outputs in random order; readers completed worked examples and severity checks before rating.Source: §A.3 · pp. 18–19
A.2 Corpus, systems, and compute
- After filtering, the corpus contains 229,395 transcription lines; systems include classical OCR, a document-specialized VLM, and general VLMs, and the release shares ratings and scorer materials under stated licenses.The release includes severities and ratings under CC BY 4.0, where source-text licenses permit, and scorer code and weights under Apache-2.0.Source: §A.2 · p. 18, Table 3 · p. 19, Table 4 · p. 19
A.4 Statistical tests; B.1 Top-of-list measure; B.2 Baselines
- The analyses use Holm correction within prespecified comparison families; the top-of-list measure contrasts severity-ranked edits with reader and random orders, while baselines include judge-free orders and class or whole-line charges.The top-of-list share is normalized so random order scores 0 and readers’ own order scores 1; the class charge maps five judge levels to three severity classes.Source: §A.4 · pp. 19–20, §B.1 · p. 20
B.3 Full results
- 70–81% of within-band edit pairs remain tied by the judge’s most likely level, and edit identity explains only 39–58% of severity variance, leaving most charge attributable to line context.Readers’ within-band reliability caps the partial correlations at roughly 0.81, 0.85, and 0.71 for the three rated languages.Source: §B.3 · p. 21
C.1 Whole transcriptions
- Normalized CER leads semCER significantly in English and by 0.03 in Vietnamese on the pair-level pool, while Chinese shows little difference across conventions.The three-class charge trails semCER by 0.03 in English and Vietnamese and leads it by 0.01 in Chinese; these intervals are uncorrected.Source: §C.1 · p. 21
C.2 Preference test and its subsets
- SemCER meets the prespecified preference-test criterion by leading normalized CER significantly in English and Chinese on near-miss pairs with a strict reader majority.The near-miss subset excludes cosmetic-only edits, requires more than 0.80 grapheme overlap with the reference, and preserves whole-number tokens and negation count.Source: §C.2 · pp. 21–22
C.3 Robustness to the judge
The appendix tests how results change with a prior judge, equal-count comparisons, language-specific grapheme costs, and a distilled scorer. It also finds that general VLM letter errors often form real words beyond chance, unlike classical OCR errors.
C.3 Robustness to the judge
- The alternative judge reproduces per-edit and whole-transcription findings, but its margin over normalized CER is significant only on normalization-proof pairs in English and Chinese, not across all near-miss pairs.The first judge’s margins remain significant on the whole pool and normalization-proof pairs in English and Chinese under both scoring versions.Source: §C.3 · p. 23
D.1 Equal-count comparison
- No language meets the prespecified criterion that every VLM have a positive equal-count severity gap over EasyOCR surviving Holm correction.All gaps are positive in Chinese and Vietnamese, though only one per language survives correction; English gaps are all negative, while German, French, and Spanish vary by system.Source: §D.1 · p. 24, Table 10 · p. 24
D.2 Grapheme load
- Table 11 reports mean judge severity for mark-only edits, content substitutions, and all edits across lines with one to five errors, comparing grapheme load across languages.The accompanying text defines grapheme load as the severity of a mark-only edit.Source: Table 11 · p. 24, §D.2 · p. 24
D.3 Real-word errors
- VLM letter errors form real words more often than chance predicts in every language and domain except Hindi scene text, while classical errors stay within 0.10 of chance.The comparison covers ten languages with word lists and at least 30 letter errors in some domain; Chinese is added in the figure without a letter-frequency baseline.Source: §D.3 · pp. 24–25, Table 12 · p. 25
E Distilled Scorer
- The distilled scorer’s reader correlation on single-edit pools comes within 0.03 of the judge’s, and it reproduces the judge’s ordering for nine in ten clear edit pairs.Its test-line mean absolute severity difference from the judge is 0.04 per edit; Table 13 reports a pooled reader-agreement share of 90.3% [89.5, 91.1], with no language below 87%.Source: E Distilled Scorer · p. 25, Table 13 · p. 25