PA-CDM: Position-Aware Character Detection Matching for Evaluating Handwritten Mathematical Expression Recognition

I. INTRODUCTION

• On fraction-operand swaps, CDM scores 0.8595 versus 0.6253 for position-aware scoring; on over rewrites, TED scores 0.8552 while CDM, PA-CDM, and IMS score 1.0.The examples illustrate complementary blind spots: CDM misses structural roles, while TED can penalize equivalent rewrites outside its parser’s normalization coverage.Source: §I · p. 1
• The frontier VLM judge correlates with human scores at ρ = 0.9613, but its calibration cost, nondeterministic outputs, unusable verdicts, and hosted-model drift limit its role as a default signal.The calibration consumed roughly 7.09M API tokens and CNY 202.46; 34 of 3,024 scoring calls returned no usable verdict after retries.Source: §I · pp. 1–2
• PA-CDM encodes formulas as position forests, aligns entity streams, and weights position-vetoed matches by divergence level; StructPerturb v2.0 contains 1,340 pairs across 15 type–intensity cells.The cross-metric protocol combines a seven-metric sensitivity matrix, a six-annotator human study, LLM-judge calibration, and a diagnostic decision table.Source: §I · p. 2
• ρ = 0.9535 (n = 990): PA-CDM has the highest correlation with human judgments among seven automatic metrics.The study reports ρ = 0.9520 under the original n = 1,008 protocol; the difference reflects restricting the sample to items with three complete judge repeats.Source: §I · p. 2

II. THE PA-CDM METRIC

• A deterministic walk encodes each formula as a position forest, assigning each symbol a role-path identifier rooted at M; identifier length represents structural depth.Superscripts and subscripts append A and B; fraction numerator and denominator likewise append A and B, respectively.Source: §II · p. 2
• Entity identifiers are aligned to rendered tokens using one-to-many normalization; when normalized streams differ, sequence alignment assigns identifiers only to exactly matching blocks.Structural commands visible only in the rendering stream are excluded from the entity-level denominator.Source: §II · p. 2
• The scorer follows CDM’s rendering and Hungarian box-matching pipeline, adding a structural alignment cost; with structural checks disabled and wa = 0, it reproduces CDM scores.Encoding failures—28 of 985 CROHME 2014 predictions, all truncated strings with unbalanced braces—fall back to pure CDM scoring.Source: §II · p. 2
• Matched pairs with differing identifiers are position-vetoed, then re-enter the denominators with a penalty weighted by the depth of the first differing role.Pairs with missing identifiers are exempt; equivalent pairs, unmatched tokens, and exempt pairs keep weight 1, so equivalence rewrites are not penalized.Source: §II · pp. 2–3
• A grid study fixed p = 2, while wa sweeps showed near-zero, non-monotone sensitivity; the relation-elimination channel was deleted after failing guardrails.The p study found veto coverage, not weight magnitude, to be the binding constraint; the aggressive relation redesign fired on 1.15% of matched pairs and broke two guardrails.Source: §II · p. 3
• On 985 CROHME 2014 predictions, PA-CDM mean F1 is 0.9136 versus CDM’s 0.9233, while perfectly scored shares are 0.6365 versus 0.6457.PA-CDM ≤ CDM holds only at the mean level; individual-pair and frozen-cell reversals are reported.Source: §II · p. 3

III. STRUCTPERTURB V2.0 BENCHMARK

• The frozen 1,340-pair benchmark spans 15 cells, using deterministic LaTeX rewrites for three structural families at L1–L3 plus equivalence and parser/glyph probes.L2/L3 select non-overlapping sites deterministically; each perturbation can be traced to the exact sites touched.Source: §III · p. 3
• The equiv left-right gate reduces but does not eliminate visual non-equivalence: 43% of 30 gated pairs remained non-equivalent on rendering re-verification.Visual equivalence uses vertical ink-profile Dice ≥0.99; strict pixel-overlap IoU is unreachable for this cell by construction.Source: §III · p. 3
• PA-CDM–CDM gaps widen at L1/L2/L3: script swap 0.0786/0.0974/0.2222, frac swap 0.2342/0.2999/0.4472, and nested move 0.0837/0.1142/0.1871.The no-veto share falls to 1% on frac swap L3; CDM recovers to 0.9694 there under repeated same-structure edits, further amplifying the gap.Source: §III · pp. 3–4, Table I · p. 4
• The matrix reports frozen-set perturbation sensitivity, not metric rankings on natural prediction distributions; nested-move L2/L3 have only 36 and 4 pairs, respectively.The caption attributes the short cells to exhausted candidate pools and notes their limited statistical power.Source: Table I · p. 4
• In the visually identical over-to-frac rewrite, CDM, PA-CDM, and IMS score 1.0, while TED, scores 0.8552 because the rewrite is outside parser normalization coverage.For sqrt order, TED stays at 1.0 while PA-CDM scores 0.9498; this is the sole matrix case where PA-CDM exceeds CDM (0.9409).Source: §III · p. 4

IV. HUMAN STUDY

• Six annotators scored each pair on an anchored 1–5 rubric using the reference and predicted renderings and both LaTeX sources.The 1,008-pair sample drew 252 pairs each from CROHME 2014, 2016, 2019, and UniMER-Test; pairs were binned into 20 corpus-by-quintile strata.Source: §IV · p. 4
• Krippendorff’s ordinal α = 0.9693 on 202 overlap items, above the pre-registered acceptance bar of α ≥0.67.The 95% CI was [0.9472, 0.9853] under 10,000 bootstrap resamples.Source: §IV · p. 5
• ρ = 0.9535: PA-CDM had the highest overall correlation with human scores among seven automatic metrics on the 990-item primary sample.Under the original n=1,008 protocol, its overall correlation was ρ = 0.9520; it ranked first among automatic metrics on CROHME 2014 and 2019, but not 2016 or UniMER-Test.Source: §IV · p. 5, Table II · p. 5
• 52.2% of the 1,008 pairs scored PA-CDM = 1.0, merging the top two quintile boxes and concentrating high-end discrimination in the non-saturated segment.Stratified sampling ensured adequate mass in that segment.Source: §IV · pp. 4–5, Figure 1 · p. 4

• The judge received the same renderings, LaTeX sources, and anchored rubric as human annotators, scored each item three times, and returned usable verdicts for 2,990 of 3,024 main calls.Source: §V · p. 5
• Mean pairwise Krippendorff’s α across repeat pairs was 0.9913 on 990 items with three complete repeats; exact three-way agreement was 92.51% on 1,001 items with at least one valid repeat.The judge-side and human-side α estimates use different sampling designs.Source: §V · p. 6
• 187 of 199 comparable swapped-probe pairs had zero score difference (mean |∆| = 0.060; p = 0.7744), with no pair differing by two or more, yielding no evidence of positional bias.Source: §V · p. 6
• ρ=0.9613 on 990 items, the judge exceeded every automatic metric and ranked first in all five corpus rows, including unimer 0.8742 versus TED 0.7999 and PA-CDM 0.7748.Source: §V · p. 6
• PA-CDM reached ρ=0.9535, within ∆=0.0078 of the judge, at zero marginal cost and deterministically, supporting its use as the default rather than a hosted judge.The calibration run cost CNY 202.46 and roughly 7.09M API tokens for 1,008 items at three repeats; the judge is nondeterministic and API-dependent.Source: §V · p. 6
• 26 of 990 items (2.6%) differed by two or more points; 24 were cases where the judge scored more strictly than humans, including two structural or scope errors humans missed.The two examples were a numerator-boundary error and a factor moved into a radical; in both, the judge and metrics were stricter than the human annotator.Source: §V · p. 6

VI. DIAGNOSTIC PROTOCOL

• PA-CDM is the primary RL reward with CDM as a control, but rendering cost, zero-scored render failures, and near-binary rewards constrain its operational use.IMS penalizes equivalence rewrites as harshly as real errors and is unfit as an equivalence-preserving reward.Source: §VI · p. 6
• Equivalence judgment uses encoder, semantic, and visual tiers because no single metric suffices.The tiers are position-forest identity, TED, and profile-Dice ≥0.99, respectively.Source: §VI · p. 6
• Table III maps six evaluation scenarios to recommended metric combinations and their expected blind spots, rather than ranking metrics.The OOD row reports PA-CDM > CDM for 33 of 799 UniMER-Test pairs (max +0.1489); the equivalence row’s 43% residual follows a single-line-height gate on 30 re-verified pairs.Source: §VI · p. 6, Table III · p. 7
• Per-sample diagnosis pairs PA-CDM veto counts with TED tree-edit paths; when no veto fires, PA-CDM reduces to CDM and adds no diagnostic information.OOD evaluation calls for per-pair disclosure because PA-CDM > CDM inversions amplify out of distribution.Source: §VI · p. 7
• Structural-compliance probing uses the gap by intensity grade, but short nested-move bins and length confounds mean conclusions should rest on script-swap and fraction-swap families.The nested-move L2/L3 bins contain n=36 and n=4 pairs because candidate pools were exhausted.Source: §VI · p. 7, Table III · p. 7

VII. RELATED WORK

• Expression rate and string metrics such as BLEU score token sequences without error location, treating a misplaced subscript and swapped fraction operand comparably.HMER model work includes bidirectional sequence modeling, coverage and counting-aware mechanisms, position-forest supervision, and scaling toward real-world inputs.Source: §VII · p. 7
• CDM renders formulas and matches localized characters, while tree-edit metrics compare normalized operator trees and can penalize equivalent rewrites beyond parser normalization.PA-CDM combines CDM-like glyph matching with structural positions audited through veto and weighting; its CDM reimplementation matches the official implementation with zero maximum absolute difference on 985 shared CROHME 2014 samples.Source: §VII · p. 7
• Human and model judges provide another evaluation line, while IMPO uses an image-level IMS reward for GRPO-trained HMER models.The authors’ independent IMS reimplementation uses a different rendering and alignment convention, so its absolute scores are not comparable with the original report.Source: §VII · p. 7

VIII. LIMITATIONS AND ONGOING WORK

• PA-CDM ≤ CDM holds only at the mean level, with observed reversals on CROHME 2014, UniMER-Test, and one frozen cell; their cause remains unresolved.The reported reversals are 2 of 985 CROHME 2014 pairs (max +0.051), 33 of 799 UniMER-Test pairs (max +0.1489), and one frozen cell (+0.0089).Source: §VIII · p. 8
• PA-CDM is not the top-correlating automatic metric in the CROHME 2016 and UniMER groups, and limited UniMER-Test alignment leaves 294 of 799 pairs scored by pure CDM.Alignment covers 63.08% of predictions in the wider UniMER-Test re-evaluation, making the PA-CDM column a mixed-coverage measure.Source: §VIII · p. 8
• Sensitivity-matrix intensity comparisons should be read within perturbation families because short nested-move bins and multi-site candidate pools bias some samples toward longer formulas.The nested-move L2/L3 cells have n=36 and n=4; candidate-pool lengths vary across structural families and grades.Source: §VIII · p. 8
• Judge-calibration results are tied to one hosted model revision, and its score column is not strictly comparable with the human column.The judge column averages three repeats per item, whereas the human column uses two-annotator means on overlap items.Source: §VIII · p. 8
• The ongoing reward-quality study tests whether PA-CDM improves recognition quality over CDM and reimplemented IMS, and whether sensitivity differences translate into downstream behavior.It crosses three rewards with PosFormer and CoMER backbones and two seeds; results will be reported separately.Source: §VIII · p. 8