PaperlayerView exact revision on arXiv

Source-linked AI summary

When Counting Meets HMER: Counting-Aware Network for Handwritten Mathematical Expression Recognition

Bohan Li, Ye Yuan, Dingkang Liang, Xiao Liu, Zhilong Ji, Jinfeng Bai, Wenyu Liu, Xiang Bai

arXiv:2207.11463v1cs.CVcs.AI

TL;DR

Encoder-decoder HMER methods can struggle with attention accuracy on complex formulas and long sequences. CAN adds jointly optimized symbol counting, and the paper reports consistent improvements and SOTA CROHME results across encoder-decoder baselines.

  • Problem

    Encoder-decoder HMER predicts markup sequences with attention, but attention can be inaccurate for complicated formula structures or long sequences, against a background of irregular handwriting.

    Source: §1 · pp. 1–2

  • Method

    CAN combines a multi-scale counting module with an attention-based encoder-decoder and jointly optimizes symbol counting and HMER using original HMER annotations.

    Source: §3.1 · p. 4, §3.2 · p. 5, §5 · p. 14

  • Results

    CAN consistently improves encoder-decoder HMER performance and achieves 57.26% on CROHME 2014 with ABM as baseline, a new SOTA result.

    Source: §1 · p. 3, §5 · p. 14

  • Takeaways & Limitations

    Joint optimization benefits both tasks: counting improves HMER, and HMER also increases symbol-counting accuracy.

    Source: §5 · p. 14

    Writing-style variation remains problematic, and the method may err when fine structure perception is needed because it does not explicitly model structure grammar.

    Source: §4.9 · p. 14

Abstract

from arXiv · show

Recently, most handwritten mathematical expression recognition (HMER) methods adopt the encoder-decoder networks, which directly predict the markup sequences from formula images with the attention mechanism. However, such methods may fail to accurately read formulas with complicated structure or generate long markup sequences, as the attention results are often inaccurate due to the large variance of writing styles or spatial layouts. To alleviate this problem, we propose an unconventional network for HMER named Counting-Aware Network (CAN), which jointly optimizes two tasks: HMER and symbol counting. Specifically, we design a weakly-supervised counting module that can predict the number of each symbol class without the symbol-level position annotations, and then plug it into a typical attention-based encoder-decoder model for HMER. Experiments on the benchmark datasets for HMER validate that both joint optimization and counting results are beneficial for correcting the prediction errors of encoder-decoder models, and CAN consistently outperforms the state-of-the-art methods. In particular, compared with an encoder-decoder model for HMER, the extra time cost caused by the proposed counting module is marginal. The source code is available at https://github.com/LBH1024/CAN.

1 Introduction

Encoder-decoder HMER can struggle to attend accurately to complex formulas and long markup sequences. CAN adds jointly optimized symbol counting and reports state-of-the-art CROHME recognition accuracy with DWAP and ABM baselines, with marginal extra time cost.

1 Introduction

  • Encoder-decoder models predict markup sequences with attention, but attention accuracy can suffer on complicated formula structures or long sequences.HMER is challenging because formulas have complex structures and irregular writing.Source: §1 · pp. 1–2
  • CAN combines a weakly supervised symbol-counting module with an encoder-decoder HMER model, using counting and recognition as complementary tasks.The proposed joint optimization consistently improves encoder-decoder HMER performance, and the authors identify counting and HMER as complementary tasks.Source: §1 · pp. 2–3
  • The counting module uses original HMER annotations and adds only marginal extra time cost compared with the original model.CAN is formed by combining MSCM with a typical encoder-decoder model such as DWAP.Source: §1 · pp. 2–3
  • 57.26% on CROHME 2014, 56.15% on CROHME 2016, and 55.96% on CROHME 2019: CAN with ABM achieves new SOTA results.With DWAP, CAN also achieves SOTA recognition accuracy: 57.00% on CROHME 2014, 56.06% on CROHME 2016, and 54.88% on CROHME 2019.Source: §1 · p. 3

2 Related Work

Prior HMER research moved from segmented, grammar-guided pipelines toward deep encoder-decoder models and other learned architectures. Object-counting work distinguishes localization-based supervision from weakly supervised count-level approaches, while CAN targets multi-class symbol counting for formulas.

2.1 HMER

  • Traditional HMER separates symbol segmentation, recognition, and grammar-guided structure analysis; deep-learning work includes attention, coverage, tree-based, and bidirectional models.The cited work also includes pair-wise adversarial learning and transformer approaches.Source: §2.1 · p. 3

2.2 Object Counting

  • Object counting includes detection and density-map regression, while weakly supervised approaches use count-level annotations instead of object positions.CAN differs from category-specific counters by targeting multiple symbol classes and using counting information at feature and loss levels.Source: §2.2 · p. 4

3 Methodology

CAN combines a shared visual backbone, multi-scale symbol counting, and a counting-combined decoder in one end-to-end HMER framework. The count vector joins spatial attention features in symbol prediction, while training uses recognition and count losses.

3.1 Overview

  • CAN uses a DenseNet backbone, MSCM, and CCAD; the backbone’s feature map feeds both modules, while the resulting counting vector also enters CCAD.Source: §3.1 · pp. 4–5

3.2 Multi-Scale Counting Module

  • MSCM extracts multi-scale features with parallel 3×3/5×5 branches, applies channel attention and class-channel projection, then sums sigmoid maps into counts and averages branch vectors for CCAD.Source: §3.2 · pp. 5–6

3.3 Counting-Combined Attentional Decoder

  • CCAD uses spatial positional encoding and coverage attention, then combines local context, the global count vector, hidden state, and prior-symbol embedding to predict each symbol.Source: §3.3 · pp. 6–7

3.4 Loss Function

  • Training combines recognition cross-entropy with smooth L1 regression between predicted and ground-truth symbol counts.Source: §3.4 · p. 7

4 Experiments

The experiments use CROHME and HME100K, with different dataset sizes and image sources. Training details include separate schedules and augmentation settings for the two datasets.

4.1 Datasets

  • CROHME has 8,836 training expressions and 986, 1,147, and 1,199 test expressions for 2014, 2016, and 2019; it has 111 symbol classes.The stroke trajectories are converted from InkML into images for training and testing.Source: §4.1 · pp. 7–8
  • HME100K contains 74,502 training and 24,607 testing images across 249 symbol classes, mainly captured by cameras from tens of thousands of writers.The dataset includes variations in color, blur, and background.Source: §4.1 · p. 8

4.2 Implementation Details

  • CROHME training runs for 240 epochs, with results reported both with and without augmentation; HME100K trains for 30 epochs without augmentation.Training uses a single Tesla V100, batch size 8, and Adadelta.Source: §4.2 · p. 8
  • Six invisible classes—“sos”, “eos”, “ˆ”, a blank symbol, “{”, and “}”—are assigned zero counting ground truth to avoid confusing the model.The passage states that counting these symbols would bring lower accuracy.Source: §4.2 · p. 8

4.3 Evaluation Metrics

Expression recognition is evaluated by exact-match and near-match rates, while symbol-counting performance is evaluated with averaged absolute and squared errors.

4.3 Evaluation Metrics

  • ExpRate is the percentage of expressions recognized correctly.It is the expression-recognition metric.Source: §4.3 · p. 8
  • The ≤1 and ≤2 rates count expressions recognized with at most one or two symbol-level errors, respectively.Source: §4.3 · p. 8
  • For symbol counting, MAE and MSE are computed per formula image and averaged across test images as MAEAve and MSEAve.C is the number of symbol classes, N the number of test images, and V_i and V̂_i are the predicted and ground-truth counts for a symbol class.Source: §4.3 · pp. 8–9

4.4 Comparison with State-of-the-Art

On CROHME, CAN-DWAP achieves state-of-the-art results across the three test sets and CAN-ABM improves on its baseline. The reported recognition examples also show fewer missed or redundant symbols with CAN-DWAP.

4.4 Comparison with State-of-the-Art

  • The comparison focuses mainly on results without data augmentation because most previous methods do not use augmentation.Source: §4.4 · pp. 9–10
  • CAN-DWAP achieves state-of-the-art results on CROHME 2014, 2016, and 2019, exceeding ABM on CROHME 2016 by 3.14%.Source: §4.4 · p. 10
  • The recognition examples indicate that CAN-DWAP is less likely than DWAP to miss symbols or predict redundant symbols.Source: §4.4 · p. 10
  • CAN-ABM outperforms its ABM baseline and achieves new state-of-the-art results, supporting generalization across encoder-decoder models.Source: §4.4 · p. 10

4.5 Results on the HME100K Dataset

On HME100K, CAN-DWAP and CAN-ABM outperform their respective baselines. The counting module adds marginal inference time, with its extra parameters and FLOPs mainly coming from two convolution layers.

4.5 Results on the HME100K Dataset

  • On HME100K, CAN-DWAP and CAN-ABM largely outperform their respective DWAP and ABM baselines.The comparison is reported for the HME100K results in Table 2.Source: §4.5 · p. 10, Table 2 · p. 11
  • CROHME's small size limits method performance; HME100K is nearly ten times larger and has greater variation in color, blur, and background.Source: §4.5 · p. 10

4.6 Inference Speed

  • The counting module adds marginal inference time; its extra parameters and FLOPs mainly come from 3 × 3 and 5 × 5 convolution layers.The speed comparison is presented in Table 3.Source: §4.6 · pp. 10–11, Table 3 · p. 11

4.7 Ablation Study

The ablations examine how joint optimization, counting vectors, positional encoding, and counting-module kernels affect HMER and symbol counting. They also test how count-vector accuracy relates to recognition performance.

4.7 Ablation Study

  • Joint optimization and adding the counting vector each boost HMER performance, while positional encoding adds a smaller gain.The three components are positional encoding, joint optimization, and the counting vector.Source: §4.7 · p. 11, Table 4 · p. 12
  • 57.00% ExpRate, 0.033 MAEAve, and 0.037 MSEAve are achieved with 3 × 3 and 5 × 5 kernels together on CROHME 2014.Using either kernel size alone yields lower counting accuracy and ExpRate.Source: §4.7 · p. 11, Table 5 · p. 12
  • 62.44% ExpRate on CROHME 2014 is reached with ground-truth counting vectors; adding more disturbances lowers ExpRate.The vectors are supplied to the decoder during training and testing.Source: §4.7 · pp. 11–12, Table 6 · p. 12
  • Jointly training HMER improves counting accuracy, lowering MAEAve by 31.25% and MSEAve by 15.91%.The comparison is between CAN trained only on symbol counting and CAN trained on both tasks.Source: §4.7 · p. 12

4.8 Case Study with Maps

The case study contrasts DWAP’s attention and symbol omissions with CAN-DWAP’s prediction and counting maps. CAN-DWAP predicts the formula correctly, with symbol locations and counts close to their ground truth despite no symbol-level position annotations.

4.8 Case Study with Maps

  • DWAP misses P and recognizes it only later, after the sequential decoding mistake has occurred.Its attention map also shows repeated attention on i when it predicts a redundant i.Source: §4.8 · pp. 12–13, Figure 8 · p. 13
  • DWAP’s repeated attention on i accompanies a redundant prediction of that symbol.Source: §4.8 · p. 13, Figure 8 · p. 13
  • CAN-DWAP predicts the formula correctly, with almost all symbols located accurately without symbol-level position annotations.Its predicted count for each symbol class is close to the ground truth.Source: §4.8 · pp. 13–14, Figure 8 · p. 13

4.9 Limitation

Symbol counting does not resolve all recognition problems: writing-style variation remains challenging, and extreme fine-structure perception can still cause errors without explicit structure grammar.

4.9 Limitation

  • Writing-style variation remains difficult, and the method may err when extreme fine-structure perception is needed because it does not explicitly model structure grammar.Source: §4.9 · p. 14

5 Conclusion

The conclusion presents CAN as an end-to-end network that jointly optimizes HMER and symbol counting using a counting module trained from original HMER annotations. Its benchmark experiments report consistent HMER gains, contributions from both joint optimization and counting results, and improved symbol counting from joint optimization.

5 Conclusion

  • CAN’s MSCM performs symbol counting using only original HMER annotations, while jointly optimizing counting and HMER.The conclusion attributes consistent HMER gains to counting and reports that HMER also improves symbol-counting accuracy through joint optimization.Source: §5 · p. 14

Related papers

Loading 2207.11463v1…