Multi-Scale Attention with Dense Encoder for Handwritten Mathematical Expression Recognition

• Handwritten mathematical expression recognition must handle complicated two-dimensional structures, ambiguous handwriting, and variant symbol scales.Source: §I · p. 1
• Sequential and global recognition approaches require difficult symbol segmentation, predefined mathematical grammars, or increasingly complex parsing algorithms.Source: §I · p. 1
• Attention-based encoder-decoder models encode expression images and decode LaTeX strings one symbol at a time while selecting relevant image regions.Source: §I · p. 1
• Dense encoding and multi-scale attention are introduced to improve attention-based handwritten mathematical expression recognition.DenseNet strengthens feature extraction and gradient propagation, while multi-scale attention addresses information loss from pooling.Source: §I · pp. 1–2

A. Dense Encoder

• DenseNet concatenates preceding feature maps as input to each succeeding layer, strengthening feature extraction and facilitating gradient propagation.Source: A. Dense Encoder · p. 2
• The encoder divides DenseNet into densely connected blocks with compression layers before pooling to manage changing feature-map sizes and model compactness.Source: A. Dense Encoder · p. 2

B. Decoder

• The decoder uses GRU units to reduce vanishing and exploding gradient problems during sequence generation.Source: B. Decoder · p. 2
• The CNN output is treated as a variable-length grid of local C-dimensional annotations, which the GRU decoder maps to a variable-length one-hot LaTeX sequence.Source: B. Decoder · p. 2
• At each decoding step, attention probabilities weight local annotations to form a fixed-length context vector for generating the next LaTeX symbol.Source: B. Decoder · p. 2

1) Multi-Scale Dense Encoder:

• The multi-scale dense encoder produces low-resolution annotations from its main branch and high-resolution annotations from a second branch extended before the last pooling layer.Source: 1) Multi-Scale Dense Encoder · p. 3
• Adding multi-scale branches before every pooling layer would impose too much computational cost as feature-map size increases.Source: 1) Multi-Scale Dense Encoder · p. 3
• The main branch uses three dense blocks, bottleneck layers, growth rate k = 24, and block depth D = 32.Source: 1) Multi-Scale Dense Encoder · p. 3

2) Multi-Scale Attention Model:

• Two single-scale coverage-based attention models generate low- and high-resolution context vectors, which are concatenated into the multi-scale context vector.Source: 2) Multi-Scale Attention Model · p. 3
• The multi-scale context vector is used when computing the current decoder state and the probability of the predicted symbol.Source: 2) Multi-Scale Attention Model · p. 3
• The low-resolution attention energy is parameterized by a multilayer perceptron conditioned on the predicted decoder state, local annotation, and coverage vector.Source: 2) Multi-Scale Attention Model · p. 3
• Coverage vectors accumulate past attention probabilities, giving the decoder information about which image regions have or have not been attended.Source: 2) Multi-Scale Attention Model · pp. 3–4

• CROHME 2014 and CROHME 2016 provide the evaluation test sets for the proposed model.The training set contains 8,836 expressions and 101 math symbol classes; the test sets contain 986 and 1,147 expressions, respectively.Source: §III · p. 4
• The CROHME dataset is described as the most widely used public dataset for handwritten mathematical expression recognition.Source: §III · p. 4

• The training objective maximizes predicted symbol probabilities using cross-entropy.The ground-truth word at time step t is represented by w_t.Source: A. Training · p. 4
• The decoder uses a single layer with 256 forward GRU units, while the embedding and decoder state dimensions are both 256.Source: A. Training · p. 4
• The multi-scale attention dimension is 512, with 11 × 11 low-resolution and 7 × 7 high-resolution coverage kernels.Both coverage-vector convolutions use 256 filters.Source: A. Training · p. 4
• Optimization uses Adadelta with gradient clipping, and the best model is selected by word error.Source: A. Training · p. 4

• A left-to-right beam search maintains 10 partial hypotheses at each decoding step.Hypotheses end when they emit the end-of-sentence token.Source: B. Decoding · p. 4
• Decoding differs from training because ground-truth previous predicted symbols are unavailable.Source: B. Decoding · p. 4
• The decoding procedure also adopts an ensemble method to improve performance.Source: B. Decoding · p. 4

• The experiments ask whether the dense encoder and multi-scale attention are effective and whether the proposed approach surpasses state-of-the-art methods.Source: §IV · p. 4
• ExpRate is the percentage of predicted mathematical expressions that exactly match the ground truth.CROHME also compares systems using symbol-level error thresholds from one to three errors.Source: A. Metric · p. 4

B. Evaluation of dense encoder (Q1)

• The dense encoder raises ExpRate by about 5.7% on CROHME 2014 and 5.5% on CROHME 2016 relative to WAP.WAP records 44.4% and 42.0% ExpRate on the two test sets, respectively.Source: B. Evaluation of dense encoder (Q1) · p. 4

• Table I compares recognition performance on CROHME 2014 and CROHME 2016 when using the dense encoder and multi-scale attention model.Source: Table I · p. 4

C. Evaluation of multi-scale attention model (Q2)

• 52.8% and 50.1% ExpRate are achieved on CROHME 2014 and CROHME 2016, respectively, with the proposed multi-scale attention system.The system improves over Dense from 50.1% to 52.8% on CROHME 2014 and from 47.5% to 50.1% on CROHME 2016.Source: C. Evaluation of multi-scale attention… · p. 5
• D = 16 is the best tested multi-scale branch depth on both CROHME test sets, whereas D = 24 degrades performance.D = 0 provides only slight improvement over Dense, indicating that additional convolution operations are needed for semantic high-resolution annotations.Source: C. Evaluation of multi-scale attention… · p. 5, Table II · p. 5
• High-resolution attention correctly detects a decimal point under-parsed by low-resolution attention and recognizes “- 1” instead of mis-parsing it as “7”.The figure compares attention on low-resolution and high-resolution annotations, with red indicating attention probabilities and predicted symbols shown beside the images.Source: C. Evaluation of multi-scale attention… · p. 5, Figure 3 · p. 5

D. Comparison with state-of-the-arts (Q3)

• On CROHME 2014, the proposed system achieves the best reported ExpRate of 52.8% after excluding MyScript because it used extra private training data.The comparison includes encoder-decoder systems with attention and traditional grammar-based systems.Source: D. Comparison with state-of-the-arts… · p. 5, Table III · p. 5
• On CROHME 2016, the proposed system reaches 50.1% ExpRate, compared with Wiris at 49.6% using only CROHME training data plus a large Wikipedia formula corpus.The authors present this comparison as a test of the approach’s generalization capability.Source: D. Comparison with state-of-the-arts… · pp. 5–6, Table IV · p. 6

• The dense encoder and multi-scale attention model improve attention-based handwritten mathematical expression recognition over state-of-the-art methods.Source: §V · p. 6