PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-Scale Attention with Dense Encoder for Handwritten Mathematical Expression Recognition

Jianshu Zhang, Jun Du, Lirong Dai

arXiv:1801.03530v2cs.CV

TL;DR

Handwritten mathematical expression recognition must map ambiguous, two-dimensional images with varying symbol scales to one-dimensional LaTeX strings. The paper combines a densely connected encoder with multi-scale attention to preserve fine-grained details, achieving 52.8% accuracy on CROHME 2014 and 50.1% on CROHME 2016.

  • Problem

    Handwritten mathematical expression recognition is difficult because expressions have complex two-dimensional structures, ambiguous handwriting, and varying symbol scales.

  • Method

    The paper combines a densely connected convolutional encoder with multi-scale attention in an attention-based encoder-decoder that generates LaTeX strings.

  • Results

    The proposed system achieves 52.8% expression recognition accuracy on CROHME 2014 and 50.1% on CROHME 2016, outperforming state-of-the-art methods.

  • Takeaways & Limitations

    Dense encoding and multi-scale attention improve the attention-based encoder-decoder for handwritten mathematical expression recognition.

Abstract

from arXiv · show

Handwritten mathematical expression recognition is a challenging problem due to the complicated two-dimensional structures, ambiguous handwriting input and variant scales of handwritten math symbols. To settle this problem, we utilize the attention based encoder-decoder model that recognizes mathematical expression images from two-dimensional layouts to one-dimensional LaTeX strings. We improve the encoder by employing densely connected convolutional networks as they can strengthen feature extraction and facilitate gradient propagation especially on a small training set. We also present a novel multi-scale attention model which is employed to deal with the recognition of math symbols in different scales and save the fine-grained details that will be dropped by pooling operations. Validated on the CROHME competition task, the proposed method significantly outperforms the state-of-the-art methods with an expression recognition accuracy of 52.8% on CROHME 2014 and 50.1% on CROHME 2016, by only using the official training dataset.

I. INTRODUCTION

Handwritten mathematical expression recognition is difficult because expressions have complex two-dimensional structures, ambiguous handwriting, and symbols at varying scales. The paper improves an attention-based encoder-decoder by combining dense convolutional encoding with multi-scale attention to preserve fine-grained information.

  • Handwritten mathematical expression recognition must handle complicated two-dimensional structures, ambiguous handwriting, and variant symbol scales.
  • Sequential and global recognition approaches require difficult symbol segmentation, predefined mathematical grammars, or increasingly complex parsing algorithms.
  • Attention-based encoder-decoder models encode expression images and decode LaTeX strings one symbol at a time while selecting relevant image regions.
  • Dense encoding and multi-scale attention are introduced to improve attention-based handwritten mathematical expression recognition.DenseNet strengthens feature extraction and gradient propagation, while multi-scale attention addresses information loss from pooling.

II. METHODOLOGY

The methodology uses a densely connected convolutional encoder and an attention-based GRU decoder to transform expression images into variable-length LaTeX strings. Attention computes context vectors from spatial annotations so each output symbol can focus on relevant image regions.

  • A. Dense Encoder: DenseNet concatenates preceding feature maps as input to each succeeding layer, strengthening feature extraction and facilitating gradient propagation.
  • A. Dense Encoder: The encoder divides DenseNet into densely connected blocks with compression layers before pooling to manage changing feature-map sizes and model compactness.
  • B. Decoder: The decoder uses GRU units to reduce vanishing and exploding gradient problems during sequence generation.
  • B. Decoder: The CNN output is treated as a variable-length grid of local C-dimensional annotations, which the GRU decoder maps to a variable-length one-hot LaTeX sequence.
  • B. Decoder: At each decoding step, attention probabilities weight local annotations to form a fixed-length context vector for generating the next LaTeX symbol.

1) Multi-Scale Dense Encoder:

The multi-scale dense encoder adds a high-resolution branch to the main low-resolution branch. This design supplies both broader semantic context and finer visual information while limiting the computational cost of adding branches at every pooling stage.

  • 1) Multi-Scale Dense Encoder:: The multi-scale dense encoder produces low-resolution annotations from its main branch and high-resolution annotations from a second branch extended before the last pooling layer.
  • 1) Multi-Scale Dense Encoder:: Adding multi-scale branches before every pooling layer would impose too much computational cost as feature-map size increases.
  • 1) Multi-Scale Dense Encoder:: The main branch uses three dense blocks, bottleneck layers, growth rate k = 24, and block depth D = 32.

2) Multi-Scale Attention Model:

The multi-scale attention model attends separately to low- and high-resolution annotations and combines their context vectors for decoding. Coverage tracks prior attention so the decoder can distinguish attended from unattended image regions.

  • 2) Multi-Scale Attention Model:: Two single-scale coverage-based attention models generate low- and high-resolution context vectors, which are concatenated into the multi-scale context vector.
  • 2) Multi-Scale Attention Model:: The multi-scale context vector is used when computing the current decoder state and the probability of the predicted symbol.
  • 2) Multi-Scale Attention Model:: The low-resolution attention energy is parameterized by a multilayer perceptron conditioned on the predicted decoder state, local annotation, and coverage vector.
  • 2) Multi-Scale Attention Model:: Coverage vectors accumulate past attention probabilities, giving the decoder information about which image regions have or have not been attended.

III. TRAINING AND TESTING DETAILS

The proposed model is evaluated on CROHME 2014 and CROHME 2016, two public handwritten mathematical expression recognition benchmarks.

  • CROHME 2014 and CROHME 2016 provide the evaluation test sets for the proposed model.The training set contains 8,836 expressions and 101 math symbol classes; the test sets contain 986 and 1,147 expressions, respectively.
  • The CROHME dataset is described as the most widely used public dataset for handwritten mathematical expression recognition.

A. Training

Training maximizes predicted symbol probabilities with cross-entropy, using a GRU decoder and specified optimization and architecture settings.

  • The training objective maximizes predicted symbol probabilities using cross-entropy.The ground-truth word at time step t is represented by w_t.
  • The decoder uses a single layer with 256 forward GRU units, while the embedding and decoder state dimensions are both 256.
  • The multi-scale attention dimension is 512, with 11 × 11 low-resolution and 7 × 7 high-resolution coverage kernels.Both coverage-vector convolutions use 256 filters.
  • Optimization uses Adadelta with gradient clipping, and the best model is selected by word error.

B. Decoding

During decoding, the model generates the most likely LaTeX string without access to ground-truth previous symbols, using left-to-right beam search.

  • A left-to-right beam search maintains 10 partial hypotheses at each decoding step.Hypotheses end when they emit the end-of-sentence token.
  • Decoding differs from training because ground-truth previous predicted symbols are unavailable.
  • The decoding procedure also adopts an ensemble method to improve performance.

IV. EXPERIMENTS

The experiments assess dense-encoder effectiveness, multi-scale attention, and comparison with state-of-the-art systems using CROHME recognition metrics.

  • The experiments ask whether the dense encoder and multi-scale attention are effective and whether the proposed approach surpasses state-of-the-art methods.
  • ExpRate is the percentage of predicted mathematical expressions that exactly match the ground truth.CROHME also compares systems using symbol-level error thresholds from one to three errors.
  • B. Evaluation of dense encoder (Q1): The dense encoder raises ExpRate by about 5.7% on CROHME 2014 and 5.5% on CROHME 2016 relative to WAP.WAP records 44.4% and 42.0% ExpRate on the two test sets, respectively.
  • Table I compares recognition performance on CROHME 2014 and CROHME 2016 when using the dense encoder and multi-scale attention model.

C. Evaluation of multi-scale attention model (Q2)

The multi-scale attention model improves recognition across CROHME 2014 and 2016, while visualizations show that high-resolution attention recovers fine-grained symbols missed by low-resolution attention.

  • C. Evaluation of multi-scale attention model (Q2): 52.8% and 50.1% ExpRate are achieved on CROHME 2014 and CROHME 2016, respectively, with the proposed multi-scale attention system.The system improves over Dense from 50.1% to 52.8% on CROHME 2014 and from 47.5% to 50.1% on CROHME 2016.
  • C. Evaluation of multi-scale attention model (Q2): D = 16 is the best tested multi-scale branch depth on both CROHME test sets, whereas D = 24 degrades performance.D = 0 provides only slight improvement over Dense, indicating that additional convolution operations are needed for semantic high-resolution annotations.
  • C. Evaluation of multi-scale attention model (Q2): High-resolution attention correctly detects a decimal point under-parsed by low-resolution attention and recognizes “- 1” instead of mis-parsing it as “7”.The figure compares attention on low-resolution and high-resolution annotations, with red indicating attention probabilities and predicted symbols shown beside the images.
  • D. Comparison with state-of-the-arts (Q3): On CROHME 2014, the proposed system achieves the best reported ExpRate of 52.8% after excluding MyScript because it used extra private training data.The comparison includes encoder-decoder systems with attention and traditional grammar-based systems.
  • D. Comparison with state-of-the-arts (Q3): On CROHME 2016, the proposed system reaches 50.1% ExpRate, compared with Wiris at 49.6% using only CROHME training data plus a large Wikipedia formula corpus.The authors present this comparison as a test of the approach’s generalization capability.

V. CONCLUSION

The study improves attention-based handwritten mathematical expression recognition with a dense encoder and a multi-scale attention model, and reports better performance than state-of-the-art methods.

  • The dense encoder and multi-scale attention model improve attention-based handwritten mathematical expression recognition over state-of-the-art methods.
Loading 1801.03530v2…