PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Spatial-Semantic Context with Fully Convolutional Recurrent Network for Online Handwritten Chinese Text Recognition

Zecheng Xie, Zenghui Sun, Lianwen Jin, Hao Ni, Terry Lyons

arXiv:1610.02616v2cs.CV

TL;DR

Online handwritten Chinese text recognition must handle large character sets, ambiguous segmentation, and variable-length input. This paper combines path-signature features, a multi-spatial-context recurrent network, and an implicit language model; its reported system outperforms earlier results.

  • Problem

    OHCTR must address large character sets, ambiguous segmentation, and variable-length inputs, while segmentation can introduce difficult-to-correct errors.

    Source: §I · p. 1

  • Method

    The system converts trajectory windows into signature features, uses MC-FCRN for multi-spatial-context recognition, and applies an implicit language model to the predicting sequence.

    Source: §I · p. 2, §III · p. 4

  • Results

    3C-FCRN+SLD was the best result among reported results, with relative error reductions of 38% in CR and 37% in AR compared with Zhou et al..

    Source: 4) Comparison with Previous Methods · p. 12

  • Takeaways & Limitations

    Path signature truncated at level two offers a reasonable trade-off between efficiency and complexity for OHCTR.

    Source: §VII · p. 13

    The character set was limited to 2,650 classes with CASIA2.0-2.2 training data, and out-of-vocabulary handling remains an open challenge.

    Source: §VII · p. 13

Abstract

from arXiv · show

Online handwritten Chinese text recognition (OHCTR) is a challenging problem as it involves a large-scale character set, ambiguous segmentation, and variable-length input sequences. In this paper, we exploit the outstanding capability of path signature to translate online pen-tip trajectories into informative signature feature maps using a sliding window-based method, successfully capturing the analytic and geometric properties of pen strokes with strong local invariance and robustness. A multi-spatial-context fully convolutional recurrent network (MCFCRN) is proposed to exploit the multiple spatial contexts from the signature feature maps and generate a prediction sequence while completely avoiding the difficult segmentation problem. Furthermore, an implicit language model is developed to make predictions based on semantic context within a predicting feature sequence, providing a new perspective for incorporating lexicon constraints and prior knowledge about a certain language in the recognition procedure. Experiments on two standard benchmarks, Dataset-CASIA and Dataset-ICDAR, yielded outstanding results, with correct rates of 97.10% and 97.15%, respectively, which are significantly better than the best result reported thus far in the literature.

I. INTRODUCTION

OHCTR poses challenges from large character sets, ambiguous segmentation, and variable-length inputs. The paper combines path-signature features, multi-spatial-context recognition, and sequence-wide semantic context to address them.

I. INTRODUCTION

  • OHCTR combines large character sets, ambiguous segmentation, and variable-length inputs, while over-segmentation can cause mis-segmentation that is barely rectifiable through post-processing.The paper describes over-segmentation as an efficient segmentation-recognition method, but says its errors degrade overall performance.Source: §I · p. 1
  • The paper’s system combines path-signature feature maps, multi-scale MC-FCRN processing, and implicit-language-model decoding into a prediction sequence and final label sequence.The authors position path signature as a theoretical alternative to empirically developed feature-extraction techniques.Source: Figure 1 · p. 2, §I · p. 2, §II · p. 3

II. RELATED WORK

  • The implicit language model predicts from the entire feature sequence in both directions, rather than deriving context only from previously predicted text.The overview describes this semantic context as operating on embeddings transformed from the predicting sequence.Source: §I · p. 2, Figure 1 · p. 2, §II · p. 3
  • Path signature translates pen-tip trajectories into offline feature maps that preserve essential online information, with truncation offering a trade-off between complexity and precision.The authors apply this mathematically grounded feature set using sliding windows to characterize trajectory information.Source: §II · p. 3, §I · p. 3
  • MC-FCRN uses multiple receptive-field scales to capture spatial context, addressing the claimed inflexibility of fixed-size FCN windows for unconstrained OHCTR characters.Its FCN, LSTM, and CTC components are jointly trained.Source: §I · p. 2, §II · p. 3

III. PATH SIGNATURE

Path signature represents pen-tip trajectories through iterated-integral features that preserve online dynamics and capture analytic and geometric path properties. The paper uses signatures from small sliding windows to reduce the effect of connections between strokes while retaining local information.

III. PATH SIGNATURE

  • Applying path signatures to extremely long sequences of hundreds of thousands of points extends prior character-level signature extraction to text recognition.The paper describes signatures as preserving online dynamics and capturing analytic and geometric properties of paths.Source: §III · p. 3
  • The path signature is the collection of a path’s iterated integrals, and its full form determines the path up to time re-parameterization.The practical system uses truncated signatures, whose dimensionality grows exponentially with degree and may yield diminishing marginal gains.Source: §III · pp. 3–4
  • Iterated integrals can be calculated for arbitrary-length pen-tip trajectories using path displacement and path-concatenation identities.The method calculates signatures for window paths from the sampled pen-tip trajectory.Source: §III · pp. 3–4
  • Sliding-window path signatures preserve local trajectory information despite added connections between adjacent strokes.The method limits paths to small windows; experiments use width W = 9 and shift V = one point, with the window signature representing its midpoint feature.Source: §III · pp. 4–5, Figure 2 · p. 4

IV. MULTI-SPATIAL-CONTEXT FCRN

The paper extends a fully convolutional recurrent network to learn multiple spatial contexts from signature feature maps for online handwritten text recognition. The architecture accommodates variable-length inputs and uses fully convolutional, recurrent, and CTC components.

IV. MULTI-SPATIAL-CONTEXT FCRN

  • MC-FCRN extends FCRN to learn multiple spatial contexts from complicated signature feature maps.The paper proposes the architecture for online handwritten text recognition.Source: §IV · p. 5
  • FCRN combines a fully convolutional network, recurrent neural network, and CTC to accommodate inputs and outputs of arbitrary size or length.The architecture is proposed for text recognition, where input sequences can have variable length.Source: §IV · p. 5
  • Variable-length feature maps and pen-tip trajectories complicate text recognition compared with normalizing individual characters to a fixed size.The paper presents FCRN as a response to this variable-length input setting.Source: §IV · p. 5

A. Fully Convolutional Recurrent Network

The fully convolutional recurrent network turns path-signature maps into an ordered feature sequence and then into per-frame predictions and a transcription. Its fully convolutional design supports variable-size inputs and preserves spatial order in the feature maps.

1) Fully Convolutional Network:

  • Fully convolutional networks adapt to variable input image sizes and generate corresponding-size feature maps with spatial order information from overlapping receptive fields.Removing the original final fully connected classification layer constructs the fully convolutional network described here.Source: 1) Fully Convolutional Network · p. 5
  • The FCRN pipeline converts signature maps into a deep feature sequence, predicts labels frame by frame with a residual multilayered BLSTM, and transcribes those predictions.Each feature-sequence frame represents a receptive field on the signature maps.Source: Figure 3 · p. 5

2) Multilayered BLSTM:

The recurrent component uses stacked bidirectional LSTMs to model context in both temporal directions, while CTC transcribes without prior alignment. The architecture also converts fully convolutional feature maps into an ordered feature sequence for multiple-spatial-context learning.

2) Multilayered BLSTM:

  • Stacked BLSTMs model context in both forward and reverse directions and access higher-level temporal information for transcription.The paper contrasts BLSTM with standard LSTM, which uses past context in one direction.Source: 2) Multilayered BLSTM · p. 6

3) Transcription:

  • CTC trains the network without prior alignment by summing probabilities over alignments that map to the same transcription.The mapping removes repeated labels and then blanks.Source: 3) Transcription · p. 6

B. Learning Multiple Spatial Context with Multi-Spatial-Context FCRN (MC-FCRN)

  • Collapsing the top feature maps vertically produces an ordered sequence whose successive frames correspond to overlapping input receptive fields.The sequence has length T = W and dimension N.Source: B. Learning Multiple Spatial Context… · p. 6

1) Spatial Context:

The baseline FCN predicts each receptive field independently, while the FCRN adds recurrent layers so predictions can use the full signature feature maps. CTC and a regularized likelihood objective train the system.

Spatial Context

  • The FCRN applies stacked residual BLSTM layers to FCN outputs, letting each time-step prediction use the entire signature feature maps.The residual multilayered BLSTM output is passed to the prediction layer.Source: 1) Spatial Context · pp. 6–7
  • The simple model predicts from only the current receptive field, whereas the FCRN incorporates spatial context from the full input maps.The simple model's per-frame distribution does not consider other feature vectors.Source: 1) Spatial Context · pp. 6–7
  • Training minimizes negative penalized log-likelihood, with L2 weight decay as the regularization term, and uses CTC for transcription.The model components include an FCN, fully connected layers, and CTC.Source: 1) Spatial Context · pp. 6–7

2) Receptive Field and Its Role with Multi-Spatial Context:

The multi-spatial-context network combines receptive fields at different scales for each time step. Their shared center positions preserve alignment while adding spatial context.

Receptive Field and Its Role with Multi-Spatial Context

  • Increasing a layer's kernel size enlarges its receptive field by the kernel-size increase multiplied by the preceding layers' strides.The receptive field is a rectangular input region represented by an FCN feature vector; kernel size, stride, and padding determine its size and center across adjacent layers.Source: 2) Receptive Field and Its Role with… · pp. 7–8
  • Different-scale receptive fields at one time step share a center, allowing multiple spatial contexts while avoiding confusion.The construction uses kernel size kl = 2pl + 1 and stride ml = 1 to keep the center position unchanged across layers.Source: 2) Receptive Field and Its Role with… · p. 8
  • MC-FCRN concatenates the output feature sequences from residual multilayered BLSTMs using different receptive-field scales.The resulting sequence supports predictions at each time step from the entire signature feature maps.Source: 2) Receptive Field and Its Role with… · p. 8

V. IMPLICIT LANGUAGE MODELING

The implicit language model predicts using the entire feature sequence rather than only preceding words. It embeds the predicting features and processes them with bidirectional recurrent layers.

Implicit Language Modeling

  • The implicit language model predicts each time step from the entire predicting feature sequence, rather than directly learning next-word conditional probabilities.Its stated purpose is to incorporate lexical constraints and prior language knowledge.Source: §V · p. 8
  • The model assigns each time-step output a distribution conditioned on the full sequence, then uses CTC to calculate transcription probabilities.Training minimizes negative penalized log-likelihood with a regularization term.Source: §V · p. 8
  • An embedding layer maps predicting feature vectors to real-valued vectors, and a probability function maps those contextual embeddings to a distribution over characters.The embedding takes predicting feature vectors rather than one-hot vectors; the probability function is implemented with multilayered BLSTM to learn forward and reverse long-term context.Source: §V · p. 8

A. Interesting Properties of Implicit Language Model

The implicit language model uses bidirectional sequence context and confidence-bearing prediction features. Its training strategy separates spatial-context learning from semantic-context learning.

A. Interesting Properties of Implicit Language Model

  • The implicit language model uses semantic context from both text directions and can leverage long-term context across the sequence.The authors report that it significantly outperforms models predicting next-word probabilities from previous words in one direction.Source: A. Interesting Properties of Implicit… · p. 8
  • Predicting feature sequences encode label confidence as well as predicted labels, giving the model information beyond a one-hot vector.The implicit language model can exploit this confidence information in addition to semantic context to improve network performance.Source: A. Interesting Properties of Implicit… · pp. 8–9

B. Training Strategy

  • The training strategy uses separate datasets for spatial- and semantic-context learning, shuffling characters for MC-FCRN training and expanding language-model data with synthesis.The authors say this separation helps each model focus on its context type and avoids directly using CASIA2.0–2.2 training samples that share the test corpus.Source: B. Training Strategy · p. 9

C. Statistical Language Model

The experiments evaluate unconstrained online handwritten Chinese text recognition on CASIA and ICDAR benchmarks. They use text and character training data, language-model corpora, and a shared experimental setup.

C. Statistical Language Model

  • The statistical trigram language model is used alongside the implicit language model in experiments to further improve recognition performance.In post-processing, the language model decodes the prediction sequence.Source: C. Statistical Language Model · p. 9

A. Databases

  • The training set includes 4072 pages, 41,710 text lines, and 1,082,220 characters from 2650 classes, split approximately 90% for training and the remainder for validation and language-model parameter learning.Training data include unconstrained text lines and isolated characters.Source: A. Databases · p. 9

VI. EXPERIMENTS

  • The system is evaluated on Dataset-CASIA and Dataset-ICDAR, with 1020 pages and 268,924 characters in the former and 3432 lines and 91,576 characters in the latter.Dataset-ICDAR excludes outlier characters not covered by the training data, making it around 2.02% smaller than the reported dataset.Source: A. Databases · p. 9

A. Databases

  • PFR, PH, and SLD corpora support language modeling, while SRILM is used to build the statistical language model.The Sogou Labs SLD corpus is extremely large, so the experiments use an extract.Source: A. Databases · p. 9

B. Experimental Setting

  • For training, shorter text samples are normalized to 126 pixels high and under 576 pixels wide; testing retains the height and increases width to 2400 pixels.Characters are shuffled during MC-FCRN training to discard semantic context; correct rate and accuracy rate are the evaluation measures.Source: B. Experimental Setting · p. 10

C. Experimental Results

The experiments examine path-signature order, spatial context, and semantic context. Higher signature orders and multiple receptive-field scales improve recognition, while semantic and statistical language modeling further enhance results, with an evaluation caveat for Dataset-CASIA.

1) Effect of Path Signature:

  • 94.52% on Dataset-CASIA and 92.35% on Dataset-ICDAR are reached at Sig3, up from 92.59% and 90.18% at Sig0, respectively.Performance gains slow as iterated integrals increase; Sig2 is selected as a trade-off between efficiency and complexity.Source: Table II · p. 10, 1) Effect of Path Signature · p. 10

2) Effect of Spatial Context:

  • Recognition performance improves monotonically from FCRN to 2C-FCRN to 3C-FCRN on both datasets as the number of receptive-field scales increases.FCRN-2 and FCRN-3, which have similar architecture and size but same-scale receptive fields, perform nearly the same as FCRN.Source: 2) Effect of Spatial Context · pp. 10–11, Table III · p. 11

3) Effect of Semantic Context:

  • 3C-FCRN+B SLD+SLD achieves superior performance, with the best correct rates on all character types.The authors attribute this advantage to SLD’s corpus size of around 56 million characters, larger than PFR and PH.Source: Table IV · p. 11, 3) Effect of Semantic Context · p. 11
  • Implicit language models trained with PFR, PH, and SLD significantly improve performance, and BLSTM-based models outperform LSTM-based models.The authors report further improvement when implicit and statistical language models are applied jointly.Source: Table IV · p. 11, 3) Effect of Semantic Context · p. 11
  • The implicit language model is not fairly evaluated on Dataset-CASIA because its corpus matches the CASIA2.0-2.2 training corpus.The authors instead decode 3C-FCRN predictions with the statistical language model and observe a significant performance improvement on Dataset-CASIA.Source: 3) Effect of Semantic Context · p. 11

4) Comparison with Previous Methods:

The reported best system outperformed prior methods on both benchmarks; error analysis identifies difficult writing and character types behind remaining errors. The paper also identifies its 2,650-class vocabulary and unresolved out-of-vocabulary handling as boundaries.

4) Comparison with Previous Methods:

D. Error Analysis

  • Only 8% of erroneous lines had AR below 70%; most fell at 80–90% or above 90%, while symbols, digits, and especially letters were difficult on average.Source: D. Error Analysis · p. 12
  • Lower ARs were mainly attributed to small writing, character touching, skew or slant, and ambiguous left-right structure; digits and letters lacked cursive training examples.Source: D. Error Analysis · pp. 12–13

VII. CONCLUSION

  • 43% and 38% relative error reductions in correct rate on Dataset-ICDAR and Dataset-CASIA, respectively, marked the best reported results over existing methods.Source: Table V · p. 11, §VII · p. 13
  • The experiments covered only 2,650 classes using CASIA2.0-2.2 training data, leaving out-of-vocabulary handling an open problem.Source: §VII · p. 13

Related papers

Loading 1610.02616v2…