PaperlayerView exact revision on arXiv

Source-linked AI summary

Drawing and Recognizing Chinese Characters with Recurrent Neural Network

Xu-Yao Zhang, Fei Yin, Yan-Ming Zhang, Cheng-Lin Liu, Yoshua Bengio

arXiv:1606.06539v1cs.CV

TL;DR

The paper addresses the limited study of automatic Chinese-character writing alongside recognition. It uses RNNs as both end-to-end discriminative models over online trajectories and conditional generative models with character embeddings, achieving state-of-the-art recognition and human-readable, highly recognizable generation.

  • Problem

    Previous work mainly emphasized recognizing handwritten Chinese characters, leaving automatic drawing of real and cursive characters comparatively underexplored.

  • Method

    The framework uses RNNs for both recognition from raw online sequential data and conditional generation jointly trained with character embeddings.

  • Results

    The discriminative RNN achieves state-of-the-art performance on the ICDAR-2013 database, while generated characters are human-readable and recognized with high accuracy.

  • Takeaways & Limitations

    RNNs support a unified framework for recognizing and drawing real and cursive Chinese characters.

Abstract

from arXiv · show

Recent deep learning based approaches have achieved great success on handwriting recognition. Chinese characters are among the most widely adopted writing systems in the world. Previous research has mainly focused on recognizing handwritten Chinese characters. However, recognition is only one aspect for understanding a language, another challenging and interesting task is to teach a machine to automatically write (pictographic) Chinese characters. In this paper, we propose a framework by using the recurrent neural network (RNN) as both a discriminative model for recognizing Chinese characters and a generative model for drawing (generating) Chinese characters. To recognize Chinese characters, previous methods usually adopt the convolutional neural network (CNN) models which require transforming the online handwriting trajectory into image-like representations. Instead, our RNN based approach is an end-to-end system which directly deals with the sequential structure and does not require any domain-specific knowledge. With the RNN system (combining an LSTM and GRU), state-of-the-art performance can be achieved on the ICDAR-2013 competition database. Furthermore, under the RNN framework, a conditional generative model with character embedding is proposed for automatically drawing recognizable Chinese characters. The generated characters (in vector format) are human-readable and also can be recognized by the discriminative RNN model with high accuracy. Experimental results verify the effectiveness of using RNNs as both generative and discriminative models for the tasks of drawing and recognizing Chinese characters.

I. INTRODUCTION

The paper unifies automatic recognition and drawing of real and cursive Chinese characters within an RNN framework. It uses RNNs for end-to-end recognition from online trajectories and conditional generation of human-readable characters.

  • Motivation and scope: Reading and writing Chinese characters are framed as complementary machine-intelligence tasks within one recurrent-neural-network framework.The paper addresses both recognition and automatic drawing of real and cursive characters.
  • Recognition: The recognition model applies bidirectional LSTM- and GRU-based RNNs directly to raw sequential data instead of converting trajectories into image-like representations.The approach is described as fully end-to-end and requiring no domain-specific knowledge.
  • Generation: A conditional generative RNN jointly trains character embeddings to generate different writing styles for a specified character class.The model targets human-readable cursive Chinese characters and samples new trajectories conditioned on the character embedding.
  • Evaluation: The generated characters are vector-format trajectories that are human-readable and recognized with high accuracy by a pretrained discriminative RNN.This evaluation connects the generative and discriminative models within the proposed framework.
  • Online handwriting representation: Chinese handwriting is represented as variable-length sequences containing pen-tip coordinates and pen states, preserving spatial information and writing order.Online data retains dynamic information that is difficult to recover from static images.

A. Removing Redundant Points

The preprocessing removes redundant trajectory points while preserving stroke shape and normalizes coordinates into a standard system. It thereby produces shorter, more informative sequences for RNN modeling.

  • A. Removing Redundant Points: Points inside a stroke are removed when they are too close to the preceding point or lie on the line connecting neighboring points.Stroke starting and ending points are preserved; removal is based on distance and cosine-similarity conditions.
  • A. Removing Redundant Points: The preprocessing preserves character shape while making each remaining trajectory point more informative.The stated goal is to reduce redundancy without losing the character’s shape information.
  • Coordinate normalization: Coordinates are globally normalized using estimated x- and y-projection statistics so characters occupy a standard coordinate system.The y-coordinate is normalized by the x-axis deviation to preserve the original height-to-width ratio and stroke direction.

C. Illustration

The recognition pipeline converts preprocessed online handwriting into six-dimensional line vectors and processes the resulting sequence with recurrent layers. LSTM units address long-term dependencies, while the RNN ultimately supports fixed-length classification.

  • A. Representation for Recognition: Each consecutive-point line is encoded by start coordinates, movement direction, and two indicators for pen-down or pen-up status.The six-dimensional representation transforms the character into a sequence of line vectors.
  • A. Representation for Recognition: The resulting sequence retains both spatial and temporal handwriting information, including stroke transitions and writing order.Online sequential data preserves dynamics that static images do not directly contain.
  • B. Recurrent Neural Network (RNN): An RNN recursively maps each current input and previous hidden state to a hidden-state sequence, which is encoded into a fixed-length vector for classification.The recurrent computation processes variable-length input sequences.
  • C. Long Short Term Memory (LSTM): LSTM recurrent units are used because their memory and gates help learn longer-term dependencies while reducing vanishing and exploding gradient problems.The cell state stores long-term memory, with input and forget gates controlling updates.
  • C. Long Short Term Memory (LSTM): The recognition system uses stacked bidirectional recurrent layers to combine forward and backward context for end-to-end recognition.The architecture also applies dropout in the mean-pooling and fully connected layers.

D. Gated Recurrent Unit (GRU)

GRUs regulate information flow through reset and update gates without a separate memory cell, providing a lighter recurrent alternative within the bidirectional RNN classifier.

  • GRU structure: GRUs use reset and update gates to modulate information flow without a separate memory cell.The update gate interpolates between the previous activation and a candidate activation, while the reset gate can enable forgetting of the previous state.
  • Bidirectional RNN integration: Bidirectional recurrent layers combine forward and backward hidden-state sequences into a fixed-length representation for classification.The representation is passed to fully connected and softmax layers for final classification.

F. Regularization and Data Augmentation

The paper uses sequential dropout to regularize and augment online handwriting sequences, while optionally averaging predictions from multiple dropped sub-sequences at test time.

  • Sequential dropout: Sequential dropout randomly removes items from an input sequence to generate sub-sequences for regularization and data augmentation.The preserved sub-sequence must retain enough information for categorization.
  • Sequential dropout: Input sequential dropout can create a large or effectively infinite training set in which each training sequence is shown once.
  • Test-time inference: At test time, the model can classify the full sequence directly or ensemble predictions from multiple randomly dropped sub-sequences.The ensemble strategy fuses classification results from the sampled sub-sequences.
  • Experimental setting: The recognition experiments use the ICDAR-2013 database, with training data from CASIA and 3,755 character classes.The dataset contains 2,693,183 training samples and 224,590 test samples produced by different writers.

B. Implementation Details

Implementation uses preprocessed online-character sequences, dropout-based regularization, and comparisons across recurrent architectures, ensemble settings, and network depths.

  • Preprocessing and regularization: Preprocessing uses T_dist = 0.01 × max{H, W} and T_cos = 0.99, producing sequences averaging about 50 elements per character.Dropout probability is 0.1 on the mean-pooling and fully connected layers, and 0.3 on the input sequence.
  • Architecture comparison: The architecture comparison varies recurrent-unit type, stacked bidirectional layers, hidden-state dimensions, full-layer size, and classification-layer size.
  • LSTM versus GRU: LSTM and GRU achieve comparable prediction accuracies, while GRU uses less memory and offers faster training and especially testing.LSTM generally has higher training accuracy but not necessarily higher test accuracy.
  • Network depth: Stacking two or three bidirectional recurrent layers improves training and test accuracy over one layer, but gains are not significant and vanish with further depth.More stacked recurrent layers also increase training and testing time dramatically.

F. Ensemble-based Decision from Sub-sequences

The paper improves online Chinese-character recognition by using sequential dropout to ensemble sub-sequences and by comparing RNN models with prior approaches. Model averaging across six networks achieves the strongest reported accuracy.

  • F. Ensemble-based Decision from Sub-sequences: Sub-sequence ensembles improve classification accuracy as more randomly sampled sub-sequences are included, although one sub-sequence performs worse than the full sequence.The paper attributes the one-sub-sequence deficit to information loss.
  • F. Ensemble-based Decision from Sub-sequences: The discriminative RNN directly processes raw online trajectories rather than transforming them into image-like representations required by prior CNN approaches.The method is end-to-end and uses generic sequential-data priors without domain-specific knowledge.
  • F. Ensemble-based Decision from Sub-sequences: 30 randomly sampled sub-sequences further improve NET4 performance while requiring only one trained model.This saves training time and memory relative to ensembles of separately trained models, but evaluation becomes time-consuming.
  • F. Ensemble-based Decision from Sub-sequences: 98.15% accuracy is achieved by averaging the six pre-trained networks NET1 to NET6.The networks are complementary because they differ in depth and recurrent type.
  • F. Ensemble-based Decision from Sub-sequences: The generative model represents characters as sequences of pen directions and pen states, enabling vector-format drawing.Pen-down, pen-up, and end-of-character states determine which lines are drawn and when generation stops.

B. Conditional Generative RNN Model

The conditional generative RNN uses character embeddings and gated recurrent processing to maintain character-specific writing information and predict subsequent drawing actions.

  • B. Conditional Generative RNN Model: Character embeddings are jointly trained with the generative RNN to distinguish particular Chinese characters from random scrawling.The embedding matrix E contains one d-dimensional vector for each of N character classes.
  • B. Conditional Generative RNN Model: The generative model is built with GRUs because prior experiments found comparable performance to LSTMs with more efficient computation.At each time step, the GRU receives the character embedding and updates its hidden state.
  • B. Conditional Generative RNN Model: The GRU gates control whether previous dynamic writing information is remembered or forgotten in the hidden state.The updated hidden state produces an output used to infer the next pen direction and pen state.
  • B. Conditional Generative RNN Model: The character embedding c is supplied throughout generation to condition the RNN on the identity of the character being drawn.The hidden state encodes the character’s evolving writing information.

C. GMM Modeling of Pen-Direction: From ot To dt+1

The model predicts the next pen direction with a Gaussian mixture model, allowing it to represent varied movement patterns and generate diverse handwriting styles.

  • C. GMM Modeling of Pen-Direction: From ot To dt+1: A 5 × M-dimensional vector derived from the RNN output parameterizes an M-component Gaussian mixture model for the next pen direction.The mixture describes component weights, means, and standard deviations for the predicted movement.
  • C. GMM Modeling of Pen-Direction: From ot To dt+1: The next pen direction is represented as dt+1 = [△xt+1, △yt+1], with x- and y-axis components modeled independently within each mixture component.The independence assumption simplifies the model while producing performance similar to a full bivariate Gaussian model.
  • C. GMM Modeling of Pen-Direction: From ot To dt+1: The Gaussian mixture captures dynamic information from different handwriting styles and permits diverse generated characters.Sampling from the mixture produces alternative pen-direction sequences.

D. SoftMax Modeling of Pen-State: From ot To st+1

A softmax models the next pen state, while cost-sensitive training addresses the imbalance among pen-down, pen-up, and end-of-character events.

  • D. SoftMax Modeling of Pen-State: From ot To st+1: Softmax outputs probabilities for the next pen-down, pen-up, or end-of-character state.These probabilities let the RNN control pen status and determine when generation ends.
  • D. SoftMax Modeling of Pen-State: From ot To st+1: The end-of-character state enables the RNN to automatically stop the drawing process.The representation therefore supports variable-length character generation.
  • D. SoftMax Modeling of Pen-State: From ot To st+1: Direct loss minimization performs poorly because frequent pen-down states dominate the training objective over the rare end-of-character state.The imbalance is especially pronounced because end-of-character occurs only once per character.

F. Automatic Drawing of Recognizable Characters

The conditional generative RNN draws Chinese characters by conditioning recurrent generation on learned character embeddings and sampling pen trajectories until an end-of-character state. Experiments show human-readable, stylistically diverse outputs across 3,755 classes, while some generated characters still omit strokes.

  • Generation process: A selected character-embedding column conditions every generation step, while sampled pen directions and inferred pen states produce the trajectory.Generation continues through the GRU until the end-of-character state is selected.
  • Training setup: The generative model was trained on more than two million cursive CASIA samples representing characters as multiple lines.The data contain varied handwriting habits from different individuals.
  • Training process: By epoch 50, generated examples for the monitored class were human-recognizable, cursive, and stylistically varied; Adam converged in about 60 epochs.Early outputs were unreadable mixtures, whereas later outputs became clearer during training.
  • Character embedding: Jointly trained 500 × 3755 embeddings placed nearest-neighbor characters near classes with similar shapes or substructures.The relationship emerged from handwriting similarity without explicit between-class information.
  • Generated characters: The model generated new characters for nine classes by randomly sampling trajectories, automatically deciding when to end, and producing multiple handwriting styles.The model can draw 3,755 classes and does not simply reproduce training examples.
  • Generated characters: The generator was human-readable, used 33.79MB of memory, and supported effectively unlimited sampling from the learned model.The reported compact generator transforms a database containing more than two million samples into a small sampling model.
  • Limitations: Some generated characters omitted strokes and were difficult to read, so the authors introduced quantitative quality estimation as a next step.The authors explicitly note that generated characters were not perfect.

F. Quality Analysis: Recognizable or Not?

The generative RNN produces Chinese characters that are generally recognizable by machines and readable by humans, while errors concentrate in visually confusable classes. These results support the model’s ability to generate many character classes, but also identify detail-sensitive cases for improvement.

  • Evaluation setup: 3,755 classes were each sampled 100 times, and the discriminative RNN evaluated the generated characters’ recognizability.Both discriminative and generative models were trained on real handwritten characters, using the NET4 recognition architecture.
  • Overall recognizability: Most generated characters were recognized with very high accuracy, verifying the generative model’s ability to write thousands of different Chinese characters.The character embedding and large real-handwriting database distinguish this model from prior LSTM-RNN generation work described in the passage.
  • Overall recognizability: Generated characters were both human-readable and recognizable by the discriminative RNN, including cursive writing produced in vector format.The reported evaluation therefore checks readability from both human and machine perspectives.
  • Quantitative result: 93.98% average classification accuracy was achieved across all generated characters, although this remained below the accuracy for real characters.Some classes produced recognition accuracies below 50%.
  • Error analysis: Recognition errors usually occurred in confusable classes whose characters differ only by subtle shape details.The generative RNN did not consistently capture these small but important distinctions.
  • Future improvement: Non-confusable character classes reached 100% recognition accuracy, whereas confusing pairs remain the main target for quality improvement.Suggested directions include emphasizing confusing class pairs in the loss and adding attention or memory mechanisms.
Loading 1606.06539v1…