Source-linked AI summary
Generating Sequences With Recurrent Neural Networks
Alex Graves
TL;DR
Generating realistic sequences with long-range structure, especially handwriting constrained by text, remains challenging for recurrent models. This paper uses LSTM next-step prediction and annotation-conditioned synthesis to generate diverse, convincing handwriting across styles.
Problem
Prediction networks cannot constrain which letters are written, limiting their ability to synthesize handwriting for given text.
Method
The paper uses LSTM recurrent networks for next-step sequence prediction and adds a convolutional mechanism to condition handwriting predictions on an annotation sequence.
Results
LSTM networks generate discrete and real-valued sequences with complex, long-range structure, while the conditioned model produces diverse, realistic handwriting that can be biased toward legibility and writer style.
Takeaways & Limitations
Next-step prediction with LSTM memory can support realistic sequence generation across text and handwriting, including controllable style and legibility.
Takeaways & Limitations
Automatically extracting high-level annotations from sequence data remains a direction for future work.
Abstract
from arXiv · showhide
This paper shows how Long Short-term Memory recurrent neural networks can be used to generate complex sequences with long-range structure, simply by predicting one data point at a time. The approach is demonstrated for text (where the data are discrete) and online handwriting (where the data are real-valued). It is then extended to handwriting synthesis by allowing the network to condition its predictions on a text sequence. The resulting system is able to generate highly realistic cursive handwriting in a wide variety of styles.
1 Introduction
The paper argues that LSTM networks can generate complex, realistic sequences with long-range structure by overcoming standard RNNs’ limited memory. It introduces deep LSTM prediction for text and handwriting, including style-conditioned synthesis and sampling techniques.
- Motivation: Standard RNNs generate sequences by predicting each next data point probabilistically and iteratively sampling from their output distributions.This framework supports generation in domains including music, text, and motion capture data.
- Motivation: RNNs interpolate between training examples rather than copying exact templates, enabling complex reconstitution and rarely producing the same sequence twice.Their fuzzy predictions are also better suited to real-valued or multivariate data than exact-match methods.
- Motivation: Standard RNNs struggle to retain information over long intervals, limiting long-range modelling and causing instability when generated predictions depend only on recent inputs.The paper identifies this loss of past information as a central limitation of conventional recurrent generation.
- Contribution: LSTM is designed to store and access information more effectively, and the paper demonstrates its ability to generate realistic sequences containing long-range structure.The architecture had already achieved state-of-the-art results in speech and handwriting recognition.
- Paper overview: The paper develops stacked-LSTM next-step prediction, evaluates text generation on Penn Treebank and Hutter Prize Wikipedia, and presents handwriting sampling, priming, and writer-style techniques.Its text model is competitive with state-of-the-art language models and performs nearly as well at character-level as word-level prediction.
2 Prediction Network
The prediction network maps input histories through a deep recurrent architecture to output vectors that parameterize next-input distributions. It uses LSTM hidden layers to model long-range dependencies and trains by backpropagation through time with gradient clipping.
- 2 Prediction Network: The network transforms an input sequence through N recurrent hidden layers into output vectors y_t that parameterize Pr(x_t+1|y_t).The first input x_1 is a null vector, so the network predicts the first real input x_2 without prior information.
- 2 Prediction Network: Skip connections from inputs to all hidden layers and from hidden layers to outputs reduce processing steps and mitigate vanishing gradients.When N = 1, the architecture becomes an ordinary single-layer next-step prediction RNN.
- 2 Prediction Network: The predictive distribution must match the input data, with high-dimensional real-valued density modelling presenting a particular challenge.The sequence probability is trained using its negative logarithm as the sequence loss.
- Long Short-Term Memory: Backpropagation through time computes weight derivatives for gradient-descent training, while input derivatives to LSTM layers are clipped to a predefined range to prevent numerical problems.Full-gradient training can produce excessively large derivatives, motivating the clipping procedure used in all experiments.
- Long Short-Term Memory: LSTM hidden layers use purpose-built memory cells and are better at finding and exploiting long-range dependencies than sigmoid-based hidden functions.The cell uses input, forget, and output gates together with a memory cell and cell-input activation vectors.
3 Text Prediction
Text prediction uses softmax multinomial outputs over one-hot discrete inputs, with character-level modeling chosen for greater generative flexibility despite slightly worse predictive performance. Experiments on Penn Treebank and Wikipedia show that LSTMs learn structure across scales, benefit from dynamic evaluation, and generate locally realistic but globally nonsensical text.
- Representation and granularity: Text is represented as one-hot vectors, and the next-step class distribution is modeled with a softmax output.For K classes, each input has length K with one active entry, and Pr(x_t+1|y_t) is multinomial.
- Representation and granularity: Character-level modeling supports novel words and strings, so the experiments target the finest data granularity despite slightly worse performance than word-level models.Word-level vocabularies can exceed 100,000 classes, increasing parameter and data requirements.
- Penn Treebank experiments: The Penn Treebank study compared 1000-unit word- and character-level LSTMs, with approximately 54M versus 4.3M weights, respectively.The comparison was acknowledged as somewhat unfair because the word-level network had many more parameters.
- Penn Treebank experiments: Bits-per-character and perplexity were used as equivalent evaluation metrics, and the word-level RNN outperformed the character-level network on the Penn corpus.Perplexity is approximately 2^5.6BPC because the test-set average word length is about 5.6 characters.
- Wikipedia experiments: Dynamic evaluation substantially improved Wikipedia performance by adapting weights during prediction to long-range topical coherence.The network updates its weights as validation data is predicted, helping capture article-specific word frequencies.
- Generated text: Generated Wikipedia text exhibits vocabulary, punctuation, balanced delimiters, non-Latin characters, internet-address patterns, and coherent regions spanning hundreds or thousands of timesteps, but lacks meaning beyond short phrases.The model also invents feasible-looking words and names, while generated large-scale regions are shorter and more jumbled than the real data.
4 Handwriting Prediction
The paper applies recurrent prediction to online handwriting, representing pen motion and stroke endings with a mixture-density output model. The resulting networks learn realistic strokes, letters, short words, and plausible character-level sequences, with adaptive weight noise improving validation loss.
- Data representation: Online handwriting is modeled as sequences of two-dimensional pen offsets plus a binary end-of-stroke feature.The data come from IAM-OnDB, containing handwritten lines from 221 writers recorded with a smart whiteboard.
- Predictive distribution: The network predicts pen offsets with a mixture of bivariate Gaussians and stroke endings with a Bernoulli distribution.Each output includes mixture means, standard deviations, correlations, weights, and an end-of-stroke probability.
- Model configuration: 20 mixture components model the offsets, producing 120 mixture parameters per timestep alongside a parameter for the end-of-stroke feature.The network input layer has size 3, and coordinate offsets are normalized to mean 0 and standard deviation 1 over the training set.
- Validation results: 15.3 nats lower average per-sequence loss was achieved by the three-layer network than by the one-layer network, although SSE was slightly lower for the single-layer network.Adaptive weight noise reduced loss by another 16.7 nats relative to the unregularised three-layer network without significantly changing SSE.
- Generated handwriting: Generated samples model strokes, letters, short words, and plausible invented English-looking words, with average characters spanning more than 25 timesteps.The samples particularly capture common words such as ‘of’ and ‘the’.
5 Handwriting Synthesis
The handwriting synthesis network conditions pen-trace predictions on text through a learned soft window that dynamically aligns characters with writing. It generates stylistically varied and sometimes human-indistinguishable handwriting, while remaining susceptible to alignment errors and readability issues.
- Alignment: The main conditioning challenge is aligning text and pen traces whose lengths differ substantially and whose correspondence is unknown during generation.Pen traces average twenty five times the text length, while character coordinate counts vary with style, size, and pen speed.
- Alignment: The network dynamically learns which character to write next by using a soft window whose location advances through positive offsets from its previous position.Using offsets rather than absolute locations was essential for aligning the text with the pen trace.
- Architecture: The synthesis model augments the prediction network with character-sequence input mediated by a window layer.The window vectors feed the hidden layers while preserving the prediction network’s stacked architecture and skip connections.
- Results: 31.3 nats of adaptive-weight-noise improvement in log-loss produced no significant sum-squared-error change, while the synthesis networks reduced sum-squared error by 44% versus the best prediction network.The regularised network appeared slightly more realistic, although the visual difference was difficult to discern.
- Sampling and style: Unbiased sampling can yield handwriting with varied but internally consistent stylistic traits, and some samples were reportedly indistinguishable from real handwriting by human observers.The network nevertheless produced missing, confused, or garbled letters, with more mistakes for less common words.
- Sampling and style: Combining primed and reduced-variance sampling preserves slant and cursiveness while making strokes smoother and more regular.The paper suggests artificial enhancement of poor handwriting as a possible application.
6 Conclusions and Future Work
The paper demonstrates that LSTM recurrent networks can generate discrete and real-valued sequences with complex long-range structure, including diverse, realistic handwriting conditioned on annotations. Future work includes speech synthesis, understanding and manipulating internal representations, and automatically extracting richer annotations.
- Conclusions: LSTM recurrent networks generate discrete and real-valued sequences with complex, long-range structure through next-step prediction.The paper applies this approach to both discrete text and real-valued online handwriting.
- Conclusions: A convolutional mechanism lets recurrent networks condition predictions on auxiliary annotation sequences, enabling diverse and realistic online-handwriting synthesis.The mechanism could support annotations beyond text, including stylistic features, alternate letter forms, and stroke-order information.
- Future Work: Future work includes applying the network to speech synthesis, which is expected to be more challenging than handwriting synthesis because speech data points have greater dimensionality.The paper also identifies better understanding and direct manipulation of the internal data representation as a future direction.
- Future Work: Another proposed direction is automatically extracting high-level annotations from sequence data.This would extend conditioning beyond manually supplied annotation sequences.