DeepWriting: Making Digital Ink Editable via Deep Generative Modeling

INTRODUCTION

• The model separates style from content to synthesize and edit handwriting while preserving personalized appearance; the work also contributes a character-annotated dataset and preliminary evaluation.The system supports typed-text synthesis and word-level editing, with applications including spell-checking and auto-correction.Source: INTRODUCTION · pp. 1–2

RELATED WORK

• The paper situates its work across HCI, handwriting analysis, and machine learning.Source: RELATED WORK · p. 2

Understanding Handwriting

• Prior handwriting-style research lacks an equivalent parametric model, and style analysis remains inexact; this work learns a latent style representation for writer-specific text.Source: Understanding Handwriting · p. 2

Pen-based interaction

• Pen-based systems often preserve individual style by avoiding character conversion, while editable ink can be interchanged with characters for processing, searching, and editing.Source: Pen-based interaction · p. 2

Handwriting Beautification

• Earlier synthesis can lack content control or imitate styles from training data; DeepWriting transfers style from one sample to arbitrary text.Unlike geometric-averaging beautification, it needs only one input sequence and retains the author’s global style.Source: Handwriting Beautification · pp. 2–3

METHOD

• Figure 3 presents inference of style and content from a sample, synthesis from those components, and reconstruction-based learning.Source: Figure 3 · p. 3
• The model represents ink as temporally ordered strokes and learns by reconstructing samples while predicting character, end-of-character, and beginning-of-word labels.These auxiliary predictions support character recognition, segmentation, and word-spacing control.Source: METHOD · p. 3
• C-VRNN combines recurrent variational modeling with conditioning on user-specified content to synthesize handwriting with natural variation.It distinguishes content from style using latent variables, supporting synthesis rather than the unconditioned generation of standard VRNNs.Source: METHOD · p. 3

Background

• The background identifies recurrent neural networks and VRNNs as related work and uses superscripts for layer information and subscripts for time steps.Source: Background · pp. 3–4

Recurrent Neural Networks

• RNNs predict each next sequence element from the current input, with a recurrent state carrying information from preceding steps.The output function maps hidden state to a probability distribution, from which the output is sampled.Source: Recurrent Neural Networks · p. 4
• Multi-layer LSTMs with GMM outputs support conditional handwriting synthesis but cannot disentangle style from content without latent random variables.Source: Recurrent Neural Networks · p. 4

Variational Recurrent Neural Networks

• VRNNs introduce time-step latent variables z to increase model expressiveness and capture data variability through a learned prior.The transition function takes samples z_t as input, adding a source of variability.Source: Variational Recurrent Neural Networks · p. 4
• Because the true posterior is intractable, VRNNs use an approximate posterior and jointly optimize a variational lower bound.The reconstruction term trains latent samples to reconstruct inputs, while the KL term aligns posterior and prior for later synthesis; the inference network can expose latent properties for transformations such as changing slant.Source: Variational Recurrent Neural Networks · p. 4
• The KL term also encourages the latent space to capture information shared across inputs.Source: Variational Recurrent Neural Networks · p. 4

• Prior recurrent approaches lack inference networks that decouple handwriting style from content, limiting control over generated text.Source: Conditional Variational Recurrent… · pp. 4–5
• Separate latent variables z and π represent handwriting appearance and content, respectively, with π_t specifying synthesized characters.Source: Conditional Variational Recurrent… · p. 5
• The approximate inference distribution factorizes as q(z_t,π_t|x_t)=q(z_t|x_t)q(π_t|x_t) to infer style and content from an input sample.Source: Conditional Variational Recurrent… · p. 5
• The output function uses z and π, rather than the internal cell state h, to generate strokes from the latent variables.Source: Conditional Variational Recurrent… · p. 5

High Quality Digital Ink Synthesis

• Separating continuous variation from character content may support individual-character synthesis, but full handwriting also requires word-spacing control and better prediction quality.Source: High Quality Digital Ink Synthesis · p. 5

Character classification loss

• Character labels supervise the content component with an added cross-entropy classification loss to prevent training divergence.Source: Character classification loss · p. 5
• Figure 6 contrasts π-only synthesis using one-hot character labels, which causes pen-up and placement problems, with synthesis using the model's GMM latent space.The figure also shows input samples used to infer style.Source: Figure 6 · p. 6

GMM latent space

• The content variable selects a character-specific Gaussian component, and reparameterized samples allow gradients to train the GMM parameters.The GMM parameters are time-invariant, unlike the dynamic style variable z_t.Source: GMM latent space · p. 6
• Similar letters lie closer in the continuous content latent space, while dissimilar letters and infrequent symbols are pushed farther apart.The t-SNE visualization shows separate letter clusters, but its plotted positions do not reflect the true latent space.Source: GMM latent space · p. 6, Figure 7 · p. 6

Word spacing and character limits

• The model uses bow to learn when to leave a word space and eoc to predict when to stop synthesizing a character.Both signals are derived from ground-truth character-level segmentations; eoc is included in the loss as a Bernoulli log-likelihood.Source: Word spacing and character limits · p. 6

Input RNN cell

• A second LSTM cell at the input layer adds capacity and lets the synthesis model capture and modulate input-level temporal patterns.The authors motivate this design by handwriting's temporal consistency, in which the previous letter influences the current letter's appearance.Source: Input RNN cell · p. 6
• During training, the model reconstructs strokes and predicts character and eoc labels before evaluating the loss and updating its parameters.The training algorithm describes reconstructed strokes and predicted character and eoc labels as outputs.Source: Input RNN cell · p. 7
• During sampling, the model draws style and content samples, uses bow to mark new words, and synthesizes strokes while predicting eoc to advance characters.The sampling algorithm can initialize the style state from a reference sample for style transfer.Source: Input RNN cell · p. 7

Character recognition

• 96% validation accuracy for the BiRNN exceeds the standard LSTM's 60% on character recognition.The BiRNN processes input in both directions, requiring the entire sequence at once.Source: Character recognition · p. 7
• A separate BiRNN guides disentanglement at sampling time; full integration of high-accuracy character recognition is left for future work.The model's q(π|x) also infers character labels, but the BiRNN's accuracy improves synthesis quality.Source: Character recognition · p. 7

APPLICATION SCENARIOS

• The paper presents these applications as proof-of-concept implementations of generating and editing digital ink in user-defined styles.The authors describe disentanglement of content and style as making digital ink editable.Source: APPLICATION SCENARIOS · p. 7

Conditional Handwriting Generation

• Users can type text and select an author's handwriting sample to generate personalized writing; Figure 8 shows abstract sentences in different authors' styles.The authors suggest personalized handwritten notes or letters as possible uses.Source: Conditional Handwriting Generation · p. 8, Figure 8 · p. 8

Content Preserving Style Transfer

• The model transfers existing handwriting to a selected style while preserving its content, with results shown in Figure 9.An interactive tablet application lets users select a reference style and see their strokes morphed in real time.Source: Content Preserving Style Transfer · p. 8, Figure 9 · p. 8
• With the user's own style as the target, the model re-generates smoother strokes while maintaining natural variability and diversity.This averaging effect suppresses local noise while preserving global style features.Source: Content Preserving Style Transfer · p. 8
• Users can change, delete, or replace individual handwritten words and regenerate them in the original style, as illustrated in Figure 10.The prototype recognizes words and characters and renders them as editable overlays.Source: Content Preserving Style Transfer · p. 8, Figure 10 · p. 8
• The spell-checking proof of concept detects spelling mistakes and synthesizes corrections in the original style, as shown in Figure 11.The implementation feeds recognized characters to existing spell-checking APIs and re-renders retrieved corrections.Source: Content Preserving Style Transfer · pp. 8–9, Figure 11 · p. 9

PRELIMINARY USER EVALUATION

• Ten participants evaluated beautification and spell-checking; beautification used 12 writing trials per participant, while spell-checking asked them to correct misspelled words without changing style.The study used an iPad Pro and Apple Pencil, and collected quantitative and qualitative feedback across the two tasks.Source: PRELIMINARY USER EVALUATION · p. 9
• M = 3.65, 95% CI [3.33-3.97] for beautified strokes; SUS = 85, classified as Rank A.The beautified strokes had non-overlapping confidence intervals with participants’ original handwriting; the authors report that Rank A indicates users are likely to recommend the system.Source: Figure 12 · p. 9, PRELIMINARY USER EVALUATION · p. 9

THE HANDWRITING DATASET

• The dataset adds character-level labels because sequence-level annotations are too coarse for disentangling handwriting content and style.Prior IAM-OnDB data was captured with a low-resolution, low-accuracy digital whiteboard and had sequence-level annotations.Source: THE HANDWRITING DATASET · p. 9
• The unified dataset contains 294 authors, 85,181 word instances, and 406,956 handwritten characters.The reported writer medians are 292 words and 1,349 characters.Source: THE HANDWRITING DATASET · p. 9, Table 1 · p. 10
• The corpus combines newly collected data from 94 authors with IAM-OnDB samples from 200 authors, segmented to character labels and manually cleaned.The combined collection spans multiple digitization mechanisms, intended to support learning across input types.Source: THE HANDWRITING DATASET · pp. 9–10

CONCLUSION AND FUTURE WORK

• C-VRNN represents style and content as separate learned latent random variables, allowing either or both to be changed during sampling.The model also uses auxiliary tasks for spacing, character segmentation, and recognition to produce realistic samples.Source: CONCLUSION AND FUTURE WORK · p. 10
• Proof-of-concept applications demonstrate conditional synthesis and word-level editing, with initial feedback largely positive about editing in one’s own or another author’s style.The conclusion notes that sampling can also generate entirely new sequences.Source: CONCLUSION AND FUTURE WORK · p. 10
• The authors release the implementation as open-source and contribute a consolidated character-annotated dataset.The dataset combines existing and newly collected handwritten text; the paper states that data and code are publicly available.Source: CONCLUSION AND FUTURE WORK · p. 10, Figure 14 · p. 10
• The model generates both disconnected and cursive styles, but performs better on disconnected handwriting; full cursive support is future work.The authors also plan to integrate the auxiliary character-recognition network into the architecture and identify a full language model as a possible direction.Source: CONCLUSION AND FUTURE WORK · p. 10

Alphabet

• The handwriting alphabet includes digits, lowercase and uppercase letters, and punctuation symbols.The listed punctuation includes prime, period, comma, minus, parentheses, and slash.Source: Alphabet · p. 14

Data Preprocessing

• Preprocessing splits long samples at end-of-character labels, shifts each sample to start at (0,0), uses relative coordinate changes, and normalizes coordinates.Samples longer than 300 strokes are split; normalization uses training-set mean and standard deviation.Source: Data Preprocessing · p. 14

Network Configuration

• The LSTM cells use 512 units, the latent variables use 32-dimensional isotropic Gaussians, and the bidirectional LSTM classifier has three 512-unit layers.The classifier’s fully connected output layer has 256 units.Source: Network Configuration · p. 14

Training

• Training uses ADAM for 200 epochs with batch size 64 and an initial learning rate of 0.001, decayed by 0.96 every 1,000 mini-batches.The optimizer uses its default parameters.Source: Training · p. 14

Additional Results

• The model synthesizes the paper’s entire abstract with a different handwriting style for each sentence, illustrating generation across varied styles.The result is shown in Figure 15.Source: Additional Results · p. 14