Source-linked AI summary
Handwriting Trajectory Recovery via Autoregressive Ordered Stroke Instance Prediction
En-Guang Wang, Yan-Ming Zhang, Fei Yin, Cheng-Lin Liu
TL;DR
Offline handwriting omits temporal information needed to recover the writing process, while existing methods often underuse its stroke-level organization. The paper proposes autoregressive ordered stroke prediction followed by within-stroke motion reconstruction, with direction-related cues supporting initialization. Experiments report better ordered and full-point recovery than compared alternatives, alongside generalization across character categories and languages and a strong sampling-density effect.
Problem
Static handwritten images lose stroke order, writing direction, and continuous pen-tip motion, while direct trajectory prediction does not explicitly exploit stroke-level organization.
Method
A two-stage framework autoregressively predicts ordered stroke instances with writing-history conditioning, then generates within-stroke trajectories using direction-related start-point initialization.
Results
The ordered formulation outperforms post-hoc ordering for stroke-order and downstream trajectory recovery, while full-point recovery is numerically better than compared baselines and generalizes across categories and languages.
Takeaways & Limitations
Modeling stroke order during instance formation is more effective than post-hoc ordering, and the framework extends beyond seen Chinese categories to English and Tamil handwriting.
Takeaways & Limitations
Trajectory sampling density substantially influences measured recovery performance, and trajectory simplification can produce more favorable recovery scores.
Abstract
from arXiv · showhide
Handwriting trajectory recovery aims to infer the dynamic writing process hidden behind a static handwritten image. Since offline handwriting preserves only the final spatial ink pattern, temporal information such as stroke order, writing direction, and pen-tip motion is lost, making recovery inherently ambiguous. Existing learning-based methods often directly predict the complete character trajectory without explicitly exploiting the stroke-level organization of handwriting. We argue that recovering the writing process should follow the writing process itself. Accordingly, we propose a two-stage framework that first recovers ordered stroke instances and then reconstructs continuous within-stroke motion. The first stage integrates stroke extraction and stroke-order recovery through autoregressive ordered stroke prediction, while direction-related structural cues further support within-stroke trajectory generation. Experiments on Chinese handwriting show that the proposed ordered prediction is more effective than post-hoc stroke ordering. Even without trajectory simplification, our full-point model achieves numerically better results than those reported by all compared baselines, while a controlled analysis shows that trajectory sampling density substantially affects measured recovery performance. Additional experiments demonstrate generalization to unseen Chinese character categories and cross-language extensibility to English and Tamil handwriting.
1. Introduction
Offline-to-online recovery must infer an ambiguous writing process from static ink, including stroke structure, order, direction, and continuous motion. The proposed framework follows that hierarchy by predicting ordered strokes autoregressively before generating within-stroke trajectories.
- Motivation: Offline images lose stroke order, writing direction, and continuous pen-tip motion, making recovery of the underlying writing process inherently ambiguous.Ambiguity is especially severe in multi-stroke characters with intersections, contacts, overlaps, or cursive connections.
- Research gap: Direct whole-character generation leaves stroke parsing, inter-stroke ordering, and long-sequence generation largely implicit, while extract-then-order methods separate these operations.The separation can prevent writing history from participating in stroke-instance formation.
- Results: The method improves stroke-order and downstream trajectory recovery over post-hoc ordering, with numerically better full-point results than compared baselines across DTW, LDTW, AIoU, and LPIPS.It also generalizes to unseen Chinese categories and extends to English and Tamil, while sampling density substantially affects measured performance.
- Method: The framework decomposes recovery into ordered stroke instance prediction followed by continuous within-stroke trajectory generation.This decomposition mirrors the hierarchy in which pen-tip movements form strokes and ordered strokes form characters.
- Method: Autoregressive ordered prediction conditions each new stroke on the preceding writing state, jointly integrating stroke extraction and stroke-order recovery during instance formation.This replaces unordered set prediction followed by post-hoc ordering.
- Method: Direction-aware start and end points provide structural supervision, and the predicted start point initializes within-stroke trajectory generation before recovered strokes are concatenated in order.The second stage reconstructs subsequent pen-tip motion from the initialized writing origin.
2. Related Work
Prior work progressed from structure-driven reconstruction to learning-based trajectory generation, but stroke structure and temporal order remain difficult to integrate. The paper motivates autoregressive ordered stroke instances as a joint alternative to spatial decomposition followed by post-hoc ordering.
- Structure-driven methods: Early methods recovered writing paths from skeletons, graphs, or hierarchical character structures using explicit structural reasoning.These approaches addressed plausible paths and stroke order through optimization, graph search, or character constraints.
- Learning-based methods: Deep learning methods shifted toward data-driven sequence generation, including next-position prediction, image-to-sequence recovery, modality conversion, and Transformer-based long-trajectory modeling.Later methods combined spatial encoding, temporal decoding, or vision-language priors.
- Remaining gap: Many learning-based formulations still predict complete character trajectories, leaving stroke extraction, ordering, and within-stroke motion entangled in character-level decoding.This motivates explicit stroke-level organization before complete trajectory generation.
- Stroke ordering: Decomposed multi-stroke elements remain unordered, so spatial stroke structure alone does not determine the writing process or temporal sequence.This issue is particularly relevant to handwritten Chinese characters requiring direction and inter-stroke order reasoning.
- Modeling perspective: Set-to-sequence research establishes autoregressive modeling as a natural approach for mapping unordered elements to order-sensitive outputs.Pointer Networks specifically support variable-length permutation problems by autoregressively outputting input indices.
- Handwriting ordering: Existing handwriting ordering work applies prediction to pre-extracted segments and then uses heuristic search, exemplifying the extract-then-order paradigm.This separation limits interaction between parsing and ordering, allowing extraction errors to propagate into ordering.
- Stroke segmentation: Segmentation studies address spatial stroke structure through semantic, instance-level, and geometric cues, but such cues do not directly encode writing direction.Generic query-based detectors and segmenters also represent instances as unordered predictions.
- Proposed direction: Autoregressive ordered stroke instance prediction conditions formation of the next stroke on preceding writing state, integrating stroke parsing and order recovery.This directly addresses the ordering gap in conventional query-based instance prediction.
3. Method
The method recovers handwriting hierarchically: autoregressive decoding first predicts ordered stroke instances, then reconstructs continuous motion within each stroke before concatenating the results.
- 3.1. Overview: Each predicted stroke instance contains a mask, bounding box, validity confidence, and start/end points.The start and end points provide direction-related structural information.
- 3.1. Overview: The framework converts an offline image into ordered stroke instances, recovers each stroke's trajectory, and concatenates the stroke trajectories in writing order.This separates inter-stroke organization from within-stroke motion.
- 3.2. Autoregressive Ordered Stroke Instance Prediction: Autoregressive ordered stroke prediction conditions formation of the current stroke on the complete image and previously recovered writing history.Unlike unordered set prediction, stroke parsing and order recovery occur jointly during instance formation.
- 3.2. Autoregressive Ordered Stroke Instance Prediction: Previously recovered strokes are encoded with spatial features and stroke-index embeddings, then interact with multi-scale character features to decode the next stroke.The writing history represents recovered content and position, while character features provide remaining spatial evidence.
- 3.3. Within-Stroke Trajectory Generation: The predicted start point initializes within-stroke generation, which uses serial coarse-to-fine multi-scale decoding to recover subsequent pen-tip motion.Coarse features capture global geometry and continuation tendency; finer features refine bends and spatial locations.
4. Experiments
Experiments evaluate the two-stage framework on Chinese handwriting, testing ordered stroke recovery, complete trajectory reconstruction, sampling density, start-point conditioning, unseen categories, and cross-language transfer.
- Experimental setup: CASIA-OLHWDB1.1 provides rendered offline images, ordered stroke annotations for Stage 1, and individual stroke trajectories for Stage 2.The annotations include masks, bounding boxes, stroke order, and start/end points.
- Experimental setup: The evaluation uses strict character-level Order Acc., which requires exact stroke matching and complete agreement with the ground-truth writing order.Missed, redundant, split, merged, or incorrectly ordered strokes make a character incorrect.
- Stroke extraction and order recovery: Our joint autoregressive formulation obtains the highest strict character-level Order Acc., although MaskDINO and Mask2Former achieve higher Mask AP50.The complete pipeline also achieves the best overall downstream trajectory-recovery performance among compared systems.
- Complete character trajectory recovery: The full-point model retains all original trajectory samples and achieves numerically better results than all compared baselines across all four evaluation metrics.This comparison uses the complete two-stage pipeline without trajectory simplification.
- Trajectory sampling density: RDP = 3 and stronger trajectory simplification improve sequence-alignment results, showing that sampling density substantially affects measured recovery performance.The controlled analysis uses ground-truth strokes and ground-truth writing order to isolate sampling-density effects from first-stage errors.
- Start/end point supervision and initialization: Start/end supervision improves sequence alignment without changing aggregate Mask AP, while transferring the predicted start point to Stage 2 further improves trajectory recovery.The first-stage start point supplies directional information unavailable when the second stage predicts the writing origin from an isolated stroke.
- Generalization and cross-language evaluation: On unseen Chinese categories and English handwriting, the method achieves numerically better results than compared baselines, while Tamil evaluation shows adaptation after target-language fine-tuning.The unseen Chinese set contains 3,025 categories and 7,900 test samples; English uses different test writers.
5. Conclusion
The framework recovers handwriting as a process by predicting ordered stroke instances before generating within-stroke motion. Experiments support this formulation, while showing sensitivity to trajectory sampling density and transfer across categories and languages.
- The method decomposes recovery into autoregressive ordered stroke-instance prediction followed by within-stroke trajectory generation.This integrates stroke extraction with stroke-order recovery and uses predicted start points to determine the writing origin for subsequent motion generation.
- Start and end points provide direction-related structural constraints, and predicted start points transfer across stages to reduce writing-direction ambiguity.
- Autoregressive ordering outperforms post-hoc ordering for stroke-order recovery and downstream trajectory recovery on Chinese handwriting.
- The full-point model achieves numerically better results than all compared baselines even without trajectory simplification.
- Trajectory sampling density substantially influences measured recovery performance, while experiments show generalization to unseen Chinese categories and extensibility to English and Tamil handwriting.
CRediT authorship contribution statement
The authors contributed across conceptualization, methodology, implementation, data, analysis, validation, supervision, and writing.
- En-Guang Wang contributed conceptualization, methodology, software, data curation, investigation, formal analysis, validation, visualization, and the original draft.
- Yan-Ming Zhang contributed conceptualization, methodology, supervision, and review and editing of the manuscript.
- Fei Yin and Cheng-Lin Liu contributed supervision, with Liu also contributing review and editing and Yin project administration.