PaperlayerView exact revision on arXiv

Source-linked AI summary

inkGround: Benchmarking Fine-grained Visual Grounding Capabilities of Vision-Language Models for Cursive Handwriting in the Wild

Sidra Hanif

arXiv:2610.38720v1cs.CVcs.AI

TL;DR

Full-page handwriting images preserve ink but not the ordered pen movements, and existing paired datasets largely come from controlled tablet settings. The paper introduces inkGround and evaluates VLMs by prompting them to locate a word and recover its strokes; closed-source models produce readable trajectories, with the ranking also holding on recorded IAM-OnDB trajectories.

  • Problem

    Existing trajectory-pair datasets come from controlled settings, and no benchmark evaluates VLM stroke grounding on full-page camera-captured cursive handwriting.

    Source: §1 · pp. 1–2

  • Method

    inkGround pairs full-page camera-captured handwriting with OCR-filtered, human-validated pseudo-label trajectories and evaluates VLMs using text-conditioned point grounding.

    Source: §1 · pp. 1–2

  • Results

    Closed-source VLMs produce readable trajectories but trade readability against style imitation, and the model ranking also holds on IAM-OnDB recorded ground truth.

    Source: §6 · p. 8, §7 · p. 9

  • Takeaways & Limitations

    Stroke trajectory recovery remains an open challenge, with no tested VLM addressing all task facets.

    Source: §6 · p. 8

    inkGround references are pseudo-labels for one plausible writing order, not recorded ground truth of the writer’s original stroke order or direction.

    Source: §4.2 · p. 5

Abstract

from arXiv · show

Human handwriting is produced as an ordered sequence of pen movements, but an offline image keeps only the final ink. Handwriting trajectory recovery aims to restore the ordered stroke points and their pen-up/pen-down transitions. Existing image-trajectory pair datasets are recorded with digitizing tablets under controlled conditions, which limits writer coverage and stylistic diversity. Moreover, there is no benchmark that measures the visual grounding capabilities of general-purpose Vision-Language Models (VLMs) for the stroke trajectory recovery task. In this work, we introduce inkGround, a benchmark for stroke trajectory recovery from full-page, camera-captured handwriting. inkGround pairs 537 page images (~18.7k words, ~163k strokes) with reference trajectories produced by a pretrained offline-to-online model, filtered by OCR agreement, and verified by human raters. Because these references are pseudo-labels that represent one plausible pen trajectory, we also evaluate VLMs on the IAM-OnDB dataset, which provides recorded pen trajectories. We cast the task as text-conditioned point grounding: given a page image and a target word, the model must locate the word and output its ordered stroke points. Outputs are scored by trajectory fidelity (DTW, Chamfer distance), readability of the rendered strokes (Accurate Rate, Correct Rate), and style similarity. We benchmark four open-source and three frontier closed-source VLMs with a shared in-context prompt. Closed-source models, notably Gemini-3.1-Pro, produce legible trajectories close to the reference, and this ranking also holds on IAM-OnDB.

1. Introduction

Handwriting trajectory recovery restores ordered pen movements and stroke transitions that static images omit. This paper asks whether VLMs can ground a target word in a full-page image and recover its ordered stroke points, introducing inkGround to evaluate that task.

1. Introduction

  • Stroke trajectories encode stroke order, pen-up/pen-down transitions, and writing dynamics, supporting applications such as handwriting biometrics, pen order prediction, and style imitation.The cited applications are handwriting biometrics, pen order prediction [3], and handwriting style imitation [14] [21].Source: §1 · p. 1
  • The task asks VLMs to locate a prompted word in a full-page image and output its ordered stroke points.The prompt includes an example image with its strokes.Source: §1 · p. 1
  • Existing trajectory-pair datasets use controlled settings, while no benchmark evaluates VLM stroke grounding on full-page camera-captured cursive handwriting.The paper frames this as a gap between controlled tablet recordings and in-the-wild page images.Source: §1 · p. 2
  • inkGround pairs 537 full-page camera-captured images with reference trajectories, filtered by OCR agreement and validated by human raters.The benchmark contains approximately 18.7k words and 163k strokes; each pair is rated by at least two raters.Source: §1 · p. 2
  • The evaluation combines trajectory fidelity, rendered-stroke readability, and style similarity across seven VLMs on inkGround and IAM-OnDB.The tested models include four open-source and three closed-source VLMs.Source: §1 · p. 2

2. Related Work

Prior work grounds language to a few image locations and recognizes handwriting text, while trajectory recovery requires long ordered point sequences. Existing recovery methods and datasets leave a gap for evaluating general-purpose VLMs on full-page, in-the-wild handwriting.

2.1. Visual grounding in VLMs

  • Visual-grounding systems predict boxes, regions, or a few points, whereas stroke recovery requires a long ordered sequence overlapping one word’s ink.Examples include segmentation-oriented models GLaMM [26] and LISA [16], free-form localization with Ferret [34], and coordinate outputs from Qwen3-VL [4] and Molmo [8].Source: §2.1 · pp. 2–3

2.2. Handwriting recognition

  • Document-level recognition systems read handwriting reliably but recover transcription rather than pen trajectory.The paper cites PaddleOCR [7] and DeepSeek-OCR [31] as document-level systems, and uses DeepSeek-OCR to filter reference trajectories.Source: §2.2 · p. 3

2.3. Stroke trajectory recovery

  • Stroke-recovery methods include recurrent encoder–decoders, differentiable rendering, and InkSight, but are trained per dataset and script on lines or isolated characters.The cited approaches include [5], [3], InkSight [22], and Chinese character reconstruction.Source: §2.3 · p. 3

2.4. Handwriting datasets

  • Tablet-based datasets provide recorded trajectories but mostly cover limited writers and isolated lines or characters; in-the-wild image datasets lack trajectories.inkGround combines full-page camera-captured images with reference trajectories, and VLMs are evaluated on both inkGround and IAM-Online.Source: §2.4 · p. 3

3. inkGround

inkGround is a benchmark of full-page camera-captured cursive handwriting paired with transcriptions and reference trajectories. Its pages range from short notes to dense messages, with many words and strokes across the collection.

3.1. Overview

  • 537 camera-captured pages of personal notes and greeting-card messages vary in writing style, layout, and background, and each is paired with a transcription and reference trajectory.The trajectory records pen coordinates and a start-of-stroke token; references are pseudo-labels produced and validated through the described pipeline.Source: §3.1 · p. 3

3.2. Statistical Analysis

  • ≈18.7k words and ≈163.4k strokes span 537 pages, averaging 34.89 words and 304.3 strokes per page.Pages range from 2 to 140 words; the average stroke contains 49 points.Source: §3.2 · p. 4
  • The 20 most frequent words include “you,” “I,” “the,” and “and”; “love” is also frequent in the personal-note and greeting-card pages.The word-frequency distribution is shown in Figure 3(b).Source: §3.2 · p. 4

4. Benchmark Construction Pipeline

inkGround builds full-page handwriting trajectories through pseudo-label generation, OCR filtering, and human validation. The retained trajectories reproduce text and appearance but represent plausible, not recorded, stroke orders.

4. Benchmark Construction Pipeline

  • The pipeline first renders InkSight-generated pseudo-labels and filters them by OCR agreement, then asks human raters to verify that trajectories overlap the handwriting ink.The human validation stage follows OCR filtering.Source: §4 · p. 4

4.1. Stage 1: Pseudo-label generation and filtering

  • InkSight Small-p segments each camera-captured page into words, predicts stroke tokens, and places the resulting trajectories back in page coordinates.Complex layouts and overlapping strokes can lead to missed or misaligned strokes.Source: §4.1 · p. 5
  • Pages were retained only when OCR transcriptions of the original and rendered trajectories matched at least 85% of words; 777 of 2,984 candidate pages passed.The threshold was chosen empirically to balance pseudo-label quality against dataset size.Source: §4.1 · p. 5

4.2. Stage 2: Human validation

  • 537 of 777 OCR-filtered page pairs were retained after independent raters unanimously gave each pair a perfect similarity score.Raters agreed on accept/reject decisions for 81.7% of pairs.Source: §4.2 · p. 5
  • OCR filtering and human review verify text and visual appearance, not the writer’s original stroke order or direction, so references represent plausible trajectories rather than recorded ground truth.Several different trajectories can render to a similar image.Source: §4.2 · p. 5

5. Experiments

The experiments compare seven VLMs on inkGround and recorded IAM-OnDB trajectories using a shared prompt and measures of trajectory fidelity, readability, and style. Closed-source models rank ahead of open-source models on both datasets, with tradeoffs between readability and style imitation.

5.1. Evaluation Metrics

  • DTW and Chamfer distance assess trajectory similarity, AR and CR assess rendered-text readability, and a writer-identification classifier provides the style score.Style scores are comparable within a table, but not between inkGround and IAM-OnDB because their classifiers use different writer sets and score ranges.Source: §5 · p. 5, §5.1 · p. 5

5. Experiments

  • The evaluation gives models a full-page image and an in-context example, then asks them to output ordered stroke points for a target word.The evaluation covers inkGround pseudo-label references and IAM-OnDB recorded trajectories.Source: §5 · p. 5, §5.2 · pp. 6–7

5.2. Inference protocol and output validity

  • All models receive the same system prompt, task prompt, and in-context example; outputs that cannot be parsed are invalid, while outputs covering only part of the words are incomplete.Invalid outputs receive zero AR, CR, and Style, and maximum DTW and Chamfer distance.Source: §5.2 · p. 6

5.3. Model evaluation

  • The tested open-source models’ inkGround rendered strokes are not readable: their AR and CR remain below 0.08, and their DTW and Chamfer distances exceed those of closed-source models.This conclusion applies under the shared prompting setup.Source: Table 1 · p. 6, §5.3 · p. 8
  • On inkGround, Gemini-3.1-Pro has the highest readability (AR 0.812, CR 0.826), while Kimi-K3 has the highest style score and lowest readability (AR 0.581, CR 0.591).Claude-Opus-4.8 produces readable text but shows the weakest style imitation; Fig. 5 shows rendered examples.Source: Table 1 · p. 6, Figure 5 · p. 7, §5.3 · p. 8
  • On recorded IAM-OnDB trajectories, Gemini-3.1-Pro ranks ahead of Claude-Opus-4.8 and open-source models, matching the inkGround ranking and showing it is not an artifact of pseudo-label references.Gemini-3.1-Pro has DTW 0.088, Chamfer 0.047, and AR 0.856; the ranking comparison is reported across datasets.Source: Table 2 · p. 6, §5.3 · p. 8

6. Discussion

The tested models leave full-page handwriting trajectory recovery open: closed-source systems recover readable strokes but trade off readability and style, while no tested VLM addresses every task facet.

6. Discussion

  • Closed-source VLMs trade style imitation against readability, and none of the tested models addresses all facets of trajectory recovery.Gemini-3.1-Pro has AR 0.812 and CR 0.826, with room for improvement.Source: §6 · p. 8
  • InkGround references reproduce text and appearance but cannot establish the writer’s original stroke order or direction, so DTW and Chamfer measure agreement with a plausible trajectory.DTW may penalize valid alternative orders; human validation is recommended before using trajectories as training targets.Source: §6 · p. 8
  • Using DeepSeek-OCR both to filter references and compute readability scores may favor outputs resembling those references; an independent OCR model would provide a stricter readability measure.Source: §6 · p. 8
  • Findings are specific to the tested model versions and shared in-context prompt; other prompting strategies, fine-tuning, or tool use may change the ranking.The references also come from a small public model, and a larger model may yield better references.Source: §6 · p. 8

7. Conclusion

This section summarizes inkGround’s evaluation of VLM stroke-trajectory recovery from camera-captured handwriting, including the reported model contrast and IAM-OnDB ranking, and contains fragments of Appendix A’s recovery prompt.

7. Conclusion

  • The closed-source VLMs recover legible trajectories, while the tested open-source models do not; the same ranking holds on IAM-OnDB with recorded ground truth.Source: §7 · pp. 8–9
  • inkGround pairs camera-captured, full-page handwriting images with reference stroke trajectories to benchmark recovery in the wild, addressing gaps in existing datasets.The conclusion notes that existing datasets either lack stroke-level supervision or focus on isolated characters.Source: §7 · p. 8

Appendix A: Prompts for Stroke Trajectory Recovery

  • The prompt frames recovery as locating words in an image and predicting ordered stroke coordinates with start-of-stroke tokens, using in-context examples.The same prompt and output format are used for IAM-OnDB, with an example from a different writer than the test sample.Source: Appendix A · p. 11
  • The prompt fragments pair a partial-oval-and-tail description of “er” with ordered point coordinates and stroke values.The excerpt includes coordinates such as x145y122 and tuples such as (145,122,1), but provides no conclusion claims.Source: Appendix A · p. 13

Appendix B: Contextual recognition module

The contextual recognition module compresses an input image into visual tokens, which a language-aware decoder uses to generate text autoregressively.

Appendix B: Contextual recognition module

  • The contextual OCR encoder compresses the input image into compact contextual tokens.The tokens capture glyph structure, spatial layout, and visual context while discarding redundant visual details.Source: Appendix B · p. 13
  • The model treats the embeddings as a compressed optical representation rather than explicitly segmenting characters.A language-aware decoder interprets this representation.Source: Appendix B · p. 13
  • Conditioned on the visual tokens, the decoder autoregressively generates the output text sequence.Source: Appendix B · p. 13

Appendix C: Instructions for human evaluation

Human raters compare each page with the rendering of its Stage 1 trajectory, assessing text accuracy, completeness, and style consistency.

Appendix C: Instructions for human evaluation

  • Each participant compares the original page with its Stage 1 trajectory rendering, starting from a score of 10.Source: Appendix C · p. 13
  • Raters deduct one point for each character or spelling error, style-inconsistent word, or missing word.Source: Appendix C · p. 13
  • A rendered image containing more than 10 errors receives a score of zero.Source: Appendix C · p. 13

Appendix D: Additional qualitative results

Appendix D adds qualitative examples for Claude-Opus-4.8, Gemini-3.1-Pro, and Kimi-K3 on inkGround, presented in the layout used for Figure 4.

Appendix D: Additional qualitative results

  • Figures 6 and 7 provide additional qualitative results for the three closed-source models in the same layout as Figure 4.Source: Appendix D · p. 13
  • Figure 7 presents the third qualitative example for Claude-Opus-4.8, Gemini-3.1-Pro, and Kimi-K3.Source: Figure 7 · p. 15

Appendix E: Data source and acknowledgment

A greeting-card company supplied cursive-style handwriting samples and agreed to their publication for research purposes.

Appendix E: Data source and acknowledgment

  • A greeting-card company supplied the cursive-style handwriting samples used in the dataset.Source: Appendix E · p. 16
  • The company holds the dataset copyright and agreed to its publication for research purposes.Source: Appendix E · p. 16
  • The authors thank the company for its support.Source: Appendix E · p. 16

Related papers

Loading 2610.38720v1…