PaperlayerView exact revision on arXiv

Source-linked AI summary

mmPencil: Toward Writing-Style-Independent In-Air Handwriting Recognition via mmWave Radar and Large Vision-Language Model

Yifan Guo, Zhu Wang, Qian Qin, Yangqian Lei, Qiwen Gan, Zhuo Sun, Chao Chen, Bin Guo, Zhiwen Yu

arXiv:2610.23419v1cs.HCcs.CV

TL;DR

RF in-air handwriting methods are limited by predefined writing planes, constrained styles, and small vocabularies, while writing position, direction, and personal habits vary the signals. mmPencil reconstructs 3D handwriting trajectories from mmWave signals and maps them to images that VLMs can process. mmPencil achieves 97.60% recognition accuracy for 200 words and 92.50% zero-shot recognition accuracy for users unseen during training.

  • Problem

    RF in-air handwriting methods are limited by predefined writing planes, constrained styles, and small vocabularies, while writing position, direction, and personal habits vary the signals.

    Source: §1 · p. 3

  • Method

    mmPencil reconstructs 3D handwriting trajectories from mmWave signals and maps them to images that VLMs can process.

    Source: §1 · p. 3

  • Results

    mmPencil achieves 97.60% recognition accuracy for 200 words and 92.50% zero-shot recognition accuracy for users unseen during training.

    Source: §1 · p. 4

  • Takeaways & Limitations

    Instruction fine-tuning with a small amount of data significantly improves handwriting word recognition performance.

    Source: §7.1.3 · p. 16

    Body swaying lowers recognition accuracy to 75.00% for mmPencil-Qwen and 70.00% for mmPencil-LLaVA; leg shaking has a smaller impact, with accuracies of 82.50% and 77.50%, respectively.

    Source: §7.3.9 · p. 24

Abstract

from arXiv · show

In-air handwriting recognition based on radio frequency (RF) signals offers a promising solution for human-computer interaction, yet existing methods are often limited by predefined writing planes, constrained writing styles, and small recognition vocabularies. To address these limitations, we propose m^2VLMs, a modality mapping architecture that bridges millimeter-wave (mmWave) radar sensing and large vision-language models (VLMs), and develop mmPencil, a 3D in-air handwriting word recognition system. mmPencil reconstructs 3D handwriting trajectories from mmWave signals using a multi-stage spatial trajectory reconstruction algorithm and maps them into visual representations through a spatial-to-visual mapping algorithm, enabling VLMs to recognize handwriting while reducing sensitivity to writing styles and environmental factors. We collect and release a 3D mmWave handwriting dataset containing 7,664 samples of 200 words from 12 users across 22 writing scenarios. Extensive experiments demonstrate that mmPencil achieves robust word recognition under variations in writing position, range, angle, speed, size, direction, and user movement. It achieves 97.60% accuracy on 200 word classes for seen users and 92.50% zero-shot accuracy on 50 word classes for unseen users, outperforming state-of-the-art baselines.

1 Introduction

The introduction motivates RF-based in-air writing as a private, non-contact interaction method, while noting that existing RF approaches face data and style-related challenges. mmPencil bridges mmWave radar and VLMs with trajectory reconstruction and spatial-to-visual mapping for 3D word recognition.

1 Introduction

  • RF-based in-air writing offers non-contact interaction and privacy protection, unlike voice- and vision-based methods that can expose personal information.It is presented as useful where touchscreens, mice, or keyboards are unsuitable, including small-screen devices and use by elderly people and people with disabilities.Source: §1 · p. 2
  • RF handwriting recognition is constrained by training-data scarcity and quality, while writing position, direction, and individual habits can alter RF signals.The introduction also notes the diversity of handwriting content and the challenge of recognizing realistic writing in 3D.Source: §1 · p. 3
  • mmPencil reconstructs 3D trajectories from mmWave signals and maps them into images so VLMs can interpret RF handwriting.Its reconstruction uses refined range-azimuth and range-elevation profiles, followed by outlier removal and 3D Kalman-filter smoothing; direction-corrected PCA maps trajectories to 2D images.Source: §1 · p. 3
  • The paper releases a 3D mmWave handwriting dataset with 7,664 samples of 200 words from 12 users across 22 writing scenarios.The authors describe this as the first attempt to apply VLMs to RF-based handwriting recognition.Source: §1 · p. 3
  • In real-world scenarios, mmPencil achieves 97.60% recognition accuracy for 200 words and 92.50% zero-shot recognition accuracy for users unseen during training, with recognition unaffected by varying writing styles and environmental factors.Source: §1 · p. 4

2 Related Work

Related work covers model-driven and data-driven RF handwriting recognition, alongside systems that use acoustic, Wi-Fi, RFID, and mmWave sensing. The section contrasts mmPencil’s combination of trajectory reconstruction and VLM-based spatial-to-visual mapping with these prior approaches.

2 Related Work

  • Existing RF in-air writing systems are compared in Table 1.The caption identifies the comparison’s scope but does not provide its entries or outcomes.Source: Table 1 · p. 4
  • Model-driven methods map signal variations to handwriting actions, but some rely on predefined writing orders or specific scenarios.The cited examples also include systems limited by Wi-Fi bandwidth, complex curved trajectories, or initial-position accuracy.Source: §2 · pp. 4–5
  • Data-driven systems train neural networks on labeled movement data, but CNN- and LSTM-based approaches can struggle to generalize to unseen complex movements and user-varying styles.The section describes examples using mmWave trajectory time series, images, and spatial-temporal features.Source: §2 · p. 5
  • mmPencil combines model-driven trajectory reconstruction with data-driven VLM perception through spatial-to-visual mapping.The section states that the system maintains high sensing performance amid handwriting-style and environmental variation.Source: §2 · p. 5

3 Preliminaries

The preliminaries explain how FMCW radar and MIMO antenna arrays provide range and spatial-angle information. They establish the basis for 3D trajectory reconstruction and the mmPencil architecture’s mapping from radar data to VLM-readable images.

3.1 Antenna Array

  • MIMO arrays combine multiple transmit and receive paths into a virtual array, enabling spatial target separation and improved angle resolution.In TDM mode, transmit antennas emit chronologically, and joint processing separates target signals across temporal and spatial dimensions.Source: §3.1 · p. 6, Figure 1 · p. 6

3.2 Range Estimation

  • FMCW radar estimates target range from the intermediate-frequency signal, with range resolution limited by signal bandwidth.The transmit frequency varies linearly over time; processing the transmitted and delayed received signals yields an intermediate-frequency signal whose FFT provides range information.Source: §3.2 · p. 6

3.3 Multi-dimensional Angle Estimation

  • Combining azimuth and elevation estimates enables 3D target detection, extending in-air writing recognition beyond a 2D plane.Azimuth derives from horizontal path differences across receive channels, while transmit antennas at different heights provide elevation information.Source: §3.3 · p. 7
  • The m^2VLMs architecture combines data collection, trajectory reconstruction, data mapping, and instruction fine-tuning to connect mmWave data with VLMs.Trajectory reconstruction addresses environmental and writing-position effects, while data mapping mitigates writing-order and direction effects.Source: §3.3 · p. 7
  • The trajectory reconstruction method extracts spatial coordinates from range-azimuth and range-elevation profiles, then removes outliers and smooths trajectories with a 3D Kalman filter.The profiles are spectrum-refined before the method dynamically extracts target-cluster centers over time.Source: §3.3 · pp. 7–8
  • Direction-corrected PCA maps 3D trajectories to 2D images while preserving handwriting shape and reducing differences from writing order and style.An instructional prompt asks the VLM to identify the image’s word and return only one correctly spelled English word.Source: §3.3 · p. 8

5 Writing Trajectory Reconstruction

The reconstruction pipeline suppresses static environmental components, refines range-angle spectra, and derives and smooths 3D handwriting trajectories. Its stages address limited radar resolution, spatially distributed reflections, and non-writing or noisy motion.

5.1 Environmental Disruption Elimination

  • Mean-subtracting the time-domain signal from each raw IF frame uses the signal’s temporal mean as a reference to remove persistent leakage and environmental reflections.The method avoids the extra no-target data collection required by reference-signal approaches.Source: §5.1 · p. 9

5.2.1 Range-Angle Spectral Refinement.

  • 4× interpolation yields approximately 1.05 cm range resolution, while spectral refinement reaches 1° azimuth and elevation resolution.The original 12-channel radar’s approximate range, azimuth, and elevation resolutions are 3.75 cm, 15°, and 30°, respectively.Source: §5.2.1 · pp. 10–11

5.2 Multi-Stage Spatial Trajectory Reconstruction

  • The reconstruction stages refine radar spectra, localize target-cluster centers, remove non-writing trajectory outliers, and smooth the resulting 3D trajectories.Cluster-center extraction uses target points from range-azimuth and range-elevation profiles; outlier removal addresses preparatory and concluding movements.Source: §5.2.2 · p. 11, §5.2.3 · pp. 11–12

5.2.2 Frequency-Domain Feature Extraction.

  • Per-frame target points above a threshold are grouped into cluster centers, whose positions across frames estimate the handwriting trajectory.The approach accounts for reflections spread across multiple range and angle bins rather than relying only on profile peaks.Source: §5.2.2 · p. 11

5.2.3 Motion Segment Extraction and Trajectory Smoothing.

  • A 3D Kalman filter smooths noisy trajectories while preserving character shape; low transition covariance over-smooths details, whereas high covariance leaves noise and clutter.The implementation typically sets observation covariance to 1 and transition covariance to 1 × 10^-3.Source: §5.2.3 · p. 12, Figure 7 · p. 12

6 Data Mapping from Spatial Trajectory to Word Image

The spatial-to-visual mapping projects 3D writing trajectories onto a 2D word image using principal directions. A direction correction addresses potential mirroring so the image follows common writing conventions.

  • PCA selects the two eigenvectors with the largest eigenvalues as a 2D projection basis, reflecting the trajectory’s primary horizontal and secondary vertical directions.The trajectories are first converted from spherical to Cartesian coordinates and centralized.Source: §6 · p. 13
  • PCA can produce mirrored trajectories because it ignores writing sequence, so a post-processing step flips horizontal or vertical orientation to align with common writing direction.The adjustment uses trajectory start and end points to determine direction.Source: §6 · p. 13
  • The projected trajectory becomes a word image, mitigating the impact of writing order on recognition and avoiding obstacles faced by time-series techniques such as LSTM.Source: §6 · p. 13

7 Evaluation

The evaluation uses a commercial mmWave radar prototype and tests recognition accuracy and robustness across users, word lengths, and varied writing conditions.

  • The experiments evaluate recognition across users and word lengths, as well as variations in writing range, angle, speed, size, direction, environment, and user movement.Source: §7 · p. 14

7.1 Experiment Setup

The setup combines commercial mmWave sensing with a dataset spanning multiple users, words, and writing scenarios. Three open-source VLMs are fine-tuned, and the authors report improved RF handwriting processing with small training data.

7.1.1 Hardware Implementation.

  • The prototype uses an IWR6843ISK mmWave radar with three transmit and four receive antennas, spanning 60–64 GHz and a 120° horizontal by 60° elevation field of view.Data collection uses a DCA1000EVM module.Source: §7.1.1 · p. 14

7.1.2 Dataset Collection.

  • The dataset contains 7,664 samples from 12 users, combining 200-word, 50-word multi-user, and 22-scenario single-user collections.The scenario collection varies range, angle, speed, environment, size, direction, and user movement.Source: §7.1.2 · pp. 14–15

7.1.3 Fine Tuning Setup for VLMs.

  • Instruction fine-tuning improves the selected VLMs’ RF handwriting processing under style and environmental variation, with small data rather than large-scale training from scratch.The evaluated models are GOT-OCR-0.5B, Qwen2-VL-7B, and LLaVA-NeXT-7B.Source: §7.1.3 · pp. 15–16

7.2 Accuracy Evaluation

Across word-level datasets, VLM-based mmPencil configurations generalize better across users than lightweight baselines, while ablations show that trajectory reconstruction components are critical to recognition.

7.2.1 Performance Comparison on the 200-Word 4-User Dataset.

  • In non-cross-user experiments, mmPencil-GOT and mmPencil-ResNet achieved the highest recognition accuracies, 97.60% and 97.80%, respectively; all methods except LSTM exceeded 90%.In cross-user experiments, VLM-based methods achieved 91.37%, 86.62%, and 80.63%; mmPencil-ResNet and mmPencil-DCNN achieved 78.87% and 52.62%, respectively.Source: §7.2.1 · pp. 17–18

7.2.2 Performance Comparison on the 50-Word 12-User Dataset.

  • 92.50% accuracy for mmPencil-GOT on 50 words with four unseen users exceeded the other tested configurations in the cross-user setting.VLM-based methods outperformed the lightweight baselines, and performance improved as training data included more users.Source: §7.2.2 · p. 18

7.2.3 Cross-User Recognition Performance under Varying Numbers of Seen Users.

  • 94.25% accuracy for mmPencil-GOT with nine seen users was its best reported result, up from 85.00% with two seen users.All models improved as the number of seen users increased, while VLM-based models remained stronger when training data included few users.Source: §7.2.3 · p. 19, Figure 13 · p. 19

7.2.4 Ablation Studies.

  • 92.50% accuracy for the complete system fell to 8.33% without Spectral Refinement and 2.33% when Trajectory Smoothing was also removed.Replacing GOT-OCR with ResNet-101 reduced accuracy to 84.13%; the study also reports residual ResNet accuracy under removed trajectory modules, attributed to local image patterns rather than image understanding.Source: §7.2.4 · pp. 19–20, Figure 14 · p. 20

7.3 Robustness Evaluation

Robustness evaluations vary word-category count and word length; recognition remains strong across category scales, and longer words do not consistently reduce accuracy.

7.3.1 Word-Category-Number Independence Evaluation.

  • 93.70%, 92.00%, 92.50%, and 93.20% were mmPencil-Qwen accuracies with 50, 100, 150, and 200 word categories, respectively.The reported variation across category counts was approximately 2%.Source: §7.3.1 · p. 20

7.3.2 Word-Length Independence Evaluation.

  • 98.10% accuracy for mmPencil-Qwen on six-letter words was higher than its 85.22% accuracy on two-letter words.Accuracy did not decrease as word length increased; authors attribute the pattern to VLM contextual understanding and trajectory reconstruction restoring handwriting details.Source: §7.3.2 · pp. 20–21

7.3.3 Writing-Range Independence Evaluation.

Tests across range, angle, speed, and writing size show strong performance across several conditions, with very small writing areas remaining difficult.

7.3.3 Writing-Range Independence Evaluation.

  • 85.00% accuracy for both mmPencil-Qwen and mmPencil-LLaVA at 80 cm showed recognition remained effective at the far tested range.At close ranges, reported accuracies ranged from 90.00% to 95.00%.Source: §7.3.3 · p. 21

7.3.4 Writing-Angle Independence Evaluation.

  • 100.00% accuracy for both systems at 0° fell to 87.50% for Qwen and 85.00% for LLaVA at 30°.The authors note that the final letter may approach the field-of-view edge at 30°.Source: §7.3.4 · p. 22

7.3.5 Writing-Speed Independence Evaluation.

  • 95.00% was mmPencil-LLaVA’s accuracy at both fast and slow writing speeds, while mmPencil-Qwen reached 82.50% at fast speed.Qwen achieved 92.50% at the two slower tested speeds.Source: §7.3.5 · p. 22

7.3.6 Writing-Size Independence Evaluation.

  • 75.00% for mmPencil-Qwen and 72.50% for mmPencil-LLaVA at 5cm × 10cm were below their 100.00% and 87.50% results at normal sizes.The authors report that overlapping connecting trajectories and letters caused misclassification at the smaller size.Source: §7.3.6 · pp. 22–23, Figure 20 · p. 22

7.3.7 Writing-Direction Independence Evaluation.

Direction and environmental variation were handled with generally strong recognition, whereas user movement—especially body sway—degraded performance and remains a boundary.

7.3.7 Writing-Direction Independence Evaluation.

  • Above 80% accuracy was maintained by both models at writing-direction deviations as large as ±40°.The authors attribute fluctuations to overlapping and rotated letter trajectories and report that VLM contextual information can help correct letter errors.Source: §7.3.7 · p. 23

7.3.8 Environment Independence Evaluation.

  • 97.50%, 90.00%, and 95.00% were mmPencil-Qwen accuracies across three tested scenes, including scenes with nearby human movement.The authors attribute environmental robustness to environmental-noise cancellation in trajectory reconstruction.Source: §7.3.8 · pp. 23–24

7.3.9 User-Movement Independence Evaluation.

  • 75.00% and 70.00% were mmPencil-Qwen and mmPencil-LLaVA accuracies during body sway, compared with 82.50% and 77.50% during leg shaking.Large user movements can cause trajectory drift and system failure; trajectory smoothing alleviates some fluctuations from minor movement.Source: §7.3.9 · p. 24, Figure 24 · p. 24

8 Discussion

The discussion reports digit-and-symbol recognition, exploratory zero-shot word recognition, and processing-latency trade-offs, then outlines sentence-level recognition as a more complex next step.

8.1 Recognition of Handwriting Digits and Symbols

  • 80.21% for mmPencil-Qwen and 75.00% for mmPencil-LLaVA on digits and symbols, averaged over 8 newly recruited users.The evaluation covered 10 digits and 2 symbols, with 4 samples per class per user.Source: §8.1 · p. 24
  • Most digit and symbol categories exceed 75% accuracy, but similar shapes such as “1” versus “7” and “6” versus “9” are more confusable.The authors propose structured priors, such as typical digit-writing order, and fine-grained feature alignment for future improvement.Source: §8.1 · p. 25

8.2 Zero-Shot Recognition of Unseen Words

  • 45.83% for mmPencil-Qwen and 38.96% for mmPencil-LLaVA on 20 unseen words, substantially outperforming mmPencil-RestNet, which had almost no zero-shot capability.The VLMs were fine-tuned on 180 word categories from the 200-Word 4-User Dataset.Source: §8.2 · p. 25
  • Zero-shot performance varies across words: “that,” “do,” “seems,” and “an” exceed 70% accuracy, while “god,” “almost,” “sir,” and “old” are challenging.The authors suggest models may favor seen words; for example, “well” may be mistaken for the similar seen word “will.”Source: §8.2 · p. 25
  • The authors plan to redesign the model and inference mechanism to mitigate interference from similarities between unseen and seen words.This is intended to address the observed variation in zero-shot performance across unseen words.Source: §8.2 · pp. 25–26

8.3 Real-Time Processing Feasibility

  • With moderate spectral refinement, trajectory reconstruction takes 155.65 ms per letter versus 468.30 ms under standard refinement, with a slight accuracy drop.Qwen2.5-VL-3B takes 32.44 ms per sample at 80.50% accuracy; the authors report this configuration and moderate refinement balance accuracy and latency.Source: §8.3 · p. 26, Table 4 · p. 26
  • Recognition is triggered after a whole word is written; despite end-to-end latency sometimes exceeding 200 ms, the authors consider it tolerable against a natural 2–3 seconds per letter.They acknowledge a latency limitation from prioritizing maximum recognition accuracy and plan more efficient reconstruction and inference optimization.Source: §8.3 · p. 26

8.4 In-Air Handwriting Sentence Recognition

  • Sentence-level recognition is a proposed next step, but complex trajectories require segmentation and models that capture local word features and global sentence semantics.The discussion also identifies large-scale, diverse RF sensing data as a potential aid to recognition performance.Source: §8.4 · p. 27

9 Conclusion

mmPencil combines mmWave radar and VLMs for writing-style-independent 3D handwriting word recognition. The authors report robustness across users, writing conditions, and environments, and explore real-time feasibility and additional recognition tasks.

  • mmPencil uses multi-stage trajectory reconstruction to extract 3D trajectories independent of writing position and direction, then maps them to 2D images for VLM processing.The authors collected a 3D mmWave handwriting word dataset and fine-tuned two VLMs.Source: §9 · p. 27
  • Experiments report robust word-level recognition across users, word lengths, writing ranges, angles, speeds, sizes, directions, environmental conditions, and user actions.The paper also evaluates real-time feasibility and explores zero-shot recognition of digits, symbols, and unseen words, as well as sentence-level recognition.Source: §9 · p. 27

Related papers

Loading 2610.23419v1…