PaperlayerView exact revision on arXiv

Source-linked AI summary

Handwritten Digit Leakage from Smartphone Motion Sensors Across Unseen Users and Phone Models

Constantino Álvarez Casado, Erkka Rantahalvari, Matteo Pedone, Matti Matilainen, Manuel Lage Cañellas, Le Nguyen, Simo Hosio, Olli Silvén, Miguel Bordallo López

arXiv:2609.32117v1cs.LGcs.HC

TL;DR

Smartphone motion can disclose touchscreen input, but evidence about whether handwritten digits remain predictable across unfamiliar people and phones is limited. This study compares motion-signal representations under controlled participant and phone-model partitions, assuming known drawing intervals. It reports digit predictability across both kinds of holdout, while acquisition-order shortcuts qualify conclusions about practical exposure.

  • Problem

    Whether handwritten digits remain predictable from motion across unfamiliar participants and phone models is insufficiently characterized, despite the privacy relevance of motion readings that may disclose touchscreen input.

    Source: §1 · pp. 1–2

  • Method

    The study compares handcrafted features, MiniRocket, and a compact sensor patch transformer for 10-class HuMIdb digit classification under participant and phone-model partitions, assuming known drawing intervals.

    Source: §1 · p. 2

  • Results

    The transformer achieves 57.74% accuracy on unseen participants and 58.77±0.95% across three seeds on unseen participants using unseen phone models.

    Source: §6 · p. 6

  • Takeaways & Limitations

    Digits remain predictable across participants and phone models in the evaluated setting; low-motion recordings remain informative, and tested training and signal extensions provide no consistent gains.

    Source: §6 · p. 6

    Recognition assumes oracle drawing intervals, and acquisition-order task-phase cues may remain despite excluding chronology from classifiers.

    Source: §3.1 · p. 3, §5.6 · p. 6

Abstract

from arXiv · show

Smartphone motion sensors support interactive applications, but their readings may also reveal touchscreen input beyond their intended use. Assuming known drawing intervals, we study whether handwritten digits remain predictable across users and devices, as a 10-class problem on 19,628 HuMIdb recordings from 481 participants. We compare handcrafted features with classical machine learning algorithms, MiniRocket kernels, and a compact sensor patch transformer on accelerometer, linear acceleration, gyroscope, and gravity signals. The transformer achieves 57.74\% accuracy and 82.64\% top-3 accuracy on 75 unseen participants, and 58.77$\pm$0.95\% over 3 seeds for unseen participants on 9 unseen phone models. Low motion recordings remain informative, accuracy is not monotonic in motion level, and the tested contrastive pretraining, augmentation, and derived signals give no consistent gains. Digits are thus predictable beyond familiar users and phone models under assumed segmentation, while acquisition-order shortcuts limit conclusions about practical privacy exposure. Code available at: https://github.com/Arritmic/motion-digit-leakage.

1. INTRODUCTION

The introduction frames handwritten-digit inference as a privacy question: how predictable is entered content from motion alone, beyond familiar users and phones? The study evaluates that question on HuMIdb with known drawing intervals and controlled participant and phone-model partitions.

1. INTRODUCTION

  • Motion sensors may disclose touchscreen input beyond their intended collection purpose, including handwritten numbers that can represent codes, amounts, or dates.The passage distinguishes this content-inference privacy question from authentication.Source: §1 · p. 1
  • Content inference asks what was entered, unlike authentication, which asks who performed an interaction.Accuracy on familiar users or phones alone does not establish transfer to unfamiliar ones.Source: §1 · p. 1
  • Prior controlled handwriting-inference studies establish feasibility, but do not separate weak-motion transfer limits from memorization of people, phones, or acquisition order.The introduction identifies motion level, sensor timing, and recording protocols as evaluation complications.Source: §1 · p. 2
  • The evaluation uses 10-class classification on HuMIdb, assumes known drawing intervals, and uses motion signals as the classifier's only input.Automatic detection of handwriting intervals is left outside the evaluated task.Source: §1 · p. 2
  • The study reports digit predictability for unseen participants and for unseen participants using unseen phone models, while low-motion recordings remain informative.The tested contrastive pretraining, augmentation, and derived channels provide no consistent accuracy gains.Source: §1 · p. 2

2. RELATED WORK

Prior work establishes that touchscreen interactions leave informative motion responses, while identity-focused HuMIdb studies do not establish how well recordings preserve digit content across people and devices.

2. RELATED WORK

  • TouchLogger, TapLogger, TapPrints, Niffler, and Lee et al. used motion to infer keystrokes, tap locations, passwords, or handwritten patterns.Their targets, recording conditions, and observation assumptions differ, so their reported accuracies are not directly comparable with isolated digit classification.Source: §2 · p. 2
  • The study extends motion-inference work with participant and phone-model evaluation, while temporal and contrastive methods motivate its representation comparisons.HuMIdb work on bot detection and user verification targets identity-related properties, not digit content across people and devices.Source: §2 · p. 2

3. PROPOSED METHODOLOGY

The methodology evaluates digit classification from aligned multisensor recordings using representations and protocols designed to test transfer across participants and phone models. It compares handcrafted features, MiniRocket, and a sensor patch transformer, alongside training and signal variants.

3. PROPOSED METHODOLOGY

  • The five-stage methodology constructs a cohort under an observation model, conditions signals, forms representations, classifies, and evaluates with controlled participant and phone-model protocols.Fitted preprocessing operations use the training partition only.Source: §3 · p. 2

3.1. Problem formulation and observation model

  • Each recording supplies four streams—acceleration, linear acceleration, angular velocity, and gravity—and a classifier maps the resulting input to one of ten digit scores.Participant and phone-model identifiers define partitions and contrastive positives but never enter the classifier; touch values, absolute timestamps, and task order also stay out.Source: §3.1 · p. 2

3.2. Signal conditioning

  • Signal conditioning crops streams to the touch-defined drawing interval and their common overlap, then interpolates and resamples them onto a shared grid.A low-pass filter is applied before downsampling; fixed length removes absolute duration, but signal shape still reflects writing speed.Source: §3.2 · p. 3

3.3. Handcrafted features and classical classifiers

  • The handcrafted baseline summarizes each channel with 29 order-insensitive statistics, yielding 406 features for RBF-SVM, random forest, and XGBoost classifiers.XGBoost is reported as the strongest handcrafted reference.Source: §3.3 · p. 3

3.4. MiniRocket representation

  • MiniRocket pools fixed convolutional-kernel responses into 9,996 features, then PCA reduces them to 256 components for an RBF-SVM.Its rows are compared with a transformer trained on the same earlier tensor, not the final transformer.Source: §3.4 · p. 3

3.5. Sensor patch transformer

  • The sensor patch transformer tokenizes each sensor separately, adds position and sensor embeddings, and uses joint attention to fuse 124 temporal tokens.Its patches contain eight samples with stride four; two encoder layers process the tokens before a linear classifier.Source: §3.5 · p. 3

3.6. Training objectives and variants

  • Training variants test shared temporal shifts and noise, contrastive objectives, negative weighting, derived channels, and conditioning on duration.SupCon pairs same-digit recordings across phone models; jerk, angular acceleration, and integrated linear acceleration expand the input channels.Source: §3.6 · pp. 3–4

4. EXPERIMENTAL EVALUATION

The evaluation uses eligible HuMIdb handwriting recordings to test how predictions generalize across participants and phone models, and examines motion levels under fixed training-derived thresholds. Accuracy is the primary metric, with clustered uncertainty estimates and a shared training budget for the partition protocols.

4.1. HuMIdb digit task and cohort

  • 19,628 recordings from 481 participants and 165 phone model strings had valid drawing intervals and all four sensors.The recordings came from a database with irregular sensor sampling; orientation was always portrait, and digit order offered an acquisition-order shortcut.Source: §4.1 · p. 4

4.2. Evaluation protocols

  • The four protocols vary whether participants and phone models are shared between training and test, with P fixed before eligibility filtering and S, M, and U using 13,642 recordings.S splits participants within phone models; M reassigns recordings within each phone model and digit stratum, creating participant and session overlap; U holds out whole phone models. S, M, and U use 31 epochs, and the three U seeds reuse one partition.Source: §4.2 · p. 4, Figure 2 · p. 4, Table 1 · p. 4

4.3. Motion level analysis

  • Motion groups use the maximum of training-distribution-scaled linear-acceleration and angular-velocity measures, with training tertiles defining low, medium, and high motion.Measures are computed on original timestamped drawing intervals before resampling; the same thresholds apply to validation and test, and comparisons include removing low-motion recordings versus matched random removal.Source: §4.3 · p. 4

4.4. Metrics and implementation details

  • Accuracy is primary, with macro-F1 and top-3 accuracy secondary; confidence intervals use cluster bootstrap resampling by participant or, for U, phone model.After early stopping selected 31 epochs, a new model was trained on all 334 training participants; S, M, and U runs reuse this budget, and implementation settings are listed in Table 2.Source: §4.4 · pp. 4–5, Table 2 · p. 5

5. RESULTS

Across unseen people and phones, motion-sensor models classify digits above chance; tested extensions offer no consistent gains, and accuracy varies non-monotonically with motion level.

5.1. Digit content across unseen participants

  • 57.74% accuracy, 57.77% macro-F1, and 82.64% top-3 accuracy on unseen participants exceed the 10% chance level; MiniRocket reaches 52.64%.The strongest handcrafted classifier, XGBoost, reaches 42.23%. Removing positional encodings lowers transformer accuracy from 55.67% to 35.97%; observations attribute useful information to motion order, while confused digits share path fragments, though shape is not isolated.Source: Table 3 · p. 5, §5.1 · p. 5

5.2. Sensor contribution

  • In the 19,251-recording handcrafted RBF-SVM comparison, gyroscope alone scores 32.99%, above accelerometer (25.86%) and linear acceleration (24.76%); rankings may differ for learned representations.The paper notes that gyroscopes directly measure slight phone rotation from finger movement, while accelerometers are dominated by gravity and orientation.Source: §5.2 · pp. 5–6

5.3. Unseen phone models and participant overlap

  • U reaches 57.71% and 58.77% over three seeds, versus 54.80% for S; the unseen-phone evaluation uses one partition with nine test models.The authors caution that the populations differ, so unseen phones are not shown to be easier; U accuracy ranges from 22.2% to 81.4% across phone models.Source: §5.3 · p. 6

5.4. Training extensions

  • Tested contrastive pretraining, confusable-pair weighting, derived channels, and duration context yield no consistent top-1 accuracy gains.Weighting confusable pairs reduces their errors from 171 to 165 but lowers accuracy to 56.69%.Source: §5.4 · p. 6

5.5. Motion level and unknown phone support

  • All-recording training yields 57.54% accuracy on low-motion, 64.65% on medium-motion, and 54.81% on high-motion recordings; removing low-motion training data lowers outside-low accuracy by 2.27 points.The low-motion group is not motionless, and the authors report that its recordings also help the model on other recordings.Source: §5.5 · p. 6

5.6. Implications for privacy

  • For new people, 82.64% top-3 accuracy could narrow a short numeric-code search, but multi-digit combinations and automatic drawing-interval detection were not evaluated.The reported inference assumes known drawing intervals; effectiveness and legitimate-application costs of proposed timing and rotation perturbations were also not evaluated.Source: §5.6 · p. 6

6. IMPLICATIONS AND CONCLUSION

The study finds handwritten digits predictable from smartphone motion across unseen participants and phone models, assuming known drawing intervals. Its conclusions remain bounded by acquisition-order shortcuts and retrospective evaluation conditions.

6. IMPLICATIONS AND CONCLUSION

  • 57.74% accuracy on unseen participants and 58.77±0.95% across three training seeds on unseen participants using unseen phone models show digit predictability across both settings.The conclusion describes the results as conditional on known drawing intervals.Source: §6 · p. 6
  • Low-motion recordings remain informative, while accuracy does not increase monotonically with motion level.Source: §6 · p. 6
  • The tested contrastive pretraining, augmentation, and derived channels provide no consistent accuracy gains.Source: §6 · p. 6
  • Fixed digit order is the principal threat to validity because indirect acquisition cues may contribute to classification even when explicit chronology is excluded.The results do not establish how much predictive information comes specifically from handwriting dynamics or demonstrate an operational attack.Source: §6 · p. 6
  • The findings are conditional on known drawing intervals, one phone per participant, a single phone-model holdout partition, and retrospective comparisons.Source: §6 · p. 6
  • The authors propose randomizing digit order, annotating posture and device support, and adding sensor-based drawing-interval detection to future evaluations.They also identify temporal and sensor perturbations as potential defenses whose effectiveness and impact on legitimate applications remain unevaluated.Source: §6 · p. 6

COMPLIANCE WITH ETHICAL STANDARDS

The study used HuMIdb human-subject data collected with participant consent and anonymized under the GDPR.

COMPLIANCE WITH ETHICAL STANDARDS

Loading 2609.32117v1…