PaperlayerView exact revision on arXiv

Source-linked AI summary

Watch and Crack: Password Inference from Smart-Glasses Video

Yoav Orenbach, Avishai Wool

arXiv:2609.06539v1cs.CR

TL;DR

An adversary can record smartphone password entry with ordinary-looking smart glasses and later reconstruct the password offline, even when users deploy privacy screens or display modes. The pipeline analyzes smart-glasses video with fingertip segmentation and keystroke prediction, then uses estimated key probabilities and password priors to compute password rank. Up to 60 bits of password entropy are reduced versus the prior baseline alone; human-chosen passwords up to 16 characters crack in hours to days, and random passwords up to 13 in under an hour.

  • Problem

    An adversary can record smartphone password entry with ordinary-looking smart glasses and later reconstruct the password offline, even when users deploy privacy screens or display modes.

    Source: §1 · pp. 1–2

  • Method

    The pipeline analyzes smart-glasses video with fingertip segmentation and keystroke prediction, then uses estimated key probabilities and password priors to compute password rank.

    Source: §3 · p. 4, §9 · p. 14

  • Results

    Up to 60 bits of password entropy are reduced versus the prior baseline alone; human-chosen passwords up to 16 characters crack in hours to days, and random passwords up to 13 in under an hour.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The attack demonstrates a privacy risk from typing passwords in public spaces, even when the screen itself is not directly readable.

    Source: §1 · p. 2, §9 · p. 14

    The evaluation is a controlled feasibility study, not an in-the-wild demonstration, and assumes a relatively stationary attacker and victim.

    Source: §8.2 · p. 14

Abstract

from arXiv · show

Typing passwords on smartphones in public places exposes users to video-based side-channel attacks, in which an adversary records the typing session and reconstructs the entered password by analyzing finger movements. Prior video-based keystroke inference attacks have targeted free-form text on tablets, numeric PINs, pattern locks, and passwords under unrealistically simplified conditions. We present the first general video-based keystroke inference pipeline that recovers rule-based alphanumeric passwords from smartphone QWERTY keyboards, covering all four keyboard layouts and requiring no assumptions about the victim or their device. Using the built-in camera of the popular Meta Ray-Ban smart glasses, our pipeline tracks the device and typing fingertips at sub-pixel resolution, predicts keystrokes via a self-supervised ensemble of neural networks, dynamically estimates the on-screen keyboard layout, and computes per-key probability distributions. These video-derived probabilities are combined with prior password-typing distributions based on 2.19 billion leaked passwords to estimate the rank, and therefore the crack time, of the observed password. In a user study using passwords that adhere to a typical enterprise password policy, human-chosen passwords of up to 16 characters can be cracked in hours to days, and password-manager-chosen random passwords of up to 13 characters in under an hour on a modern GPU, reducing password entropy by up to 60 bits relative to the prior baseline. For human-chosen passwords, combining video evidence with prior statistics yields an additional statistically significant reduction of approximately 20 bits compared to either source alone. The attack succeeds at distances up to 1.8 m, across diverse attacker-victim postures and viewing angles, and generalizes across three popular smartphones.

1 Introduction

The paper addresses video inference of smartphone passwords, which prior attacks handled only under restricted conditions. It introduces a smart-glasses pipeline for estimating password rank and reports successful cracking of policy-compliant passwords.

1 Introduction

  • Privacy screens and display modes address screen reading, whereas smart-glasses video can capture finger movements from an ordinary frontal or side view.The attack processes recorded video offline, and the glasses provide a stable, hands-free first-person view without a conspicuous posture.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • Earlier video attacks targeted text on tablets, PINs, patterns, or passwords under restrictions that left general smartphone QWERTY passwords unaddressed.Prior password attacks limited character sets, device types, or typing behavior; the paper targets rule-based alphanumeric passwords across all layouts without assumptions about the user or device.Source: §1 · p. 1
  • The pipeline tracks fingertips at sub-pixel resolution, predicts keystrokes with a self-supervised neural-network ensemble, estimates keyboard layout dynamically, and produces per-key probabilities.It covers lowercase, uppercase, numbers, and special symbols; the keyboard geometry is inferred without device dimensions.Source: §1 · p. 2
  • Video-derived keystroke probabilities are combined with a prior from 2.19 billion leaked passwords to estimate password rank and crack time.The prior is based on the Weakpass 4 corpus, and ESrank estimates enumeration effort.Source: §1 · p. 2
  • Human-chosen passwords up to 16 characters can be cracked in hours to days, and password-manager-chosen random passwords up to 13 characters in under an hour.Combining video and prior statistics yields an approximately 20-bit advantage over either source alone for human-chosen passwords; entropy falls by up to 60 bits versus the prior baseline alone.Source: §1 · p. 2
  • The pipeline detects backspace corrections and removes them from the predicted sequence before estimating the password rank.This produces a corrected password candidate for rank estimation.Source: §1 · p. 2

2 Background and Related Work

The section surveys side-channel work by channel and input type, then contrasts prior video attacks with the broader smartphone-password setting addressed here.

2 Background and Related Work

  • Prior work spans video and non-video channels and targets free-form text, PINs or patterns, and full passwords.The section organizes the literature along these two dimensions.Source: §2 · p. 2

2.1 Non-Video Side Channels

  • Non-video attacks infer keystrokes from acoustic, electromagnetic, Wi-Fi, or wearable-inertial signals, while indirect video attacks use reflections, eye movements, or tablet vibrations.These approaches may require environmental conditions, specialized hardware, or local device compromise.Source: §2.1 · pp. 2–3

2.2 Video-Based Side Channels

  • Video attacks have addressed PINs, patterns, free-form text, and password entry, while emerging VR/AR work uses head or eye tracking in virtual environments.These input types and modalities differ from physical smartphone typing on a full QWERTY keyboard.Source: §2.2 · p. 3

2.3 Closely Related Work

  • Earlier text inference on tablets relied on language models and larger keys, while password recovery from physical-keyboard video achieved only 18.9% within top-50 predictions.PIN-focused work does not address alphanumeric layout switching, and PILOT [3] targeted computers and ATMs with a restricted character set.Source: §2.3 · p. 3
  • Shukla and Phoha required single-finger typing, omitted most special symbols, and relied on manual initialization and known device dimensions; this work removes those restrictions.The proposed approach supports natural multi-finger typing, all four QWERTY layouts, strict password policies, and formal rank estimation.Source: §2.3 · p. 3

3 Preliminaries

The preliminaries describe the capture device and models used to track the phone and fingers, detect keystrokes, combine probability distributions, and estimate password rank.

3 Preliminaries

  • Meta Ray-Ban glasses provide a commercially available forward-facing camera capable of recording up to 1080p video.The experiments used the built-in camera without optical zoom or external accessories, and did not hide its capture LED.Source: §3 · pp. 3–4
  • MediaPipe supplies initial hand landmarks, while SAM 3 segments the phone screen and fingertips for tracking and sub-pixel localization.SAM 3’s text-prompted masks supplement MediaPipe, which can fail under partial hand occlusion.Source: §3 · p. 4
  • Horn–Schunck optical flow bridges gaps when MediaPipe detections fail, and an ensemble of five R3D-18 networks classifies keypresses from 16-frame windows.R3D-18 is pretrained for video action recognition and fine-tuned for binary keypress classification.Source: §3 · p. 4
  • Conflation combines video-derived and prior password-typing distributions by a normalized element-wise product, giving greater weight to distributions with smaller variance.In this pipeline it combines the two distributions for each keystroke position.Source: §3 · pp. 4–5
  • ESrank estimates a password’s position in probability order without enumerating the full keyspace, enabling rank and crack-time estimates.The reported log2 rank expresses the size of the search; a rank of r means r−1 candidates have higher probability than the true password.Source: §3 · p. 5

4 Threat Model

The threat model considers opportunistic recording of smartphone password entry with smart glasses, followed by offline password guessing using a leaked hash.

4 Threat Model

  • The target setting is password entry on smartphones in public places, with passwords tested under a policy requiring at least eight characters, an uppercase letter, and a digit or special character.The policy reflects composition rules still used by deployed services and enterprise systems.Source: §4 · p. 5
  • The attacker records from a frontal or oblique angle using built-in smart-glasses video, then processes the footage offline with public tracking, segmentation, and classification tools.The attacker knows the victim uses a commonly deployed QWERTY keyboard application but need not know which one.Source: §4 · p. 5
  • The attacker needs no knowledge of the phone model, keyboard application, or victim-specific labeled data, and no access to sensors beyond the camera.The victim’s screen need not be readable, including when a privacy screen or display mode is used.Source: §4 · p. 5
  • Offline guessing assumes the attacker has obtained the password hash; reported crack-time estimates use a fast-hash rate and rescale for slower key-derivation functions.Identifying when a password is being entered is largely outside the pipeline’s scope, and capture can also come from a stable camera other than smart glasses.Source: §4 · p. 5

5 Properties of Smartphone Password Typing

Smartphone password entry involves multiple keyboard layouts and device-dependent typing sequences. The paper uses leaked-password statistics to construct a prior over typed sequences.

5.1 Smartphone Typing Characteristics

  • Password-policy requirements trigger transitions among four QWERTY layouts, so an eight-character password can require ten keystrokes.For example, Myspace0 requires shift and layout-switch keystrokes.Source: §5.1 · p. 6
  • Keyboard applications differ in number and symbol placement, so the same password can require different typed sequences and touch counts.The pipeline therefore accounts for Gboard, iOS, and Samsung configurations.Source: §5.1 · p. 6

5.2 A-Priori Password Typing Distribution

  • From 2.19 billion leaked passwords, the authors derive per-position typed-character probabilities by sequence length for three keyboard applications.The resulting prior supports dictionary attacks ordered by probability when only keystroke count is known.Source: §5.2 · p. 6

6 The Video Processing Pipeline

The pipeline transforms a single typing video into ranked password guesses by tracking the phone and fingers, estimating keystrokes and key probabilities, and accounting for typing and device uncertainty.

6.1 Step 1: Smartphone Tracking

  • SAM 3 segments the phone in each frame and tracks its screen corners as a per-frame quadrilateral.This accommodates natural smartphone movement during typing.Source: §6.1 · p. 6

6.2 Step 2: Initial Hand Tracking

  • MediaPipe supplies initial fingertip positions, while optical flow propagates the last valid prediction through frames where hand tracking fails.The method also back-fills failed initial frames from the first valid detection.Source: §6.2 · p. 6

6.3 Step 3: Sub-Pixel Finger Tracking

  • SAM 3 refines fingertip tracks to sub-pixel precision, merging overlapping masks and matching them to initial tracks to maintain finger identities.A Hungarian-algorithm tracker maintains identities across frames.Source: §6.3 · p. 7

6.4 Step 4: Touch Estimation

  • Candidate keypresses require a vertical-position peak, a corresponding negative acceleration peak, and a fingertip within the screen bounds.A two-component Gaussian mixture sets the prominence threshold, and retained events become high-confidence pseudo-labels.Source: §6.4 · p. 7

6.5 Step 5: Keystroke Prediction via Ensemble CNNs

  • Five R3D-18 networks train on confident pseudo-labels while ambiguous peaks are withheld, then scan full videos at multiple thresholds to retain candidate keystroke lengths.Mixup creates synthetic training samples by interpolating pairs of inputs and labels.Source: §6.5 · p. 7

6.6 Step 6: Video Probability Estimation

  • Observed touch positions estimate keyboard extent, then map to candidate keyboard templates where Gaussian touch mass within each key’s bounds yields per-key probabilities.The keyboard extent estimate relies on touches spanning at least two rows; uncertainty uses globally fixed standard deviations.Source: §6.6 · pp. 7–8

6.7 Step 7: Backspace Detection

  • When backspace ranks among the top five keys, the pipeline removes it and the preceding erroneous character, retaining original and reduced sequences for parallel attacks.Source: §6.7 · p. 8

6.8 Step 8: Rank Estimation

  • ESrank computes prior-only, video-only, and conflated ranks, then adds a multiplicative penalty for parallel candidate lengths, keyboard configurations, and backspace variants.Crack-time conversion assumes a fast-hash rate of 10^10 guesses per second; slower password hashing changes time estimates but not rank or entropy reduction.Source: §6.8 · pp. 8–9

7 The User Study

Across a 16-participant study, the attack was evaluated on policy-compliant human-chosen and password-manager-generated passwords, then tested across physical scenarios, devices, failure cases, and pipeline components. Combining video evidence with password priors substantially improves results for human-chosen passwords, while performance varies with password type and typing conditions.

7 The User Study

  • The study used 16 participants and 18 policy-compliant passwords—nine human-chosen and nine generated—plus repeated backspace trials, for 336 videos total.Participants typed on a Pixel 10 by default; two additional phones were used in selected scenarios, with keyboards left at default settings.Source: §7 · p. 9, §7.1 · p. 9, §7.2 · p. 9

7.2 Performance Across Users

  • 20–60 bits: combining video and prior reduces human-password search space versus prior alone; human passwords up to 16 characters crack in hours to days, and generated passwords up to 13 in under an hour.For generated passwords, video evidence drives performance; from 14 characters onward, crack times extend to centuries even with video.Source: §7.2 · pp. 9–10, Figure 3 · p. 10
  • Human-chosen passwords benefited more from prior statistics than generated passwords, whose prior ranks exceeded 80 bits at lengths as short as nine characters.For generated passwords, video alone reduced search space by 40–70 bits relative to the prior, while combining sources added only slight average benefit.Source: §7.2 · p. 10

7.3 Performance Under Varying Scenarios

  • The attack remained effective to 1.8 m; it worked across tested camera elevations, performed best frontally, and generalized across three smartphones without significant rank differences.Oblique and side views degraded results; low and high elevations also made screen-corner estimation or finger-contact localization harder.Source: §7.3 · p. 11

7.4 Overall Accuracy and Failure Analysis

  • Overall keystroke-recovery accuracy was 77.8%, not a password-cracking rate; missed or extra keystrokes invalidate a candidate sequence and exclude its rank from the attack-strength results.Fast subtle movements, near-simultaneous thumb presses, and hesitations caused distinct detection failures; success fell from 93.8% of users for an eight-character human password to 50.0% for a 16-character random password.Source: §7.4 · pp. 11–12

7.5 Component Ablation

  • Removing SAM 3 finger tracking sharply reduced accuracy, and removing the CNN ensemble left GMM-only accuracy at 11.1% for P3 and 27.8% for P8.Dynamic keyboard ratios performed consistently across devices, unlike fixed ratios; the per-key model was robust for c=0.5, 0.75, and 1.0 but degraded sharply at c=0.25.Source: §7.5 · pp. 12–13, Figure 10 · p. 13, Figure 11 · p. 13

7.6 Computational Cost

  • A one-minute attack video took 10–12 minutes to process on an Intel Core i9-12900KF and NVIDIA RTX 3080 Ti PC.SAM 3 tracking and CNN ensemble training accounted for about five minutes each.Source: §7.6 · p. 13

8 Discussion

The discussion compares the evaluation with prior video-based inference work, presents defenses, and identifies boundaries of the current feasibility study.

8 Discussion

  • The evaluation matches Yang et al.’s 16 participants and 0.8–3.0 m distances while tackling full-QWERTY smartphone passwords rather than tablet text inference.Its 568-video corpus also compares favorably with the cited smartphone-password study by Shukla and Phoa.Source: §8 · p. 13

8.1 Defenses

  • Physically obstructing the attacker’s view of typing fingers is the most effective defense; privacy screens alone do not stop finger-motion inference.Tilting the phone toward the body is one suggested way to block that view.Source: §8.1 · p. 13
  • Extra layout-switch presses can multiply candidate sequences, while longer, more random passwords are harder to crack and generated passwords of 14+ characters remain secure under a successful attack.Biometrics and passkeys remove the typing side channel for authentication, but not for other sensitive text entry; two-factor authentication and attempt limits do not stop offline guessing.Source: §8.1 · p. 13

8.2 Limitations and Future Work

  • Attack performance depends on hand tracking and segmentation; fast typists with minimal finger travel can cause the GMM and CNN stages to fail.The study also used seated participants typing practiced research-selected passwords, and did not systematically evaluate shielding postures.Source: §8.2 · pp. 13–14
  • The evaluation is a controlled feasibility study with relatively stationary attacker and victim, not an in-the-wild test of crowds, low light, movement, or occlusion.The authors expect such active conditions to reduce accuracy because they affect screen-corner stability and fingertip localization.Source: §8.2 · p. 14
  • The pipeline assumes standard QWERTY and manual password entry; swipe input, alternative or custom keyboards, long-press symbols, autofill, copy-paste, and biometric unlock are outside its current scope.Extending the approach beyond short policy-constrained strings is left for future work.Source: §8.2 · p. 14

9 Conclusions

The paper presents a general video-based attack for estimating ranks of policy-constrained smartphone passwords. User studies report crack times and entropy reductions, with scenario analysis spanning distances, angles, and phone models.

9 Conclusions

  • Human-chosen passwords up to 16 characters were cracked in hours to days, while password-manager-chosen random passwords up to 13 characters were cracked in under an hour.The attack reduced password entropy by up to 60 bits versus the prior baseline, and combining video with prior statistics yielded an approximately 20-bit significant reduction versus either source alone for human-chosen passwords.Source: §9 · p. 14
  • The pipeline estimates ranks for rule-based passwords on smartphones without assumptions about the victim or device beyond a frontal view of the typing hand and screen.It combines sub-pixel finger tracking, self-supervised keystroke prediction, dynamic keyboard estimation, and rank computation.Source: §9 · p. 14
  • Scenario analysis found robustness across distances up to 1.8 m, camera elevations and viewing angles, and three popular smartphone models.The authors connect these findings to privacy risks from typing passwords in public amid wearable cameras.Source: §9 · p. 14

A Ethical Considerations

The study describes consent, de-identification, recording, and data-handling measures for its human-subject experiments, and explains its dual-use precautions.

A Ethical Considerations

  • Participants consented, were assigned numeric IDs without linked personal information, and typed research-selected passwords on phones supplied by the researchers.Audio was muted, and recordings were stored on a secure local drive accessible only to the research team.Source: A Ethical Considerations · p. 15
  • The protocol received IRB approval, and published setup screenshots used a researcher and a nonparticipant who approved publication.The paper states that neither person typed sensitive content during filming.Source: A Ethical Considerations · p. 16
  • The authors report no attacks against non-consenting people and withhold the integrated attack code to reduce turnkey misuse.All recordings were made in controlled settings with participant awareness; the authors frame disclosure as a way to raise awareness and motivate defenses.Source: A Ethical Considerations · p. 16

B Smartphone Layouts

The appendix presents four layouts for each of three smartphone keyboard families and notes that their differences affect password typing sequences.

B Smartphone Layouts

  • Figure 12 shows four Gboard layouts; keyboard differences in character placement and primary-layout row counts can change touches and layout switches for the same password.The multi-configuration procedure addresses this heterogeneity.Source: Figure 12 · p. 16, B Smartphone Layouts · p. 16
  • Figure 13 shows four iOS keyboard layouts, part of the layout variation that can alter the touch and layout-switch sequence for a password.The paper says its multi-configuration procedure resolves this heterogeneity.Source: Figure 13 · p. 16, B Smartphone Layouts · p. 16
  • Figure 14 shows four Samsung keyboard layouts; differing character distributions and primary-layout row counts can change how the same password is typed.The multi-configuration procedure handles these keyboard differences.Source: Figure 14 · p. 16, B Smartphone Layouts · p. 16

C Device and Software Versions

The evaluation documents the smartphones, operating systems, and keyboard applications used, with all keyboards in their default configuration.

C Device and Software Versions

  • The evaluation lists smartphone, operating-system, and keyboard-application versions; all keyboards were used in their default configuration.Table 3 covers the devices and software installed at the time of the study.Source: C Device and Software Versions · p. 16, Table 3 · p. 16

D Participants and Passwords

The study reports information about 16 participants and describes its test sets for human-chosen and password-manager-chosen random passwords.

D Participants and Passwords

  • The participant information covers typing behaviors and general details for 16 participants, labeled P0–P15.Source: Table 4 · p. 17
  • The password test set includes human-chosen passwords and password-manager-chosen random passwords.Source: Table 5 · p. 17
Loading 2609.06539v1…