Magic Pen: Automatic Pen Mode Switching for Document Annotation

• Human-centered computing is the broad CCS classification for this work.Source: CCS Concepts · p. 1
• Pointing devices and interaction techniques are the listed interface-related CCS categories.Source: CCS Concepts · p. 1
• Empirical studies in HCI is the listed research-method category.Source: CCS Concepts · p. 1

• Pen interfaces and digital pens frame the interaction technology studied.Source: Keywords · p. 1
• Mode switching and machine learning identify the central technique and approach.Source: Keywords · p. 1
• Annotation and error mitigation identify the task and correction focus.Source: Keywords · p. 1

• Menu-based mode switching can slow annotation through round trips between the document and toolbar, despite annotation often interleaving ink, highlighting, and underlining.Source: §1 · p. 1
• The paper tests machine-learning and rule-based automatic switching and proposes flick and tap gestures to change modes or quickly remove strokes.Source: §1 · p. 1

2.1 Studies of Annotations

• The work focuses on common annotation practices, including marking text, highlighting keywords, and writing in margins, identified in prior studies.Source: §2.1 · p. 2
• Prior annotation research emphasizes directly marking content, supporting varied personal practices, and limiting disruptions; Magic Pen seeks to reduce mode-switching disruptions with automatic methods.Source: §2.1 · p. 2

2.2 The Mode Problem and Optimization Efforts for Pen Interfaces

• The mode problem arises when a pen must support more commands than it can offer at once; forgotten deliberate switches can produce meaningless strokes and require cleanup and repetition.Source: §2.2 · p. 2
• Button-based switching can be awkward during a gesture and still requires users to remember the initial action.Source: §2.2 · p. 2
• Earlier inferred-mode systems used stroke shape and context, but a posteriori detection could alter strokes after creation and be visually disruptive.Source: §2.2 · p. 2
• Magic Pen revisits inferred modes with deep learning and decision trees, analyzing strokes tens of times per second and offering a gesture fallback to cycle recent strokes between modes.Figure 1 shows example recorder task screens for highlighting and drawing.Source: §2.2 · p. 2, Figure 1 · p. 3

3.1 Methodology

• Nine right-handed digital-pen users completed a one-hour session on a Surface Pro 6, with the tablet fixed flat to control external variables.The authors note that this restriction reduced external validity and initially felt unnatural to some participants.Source: §3.1 · p. 3

Setup:

• Participants were encouraged to write naturally and were told handwriting legibility and performance would not be assessed.The study recorded each interaction during a series of text-annotation tasks.Source: Setup · p. 3

Procedure:

• Participants completed four mode-specific tasks twice, then used all tools in an open-ended markup task to capture interleaved annotation.Tasks included highlighting, underlining, striking out, and drawing symbols or writing near text.Source: Procedure · p. 3

3.2 Stroke Recorder System

• The web-based recorder targeted low latency, hardware flexibility, and data completeness, running on pen-enabled devices compliant with the W3 standard.It used the JavaScript Pointer-Event API to capture pen and software state; samples were typically 10–15 ms apart.Source: §3.2 · pp. 3–4

3.3 Results

• 11,338 strokes were collected and divided into ten-sample windows; strike-out was dropped after confusion with highlighting proved error-prone.Pen location and pressure were converted to deltas to reduce learning of tablet-specific absolute positions.Source: §3.3 · p. 4

3.4 Final Feature Set

• The selected feature set included pen movement and pressure, whether the pen was over text, and distance to the nearest text object's bottom.The distance was expressed as a percentage of object height to make it font-size invariant.Source: §3.4 · pp. 4–5

4 Design of Magic Pen

• The design compared learned classification with a manually constructed decision tree, balancing potential hidden-pattern discovery against predictability and explainability.The rule-based system uses relative pen position over text; it does not use pressure or motion dynamics to distinguish strokes outside text.Source: §4 · p. 5, §4.2 · p. 6

4.1 Machine Learning Approach

• An LSTM was chosen to recognize temporal stroke data, with its architecture designed to classify pen modes from collected features.The model was selected based on prior success with temporal data, including stroke and speech recognition.Source: §4.1 · p. 5

4.3 Model Performance

• 88.55% overall accuracy for ML exceeded the Rule-based approach's 72.67% in 10-fold cross-validation.The evaluation used 11,338 strokes randomly sampled from nine participants, with training on 293,997 stroke windows.Source: §4.3 · p. 6
• The ML approach consistently detected highlight and draw despite unusual stroke placement, while underline accounted for most errors; the Rule approach was predictable but confused highlight and underline.Rule-based errors occurred when strokes were not near the text's top or bottom; users adapting their writing could mitigate them, but that conflicts with naturalistic classification.Source: §4.3 · p. 6

4.4 Real-time Mode Selection

• The system forms stroke windows and sets the displayed mode from the three most recent consecutive classifications of the same class.This N=3 setting was reported as a balance between incorporating history and favoring recency.Source: §4.4 · p. 6

4.5 Mitigating Errors

• An eraser-end flick on an empty screen cycles the last stroke's mode, while the eraser removes existing strokes normally.The correction cycle is Highlight → Underline → Draw; a toolbar button can activate eraser mode when no dedicated eraser is available.Source: §4.5 · pp. 6–7

5.1 Methodology

• The remote-deployment study compared ML, Rule, and menu switching with 18 participants using counterbalanced conditions and practice and markup phases.Participants completed 108 annotation tasks across the three conditions.Source: §5 · p. 7, §5.1 · pp. 7–8, Tasks · p. 8

Tasks:

• Participants used different pen modes on randomly selected text passages, then marked all specified passages and reported effort, satisfaction, and use of the flick correction gesture.The task instructions encouraged natural writing and did not assess handwriting legibility.Source: Tasks · p. 8

5.2 Results

• Mode changes between consecutive strokes were more common with ML (29.3%) and Rule (22.2%) than Menu (10.6%), whose users often annotated in mode-specific blocks.The authors note that revealing all target passages at once may have encouraged planning in the Menu condition.Source: §5.2 · p. 8

Corrections and Erasures.

• 749 strokes were edited (15.2% of markup strokes); 212 were mode-corrected with the flick gesture and 537 were erased.Erased strokes could reflect mode errors or writing mistakes, and participants raised concerns about accidental erasure when using the flick gesture.Source: §5.2 · p. 8, Corrections and Erasures · p. 9

subjective feedback.

• Rule was the most frequent favorite (8 participants), but no technique was significantly preferred; Menu scored highest for predictability, while Rule required less effort than ML.The predictability difference favored Menu over ML and Rule; the effort difference favored Rule over ML.Source: Figure 9 · p. 8, Corrections and Erasures · p. 9

5.3 Discussion

• Participants’ preferences reflected trade-offs: some valued menu control and predictability, while others found automation more efficient, especially for frequent mode changes.The authors suggest that the guided, planned tasks may understate the Menu burden in natural annotation, and call for further study in less planned scenarios.Source: Corrections and Erasures · p. 9, §5.3 · p. 10
• 85.7% accuracy was reached by the ML approach, compared with 72.0% for Rule; despite this, participants preferred Rule.The authors identify predictability and handling short strokes as improvement needs for ML.Source: §5.3 · p. 9
• The study identified orientation variation as a possible disadvantage for ML, while the flick gesture could accidentally erase strokes and was not always usable in blank space.The model was trained in a fixed orientation but deployed across device orientations; the authors also note users’ concerns about trusting automatic mode selection.Source: §5.3 · pp. 9–10

6.1 Updated Mode Classification

• Short strokes and strokes unlike learned norms challenged ML classification, including dots on i or j and underlines drifting into the line below.The latter could confuse underline and highlight classifications.Source: §6.1 · p. 10
• A 500 ms rapid heuristic reuses the previous stroke’s mode for a subsequent stroke drawn within the threshold, aiding ambiguous short strokes.A separate confidence heuristic substitutes the previous mode’s stroke when the latest classification confidence is below 75%.Source: §6.1 · p. 10

6.2 Updated Mode Selection

• The revised mode selector requires 25% more sequential classifications for a new mode, building confidence while reducing rapid mode thrashing.This was introduced because lower thresholds allowed incorrect changes and higher thresholds also blocked correct changes.Source: §6.2 · pp. 10–11

6.3 Updated Eraser Mitigation

• The original flick correction was little used because its dedicated gesture, memorized mode sequence, and eraser-tip activation could add effort or cause accidental erasure.The authors conclude that a useful mitigation needs an activation method that is clearly distinguishable.Source: §6.3 · p. 11
• The revised interface lets users remove the most recent stroke by tapping during its five-second orange highlight, avoiding a toolbar round trip.The highlight ends when the next stroke is drawn.Source: §6.3 · p. 11, Figure 12 · p. 11

7 Transfer Learning

• Transfer learning tunes the base model to an individual user’s annotation style, addressing the small dataset and variation across users.The authors connect this approach to the highly individual nature of annotation strokes.Source: §7 · pp. 11–12

7.1 Performance

• Across 27 participants, transfer learning improved average classification performance by 3.44%; P9’s accuracy rose from 72.84% to 86.71%.The simulation used 20% of each held-out participant’s data for transfer learning and the remaining 80% for evaluation.Source: §7.1 · p. 12

7.2 Implementing Transfer Learning in the Annotation Interface

• The interface adds strokes not removed before the next stroke to future training, assuming users remove incorrectly moded strokes before continuing.Training runs at session end or after switching to a manual mode, with a ten-second delay and a minimum of 500 stroke windows.Source: §7.2 · p. 13, stroke data? · p. 13

stroke data?

• New model iterations are retained only when they exceed 80% accuracy on the user’s validation samples, and toolbar colors communicate model and mode-specific performance.Orange indicates 80%–90% performance and green indicates greater than 90%.Source: Figure 14 · p. 12, stroke data? · p. 13

8 Web Annotation System

• The system supports PDF upload, annotation, and download, with manual or automatic mode selection in Chromium-based browsers using compatible pens.The document viewer converts PDFs to SVG and back.Source: §8 · p. 13
• TensorFlow.js moved classification to the client, removing network latency and enabling on-device personalization; Web Workers keep training and classification off the UI thread.The authors note that a deployed application could keep documents and annotations local.Source: §8 · p. 13
• PaperJS canvas rendering was used to address slow SVG rendering for long documents or many annotations, while caching text-box locations sped pen-position feature calculations.The canvas is converted to SVG and then PDF for download.Source: §8 · p. 13

9.1 Methodology

• Eight participants completed five remote sessions over two weeks, annotating freely and retraining after at least 10 unique, somewhat long strokes; only one supplied their own documents.The other participants used provided samples.Source: §9.1 · pp. 13–14, §9.2 · p. 15

Introductory Session.

• Questionnaire measures improved from the first to final session, while automatic-mode use averaged over half of strokes and increased over time.Fewer strokes were removed in later sessions.Source: Figure 16 · p. 15, Introductory Session · pp. 15–16, Figure 18 · p. 16
• Hold-out validation accuracy ranged from 80–100% across retraining, but changing recent-stroke test sets made session comparisons difficult; final accuracy averaged 5% lower than the first transfer-learning session.Despite this, all but one participant ranked final accuracy 4 or 5.Source: Introductory Session · p. 16
• Seven participants reported underline-mode errors, but by the second-last session seven said all modes worked similarly; some removed strokes appeared unrelated to mode errors.For P5, most removed strokes appeared correctly moded and were erased in a single event.Source: Introductory Session · p. 16

9.2 Results

• Magic Pen preference averaged M=3.88 (SD=1.0), and Frustration was the strongest correlate of Preference (R=0.78, p=0.02).Participants who reported lower preference were largely P1 and P3.Source: Introductory Session · p. 16

9.3 Discussion

• As users learned how the model worked, they could adapt to improve accuracy and reduce blame and frustration, although this departs from the goal of fully naturalistic inference.The authors suggest some human–model co-evolution may be inevitable.Source: §9.3 · p. 17
• Because recent-stroke validation sets produced fluctuating accuracy, the authors recommend accumulating a longer-term validation set across sessions to improve measurement stability.The authors describe this as a future recommendation.Source: §9.3 · p. 17
• Close line spacing or documents without text caused unsatisfactory performance for one participant, motivating training across varied documents as well as users.The authors note that some useful data features depend on document elements.Source: §9.3 · p. 17

10 Conclusion and Future Work

• Magic Pen uses low-latency implicit pen-use signals, refined machine-learning classification, short-stroke heuristics, error mitigation, and iterative transfer learning for personalization.Source: §10 · p. 17
• Participants preferred refined automatic switching over menus, with preference related to model accuracy, predictability, and frustration.Source: §10 · p. 17
• Browser implementation required significant engineering; future work could add hover or gaze features, continuous learning, GPU processing, or alternative models.Source: §10 · p. 17
• Explainable ML may help users build a mental model and support co-evolution of the model with annotation behavior.Source: §10 · p. 17
• Inferring intents beyond ink annotation modes, including pen-based selection or word-definition queries, remains an improvement opportunity.Source: §10 · p. 17
• Success with a small feature set, in-browser classification, and training initialized with 9 participants suggests promise, while graceful failure handling remains open.Source: §10 · p. 17