PaperlayerView exact revision on arXiv

Source-linked AI summary

NeuroClick: Preserving Surgeon Autonomy through Hands-Free Earable Tooth-Click Control in Neurosurgery

Jonas Hummel, Maximilian Burzer, Clara Sayffaerth, Valeria Zitz, Amir El Rahal, Michael Küttner, Tobias Röddiger, Jürgen Beck, Michael Beigl

arXiv:2609.12910v1cs.HC

TL;DR

Neurosurgeons need to interact with OR technologies despite sterility and occupied hands, while existing control options draw on resources that can be constrained. This paper develops and evaluates earable tooth-click control, grounded in OR observations and expert interviews. The classifier achieved a median macro F1-score of 98.6%, and the simulated task found higher autonomy ratings with Earable but faster task completion with Delegation under continuous assistant availability.

  • Problem

    Sterility and occupied hands constrain neurosurgeons’ interaction with OR technologies, while available control approaches draw on resources that may themselves be constrained.

    Source: §1 · p. 2

  • Method

    The authors grounded earable tooth-click control in multidisciplinary conceptualization, OR observations, and expert interviews, then developed and evaluated a real-time recognition pipeline and interaction technique.

    Source: §1 · p. 2, §3 · p. 4, §3.1 · p. 4

  • Results

    98.6% median macro F1-score was achieved under leave-one-subject-out cross-validation, and neurosurgeons rated autonomous control higher with Earable while Delegation enabled faster task completion under continuous assistant availability.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    Earables add a direct, hands-free option for controlling selected functions alongside established workflows.

    Source: §1 · p. 2

    The neurosurgeon evaluation used a simulated task that did not reproduce the complexity, duration, or risk of complete procedures.

    Source: §6.5 · p. 20

Abstract

from arXiv · show

Neurosurgeons frequently interact with operating room (OR) technologies while sterility and occupied hands constrain control. We introduce earables as a direct, hands-free control platform for neurosurgery using tooth-click input. Formative OR observations and interviews with 10 domain experts grounded the design. Using OpenEarable 2.0 data from 12 participants, we developed a real-time recognition pipeline whose classifier achieved a median macro F1-score of 98.6% under leave-one-subject-out cross-validation. We evaluated the technique with 20 neurosurgeons during a simulated resection task in a neurosurgical OR. Participants reported few focus shifts and rated Earable favorably for workflow integration and perceived safety. Autonomous microscope control was rated significantly higher with Earable than Delegation, whereas Delegation enabled faster task completion under continuous assistant availability. Workload, usability, and task errors showed no significant differences. Preferences depended on training, reliability, context, and assistant availability. Earables thus add a direct, hands-free option for controlling selected functions alongside established workflows.

1 Introduction

The paper introduces earables as a direct, hands-free control option for neurosurgery, addressing OR interaction constraints while preserving surgeon autonomy. It grounds tooth-click control in clinical input, develops a recognition pipeline, and evaluates the technique against Delegation.

1 Introduction

  • The OR constrains interaction because sterility and occupied hands limit direct control, while delegation and other hands-free approaches draw on resources that may also be constrained.The paper frames the challenge as a need for additional direct-control options, not a lack of interaction techniques.Source: §1 · p. 2
  • Earables are introduced as an ear-worn interaction platform, extending prior earable use beyond communication support toward direct control in a safety-critical clinical setting.The design tension is to support relevant functions while remaining simple, reliable, and unobtrusive.Source: §1 · p. 2
  • In a simulated resection task, Earable was rated higher than Delegation for autonomous control, while Delegation enabled faster task completion under continuous assistant availability.Workload, usability, and task errors showed no statistically significant differences; preferences varied with training, reliability, context, and assistant availability.Source: §1 · p. 2
  • The study grounds tooth-click interaction in OR observations and interviews with 10 domain experts, then develops a real-time recognition pipeline for OpenEarable 2.0.The contributions include an empirical characterization of interaction challenges and design requirements.Source: §1 · p. 3
  • 98.6% median macro F1-score was achieved by the tooth-click classifier under leave-one-subject-out cross-validation with 12 participants.This is the reported recognition result for the OpenEarable 2.0 pipeline.Source: §1 · p. 3

2 Related Work

Prior OR interaction approaches trade off direct control, hands-free operation, attention, ergonomics, and workflow fit. Tooth-based earable input has technical precedent, but its design and appropriate use in safety-critical clinical interaction remain underexplored.

2.1 Direct Control Options in the OR

  • In microscope-based neurosurgery, external controls can interrupt workflow and attention, and surgeons may use workarounds such as temporarily breaking sterility to preserve procedural flow.Prior work therefore frames interaction as preserving attention and continuity, not merely issuing commands.Source: §2.1 · p. 3
  • Hand controls may require releasing instruments or reaching away; pedals and mouthpieces impose ergonomic constraints, gaze and speech can compete with attention or communication, and instrument control couples interaction to the surgical task.Prior work motivates complementary control options rather than a single optimal modality.Source: §2.1 · p. 3

2.2 Earable Interaction and Tooth-Based Input

  • Tooth-based earable systems range from simple clicks to more expressive gesture vocabularies, but have predominantly been developed for everyday rather than safety-critical clinical settings.Neurosurgical use raises design needs around unobtrusiveness, reliable operation, unintended activation, and interaction complexity.Source: §2.2 · p. 4

3 Concept Development and Clinical Design Implications

The authors combined multidisciplinary concept development, observation of neurosurgical procedures, and expert interviews to ground earable interaction design in clinical workflows. The formative work characterized existing controls and communication practices that informed subsequent design requirements.

3.1 Initial Conceptualization

  • Multidisciplinary conceptualization treated earables as a candidate platform and used clinical and technical perspectives to derive design requirements.Tooth-clicking was considered promising for anticipated detection reliability, hands- and eyes-free operation, limited workflow disruption, and low cognitive effort.Source: §3 · p. 4, §3.1 · p. 4

3.2 Formative OR Observation

  • Across three observed procedures, surgeons relied primarily on foot pedals and microscope handgrips, while handgrip buttons were barely used during surgery.The microscope and its distributed controls are shown in the supplied figures; a pedal slid out of reach during one instance.Source: §3.2 · p. 5, Figure 2 · p. 5, Figure 3 · p. 5
  • Communication during the observed procedures was constant and complicated by masks and equipment noise, with surgeons repeatedly requesting assistance and status information.A floating nurse also navigated a non-sterile screen and repeatedly held a telephone to the surgeon’s ear.Source: §3.2 · pp. 5–6

3.3 Domain Expert Interviews

  • The formative interview study involved 10 domain experts from nine university hospitals in Germany and Switzerland, including nine neurosurgeons and one neuromonitoring specialist.Interviews were semi-structured and examined interaction challenges and requirements in greater depth.Source: §3.3 · p. 6

Appendix A.

The supplementary material describes the interview coding process, while the formative findings identify interaction disruptions, deployment requirements, and suitable tooth-click use cases. These findings inform the clinical evaluation criteria and comparison with Delegation.

Appendix A.

  • Interview analysis used two coders, refined codebooks through discussion, and produced a final codebook of 188 codes organized into 20 higher-level categories.Intercoder agreement rose from α_cu = .67 to α_cu = .79 during the described procedure.Source: Appendix A · p. 6

3.3.1 Current Interaction Challenges.

  • Participants reported interaction challenges involving reduced autonomy, workflow disruptions, OR constraints, and time losses.The passage reports that 80% described existing mechanisms as restricting autonomy and 70% reported workflow disruptions overall.Source: §3.3.1 · p. 6

3.3.2 Earable Opportunities and Deployment Requirements.

  • Participants envisioned earables supporting broader OR workflows, while stable fit was the most prominent deployment requirement and some raised concern about reduced auditory perception.Stable fit was mentioned by 60% and reduced auditory perception by 20%.Source: §3.3.2 · p. 6

3.3.3 Tooth-Clicking for Surgical Control.

  • Tooth-clicking was primarily envisioned for discrete microscope commands, with participants favoring a small, simple gesture set and reliable recognition.Suggested uses included illumination, image or video capture, fluorescence, focus, and zoom; double clicks were proposed most frequently, and participants also raised unintended activation and context-of-use concerns.Source: §3.3.3 · pp. 6–7

3.4 Design Requirements and Research Questions

  • The formative findings positioned tooth-clicking for a small set of microscope functions and made focus shifts, autonomy, workflow integration, and perceived safety evaluation criteria.Delegation served as the clinical reference for the within-participant comparison of the same microscope functions.Source: §3.4 · p. 7

4 Developing the Real-Time Tooth-Click Interaction Technique

The paper develops a real-time tooth-click interaction pipeline for OpenEarable 2.0, separating physical click detection from configurable gesture and command mapping. It evaluates recognition using data collected under simulated surgical conditions.

4.1 Design and System Overview

  • OpenEarable detects physical clicks and sends timestamped events to a phone, which recognizes temporal gestures and maps them to commands.This separation keeps click detection fixed while allowing gesture definitions and command mappings to be adapted without retraining or firmware changes.Source: §4.1 · p. 7, Figure 4 · p. 8, §4.3.2 · pp. 9–10

4.2 Data Collection

  • Twelve participants recorded six tooth-click conditions and five competing activities, with and without simulated OR audio, using microphones and IMU sensors.The earable was removed and reinserted between blocks to introduce variation in device placement.Source: §4.2 · p. 8, Figure 5 · p. 8, Figure 6 · p. 9

4.3.1 Tooth-Click Detection.

  • 98.6% median LOSO macro F1 was achieved by a compact MLP using 20 bone-conduction features, the best performance-complexity trade-off among evaluated configurations.The lowest-performing held-out participant reached 93.8%; dental information suggested missing teeth may contribute to inter-participant variability.Source: §4.3.1 · p. 9, Figure 7 · p. 10

4.3.2 Real-Time Interaction Pipeline.

  • The evaluation used fast and slow double-clicks and triple-clicks as three discrete commands, omitting single-click commands to prioritize resistance to accidental activation.Gesture definitions were configurable by click count and inter-click timing.Source: §4.1 · p. 7, §4.3.2 · p. 10

5 Clinically Situated Evaluation with Neurosurgeons

The study compared direct earable control with assistant-mediated delegation in a counterbalanced, within-participant simulated resection task. Neurosurgeons used the same microscope functions in both conditions while removing fluorescent targets from a phantom.

5.1.1 Study Design and Conditions.

  • Neurosurgeons completed a counterbalanced, within-participant comparison of direct Earable control and verbal requests to the same physician assistant.Both conditions provided access to fluorescence, autofocus, and image capture.Source: §5 · p. 10, Figure 8 · p. 10, §5.1.1 · p. 11

5.1.2 Simulated Resection Task.

  • Participants removed 20 fluorescent targets from a 40-thumbtack phantom, switching between fluorescence for identification and normal illumination for removal.The same target layout was used with counterbalanced assignments across conditions.Source: §5.1.2 · pp. 11–12, §5.1.1 · p. 11, Figure 10 · p. 12

5.1.3 Measures.

  • Measures covered four interaction criteria, workload, usability, completion time, aggregate errors, and interview accounts of experience and preferences.Aggregate errors combined erroneously removed non-targets and missed targets.Source: Table 1 · p. 11, §5.1.3 · p. 12

Appendix D.

In the simulated resection study, Earable received higher ratings than Delegation for autonomous microscope control, while task completion took longer with Earable. Participants described benefits and trade-offs that varied with training, reliability, surgical context, and assistance availability.

5.1.4 Experimental Apparatus.

  • Three FingerBots on the microscope handgrip actuated fluorescence, autofocus, and image capture without modifying the microscope’s internal control system.A command-specific tone was played through the earable after phone-side gesture recognition.Source: §5.1.4 · p. 12, Figure 11 · p. 13

5.1.5 Procedure.

  • Before the Earable condition, participants verified device fit and practiced until they executed each tooth-click command twice without assistance.They completed the task and post-condition measures in both conditions, then took part in an interview.Source: §5.1.5 · p. 12

5.1.6 Participants.

  • The analysis included 20 neurosurgeons with varied seniority and an average of 9.35 years of neurosurgical experience.One recruited participant was excluded from all analyses after requiring repeated procedural guidance and withdrawing consent for interview audio recording.Source: §5.1.6 · pp. 12–13

5.1.7 Analysis.

  • Paired outcomes were analyzed with two-sided permutation Wilcoxon signed-rank tests and Holm correction, while interview transcripts were coded into seven higher-level categories.Agreement on four additional interviews reached Krippendorff’s αcu = .95.Source: §5.1.7 · p. 13

5.2.1 Interaction Criteria.

  • Earable was rated significantly higher than Delegation for autonomous microscope control (pHolm = .008, rrb = .80); its focus-shift ratings were favorable, but the condition difference was not significant after correction.Workflow integration and perceived safety were rated favorably for Earable, with no significant condition differences.Source: §5.2.1 · p. 14, Table 2 · p. 14, Figure 12 · p. 14

5.2.2 Workload and Usability.

  • Overall workload and usability did not differ significantly between conditions, and none of the six NASA-TLX dimensions was significant after Holm correction.Effort and frustration were descriptively higher with Earable, while the other dimension differences were smaller.Source: §5.2.2 · pp. 14–15, Table 3 · p. 15, Figure 13 · p. 15

5.2.3 Task Performance.

  • 146.5 s with Earable versus 136.5 s with Delegation was the median completion time; aggregate errors were zero in both conditions and did not differ significantly.The completion-time difference was significant (pHolm = .016, rrb = .66).Source: §5.2.3 · p. 15, Table 3 · p. 15, Figure 13 · p. 15

5.2.4 Qualitative Benefits and Trade-Offs.

  • Participants associated direct control with autonomy and reduced dependence on assistance, while reporting learning overhead and reliability friction; preferences and proposed design changes varied by context and training.Recognition refinement was the most frequently proposed improvement, mentioned by 50%.Source: §5.2.4 · pp. 16–17

6 Discussion

The discussion frames tooth-click earables as a complementary control channel whose value depends on the surgical task and available assistance. It also identifies design and evaluation boundaries for the prototype.

6.1 Earables as a Complementary Direct-Control Resource

  • Tooth-clicking supports discrete commands, not continuous or fine-grained control, so it complements rather than displaces established inputs.Participants reported few focus shifts, and focus-shift ratings were descriptively lower than for Delegation.Source: §6.1 · pp. 17–18

6.2 Situational Autonomy: When to Control Directly and When to Delegate

  • Earable preserves direct control for selected commands, but Delegation completed tasks faster with a continuously available assistant.Autonomy ratings were significantly higher with Earable; preferences varied with assistant availability, interaction frequency, access constraints, and procedure length.Source: §6.2 · p. 18

6.3 Designing Adaptable and Reliable Earable Input

  • Separating click detection from gesture interpretation permits configurable timing and mappings without retraining the detector.All 20 evaluated neurosurgeons were unseen during classifier development and used the same pretrained classifier without participant-specific calibration.Source: §6.3 · p. 18

6.4 Earables as a Platform for Control and Communication

  • Earables could combine discrete tooth-click control with communication and other interaction modes at a location available when hands and gaze are occupied.Participants suggested using a click as a clutch for speech interaction.Source: §6.4 · p. 19

6.5 Limitations

  • Evidence is bounded by a simulated task, experimental prototype, initial-use evaluation, and self-reported interaction criteria rather than full-procedure or behavioral measures.The study also compared Earable only with Delegation at one site; the assistant was continuously available, and the fixed mappings left personalization untested.Source: §6.5 · p. 20

6.6 Outlook

  • A behind-ear bone-conduction form factor may address fit and auditory-occlusion concerns, while the interaction concept may also suit other hands-occupied technical settings.The cited prior work demonstrated behind-ear tooth-click detection [60].Source: §6.6 · pp. 20–21

7 Conclusion

The conclusion presents tooth-click earables as a hands- and eyes-free option for selected functions, with autonomy benefits but no universal advantage over Delegation. The appendices describe the interview guide and classifier-development procedures.

7 Conclusion

  • Earables provide direct control for selected discrete functions, while Delegation remained faster with continuous assistant availability and no significant differences appeared in workload, usability, or task errors.The conclusion characterizes the technique as an addition to, not a replacement for, established controls.Source: §7 · p. 21

Appendix A: Domain Expert Interview Guide

Appendix B: Detailed Description of the Tooth-Click Interaction Technique Development

  • The appendix details how recordings became training examples and how the classifier was developed and integrated into an on-ear pipeline.The data-collection protocol is reported in the main text.Source: Appendix B · p. 28

B.1.1 Click Annotation.

  • Click annotations were localized within prompted intervals by combining in-ear and bone-conduction signal evidence, then checking candidate peaks against expected click counts.Candidates were initially separated by at least 100 ms and visually confirmed.Source: §B.1.1 · p. 28

B.1.2 Window Generation.

  • Training examples used 100 ms windows, shifted positive windows, and balanced click and non-click sampling that included challenging non-click activities.Negative sampling excluded prompted click intervals and regions within 150 ms of annotated clicks.Source: §B.1.2 · p. 28

B.1.3 Feature Extraction.

  • The feature set used time-domain statistics from microphone, bone-conduction, accelerometer, and gyroscope signals, with training-only standardization for inference.Features included RMS, peak amplitude, crest factor, difference RMS, and zero-crossing rate; time-domain features avoided FFT and buffering requirements during embedded inference.Source: §B.1.3 · pp. 28–29

B.1.4 Classifier Training and Evaluation.

The classifier was trained to detect individual tooth clicks in 100 ms windows and evaluated on held-out participants. The supplied figure caption identifies modality and architecture comparisons, but gives no outcome values.

B.1.4 Classifier Training and Evaluation.

  • Each binary MLP classified standardized features from a 100 ms window as containing an individual tooth click or not.A sigmoid converted its output logit to a click probability, with 0.5 as the classification threshold.Source: §B.1.4 · p. 29
  • The evaluation used 12-fold leave-one-subject-out cross-validation, holding out each participant and grouping complete experimental intervals in the training-validation split.Macro-F1 across click and non-click classes was summarized by the median across held-out participants.Source: §B.1.4 · pp. 29–30
  • The comparison figure covers median LOSO macro-F1 and performance–complexity across sensor configurations and MLP architectures.Its caption specifies bone-conduction microphone features for the architecture comparison; Table 4 summarizes candidate feature dimensionalities.Source: Table 4 · p. 29, Figure 14 · p. 30

B.1.5 Sensor and Architecture Selection.

The final classifier uses a compact bone-conduction microphone representation and single-hidden-layer architecture, with click detection deployed as a continuous streaming pipeline.

B.1.5 Sensor and Architecture Selection.

  • 0.986 median LOSO macro-F1 was achieved by the final classifier, with scores from 0.938 to 0.993 across 12 held-out participants.Activity-stratified analysis found chewing gum had the lowest median correct rejection rate (97.6%), while posterior double clicks had the lowest median correct detection rate (97.1%).Source: Figure 15 · p. 31, §B.1.5 · p. 31
  • Bone-conduction microphone features alone achieved 0.986 median LOSO macro-F1 with 20 input features, versus 0.946 for conventional-microphone-only, 0.800 for gyroscope-only, and 0.700 for accelerometer-only configurations.Adding other sensor streams changed performance only marginally while increasing input features and required streams.Source: §B.1.5 · p. 31
  • The selected single 32-unit hidden layer used 705 trainable parameters; deeper architectures did not improve median LOSO performance.Single-hidden-layer architectures formed the performance–complexity Pareto frontier.Source: §B.1.5 · p. 31

B.2.1 Model Deployment.

  • The model was exported to a C99 inference bundle for OpenEarable 2.0, and comparison on identical inputs verified numerical equivalence with the Python implementation.The generated implementation used fixed-size memory buffers and required no dynamic memory allocation during inference.Source: §B.2.1 · p. 32

B.2.2 Continuous Click Detection.

  • The streaming detector uses a signal-based wake-up trigger, evaluates overlapping 100 ms windows, and consolidates thresholded predictions into timestamped click events.The trigger uses bone-conduction microphone signals; event consolidation groups predictions and retains the highest-scoring event when accepted events are within 125 ms.Source: §B.2.2 · p. 32

B.2.3 Firmware Integration.

The embedded detector publishes timestamped click events to a paired phone, where editable temporal rules compose them into gestures. The supplied passages also describe the questionnaires and participant information collected for the study.

B.2.3 Firmware Integration.

  • The firmware integrated the inference pipeline as a virtual sensor on the nRF5340 application core, processing the bone-conduction stream and publishing accepted click events with timestamps to the paired phone.Starting the virtual sensor automatically enabled the stream; processing ran in a dedicated inference thread.Source: §B.2.3 · pp. 32–33

B.2.4 Phone-Side Gesture Recognition.

  • The phone application applies editable gesture rules based on click count, timing intervals, and priority, keeping gesture definitions configurable without retraining or reflashing the earable.The evaluation used fast double-click, slow double-click, and triple-click gestures.Source: §B.2.4 · p. 33, Table 6 · p. 33

Appendix C: User Study Questionnaires

  • The questionnaires supported a counterbalanced within-participants comparison of delegation with earable-detected tooth-click control of the microscope.The supplied questionnaire section identifies the two conditions but does not provide further questionnaire items.Source: Appendix C · p. 34

Study and Demographic Information.

Dental Information.

  • Dental information covered wisdom teeth, other missing teeth, restorations and their locations or materials, and malocclusions.Participants were asked to describe applicable malocclusions.Source: Dental Information · p. 34

Post-Condition Questionnaires

  • The post-condition measures used introductory prompts adapted to the delegation or earable condition.The supplied passage describes condition-specific prompts but does not enumerate the measures.Source: Post-Condition Questionnaires · p. 34

NASA Task Load Index.

The supplied passages identify workload, usability, and interview instruments used after the study conditions. They specify the usability topics and interview questions, but provide no responses or outcomes.

NASA Task Load Index.

  • Workload measurement followed the NASA Task Load Index described by Hart and Staveland (1988).The supplied passage does not specify the instrument's items.Source: NASA Task Load Index · p. 34

System Usability Scale.

Semi-Structured Interview.

  • The interviews asked about overall experience, advantages and limitations, condition preference, and possible improvements to tooth-click control.Questions were asked following both conditions.Source: Semi-Structured Interview · p. 36
Loading 2609.12910v1…