Source-linked AI summary
A Real-Time Error Prevention System for Gaze-Based Interaction in Virtual Reality Based on Anomaly Detection
Björn R. Severitt, Yannick Sauer, Nora Castner, Siegfried Wahl
TL;DR
Gaze-based interaction offers hands-free input but can register unintended selections, and such errors persist even with confirmation mechanisms. This paper evaluates a real-time gaze-anomaly error prevention system across three selection methods in VR; it reduced errors, especially for methods prone to them, and was generally well received.
Problem
Unintended gaze can be mistaken for intentional input, and false selections persist even with confirmation mechanisms.
Source: §1 · p. 1Method
A real-time autoencoder with a Temporal Convolutional Network reconstructs normal gaze patterns and flags deviations, and is tested with three selection methods in a VR visual-search task.
Source: §1 · p. 2, §3.1 · p. 2, §3.2 · p. 4Results
The system significantly reduced incorrect selections across all three methods, with higher task ratings for Dwell Time and Gaze and Head and minimal benefit for Nod; participants generally received it positively.
Source: §5 · pp. 9–10Takeaways & Limitations
The system supports real-time error prevention for methods prone to incorrect selections, while participant variability points to a need for adaptive approaches.
Source: §5 · p. 10, §7 · p. 10Evidence comes from a controlled VR visual-search task, and transfer to other tasks or environments may require adjustments or new training data.
Source: §6 · p. 10
Abstract
from arXiv · showhide
Gaze-based interaction enables intuitive, hands-free control in immersive environments, but remains susceptible to unintended inputs. We present a real-time error prevention system (EPS) that uses a temporal convolutional network autoencoder (TCNAE) to detect anomalies in gaze dynamics during selection tasks. In a visual search task in VR, 41 participants used three gaze-based methods - dwell time, gaze and head direction alignment, and nod - with and without EPS. The system reduced erroneous selections by up to 95% for dwell time and gaze and head, and was positively received by most users. Performance varied for nodding and between individuals, suggesting the need for adaptive systems. Objective metrics and subjective evaluations show that anomaly-based error prevention can improve gaze interfaces without disrupting interaction. These findings demonstrate the potential of anomaly-based error prevention for gaze interfaces and suggest applications in VR, AR, and assistive technologies.
1 Introduction
Gaze-based interaction offers intuitive, hands-free control, but unintended gaze can be mistaken for intentional input. This study evaluates a real-time anomaly-based error prevention system across three gaze-selection methods and examines both error reduction and user experience.
1 Introduction
- Gaze interaction offers intuitive, hands-free input in immersive and assistive settings [Severitt et al. 2024a; Sidenmark et al. 2021], but the Midas touch problem registers unintended gaze as input [Jacob 1990].Source: §1 · p. 1
- Head gestures, dwell times, and gaze-head alignment can confirm selections, yet false selections persist; this work targets gaze dynamics near selection for early detection.The introduction notes that gaze behavior can reflect confusion or correction before confirmation [Severitt et al. 2024b].Source: §1 · p. 1
- The system is intended to reduce errors in the background without disrupting interaction, supporting smoother eye-based interaction in consumer and assistive technologies.Source: §1 · pp. 1–2
- The study evaluates a real-time autoencoder with a Temporal Convolutional Network in a VR visual-search task, comparing Dwell Time, Gaze and Head, and Nod with and without the system.It asks whether the system prevents errors across methods and how participants assess its usefulness, trust, and overall preference.Source: §1 · p. 2
2 Related Work
Prior work applies machine learning and sequence models to gaze behavior, but approaches based on fixed error classes can struggle with unknown errors. This study instead treats selection errors as anomalies in continuous gaze signals.
2 Related Work
- Gaze interaction spans assistive technologies, AR/VR, and HCI, with reducing unintended selections identified as a goal [Istance et al. 2008; Parisay et al. 2021; Rajanna and Hammond 2016; Gardony et al. 2024; Mohan et al. 2018; Schweigert et al. 2019; Velloso et al. 2016; Gemicioglu et al. 2023].Source: §2 · p. 2
- Machine learning applied to gaze data in a visually complex game reduced false alarms and improved user satisfaction without explicit intention confirmation [Shevtsova et al. 2025].Source: §2 · p. 2
- TCN sequence models have classified errors and predicted intentions from gaze histories, while multimodal inputs improved error classification [Sendhilnathan et al. 2025]; fixed error taxonomies hinder generalization to unknown errors.Source: §2 · p. 2
- This study treats selection errors as anomalies in continuous gaze signals, extending a theoretical proof-of-concept [Severitt et al. 2025a] by evaluating models on real VR target-selection recordings.The approach frames errors as rare, statistically distinct events rather than predefined classes [Schmidl et al. 2022].Source: §2 · p. 2
3 Methods
The methods pair reconstruction-based anomaly detection of gaze dynamics with a VR visual-search evaluation of three gaze-selection techniques, measuring both task performance and user perceptions.
3.1 Anomaly detection model
- The TCNAE reconstructs normal-selection inputs, and a fixed reconstruction-error threshold classifies selections as correct or incorrect.Its encoder compresses the sequence into a latent vector before the decoder reconstructs the original shape.Source: §3.1 · p. 2, Figure 1 · p. 3
- Angular-velocity sequences from gaze vectors recorded around selection replace vector differences to reduce sensitivity to frame losses.The window spans approximately 200 ms before and after selection at 90 fps, assuming no dropped frames.Source: §3.1 · p. 3
- Separate networks were trained for each interaction method on correct selections, using a 95th-percentile training-error threshold and five-fold cross-validation.Training used data from a previous experiment; the network used MSE loss and the Adam optimizer.Source: §3.1 · pp. 3–4
3.2 Game
- The VR visual-search game asks participants to select a “C” target among “O” distractors, using Dwell Time, Gaze and Head, or Nod.Dwell Time used 0.4 seconds and Gaze and Head used 0.3 seconds; incorrect selections incurred a 21-point penalty.Source: §3.2 · p. 4
3.3 Study
- Forty-one participants were recruited, and four withdrew; the remaining participants completed randomized rounds with and without EPS for each gaze method.The experiment used an HTC Vive Pro Eye with a built-in eye tracker sampling at 120 Hz.Source: §3.3 · p. 5
3.4 Measurements
- Evaluation combined participant ratings of six EPS experience dimensions with points, incorrect selections, and method-specific classification accuracy.The questionnaire also asked whether participants found EPS helpful and which interaction method they preferred.Source: §3.4 · p. 5
3.5 Data analysis
- The analysis used Friedman tests for related-group comparisons and Wilcoxon signed-rank tests for pairwise comparisons, with Bonferroni correction for multiple comparisons.Effect sizes were reported as eta-squared for Friedman tests and r for Wilcoxon comparisons.Source: §3.5 · p. 5
4 Results
The results combine participants’ perceptions of the EPS and interaction methods with objective measures of task performance and system accuracy. With EPS active, errors fell for Dwell Time and Head and Gaze, while Nod and individual performance showed variation.
4.1 Subjective measurements
- Most participants found the EPS helpful, while only a small number were uncertain and one participant rated it not helpful.Table 2b summarizes perceived helpfulness.Source: §4.1 · p. 6, Table 2 · p. 7
- 79.5% for Dwell Time, 70.8% for Gaze and Head, and 55% for Nod were the accuracies for recognizing whether EPS was active.Nod recognition was only slightly better than chance.Source: §4.1 · p. 6
- Helpfulness, trust, confidence boost, and preference ratings differed significantly by method, with Nod rated lower than Dwell Time and Head and Gaze.Frustration and disruption ratings did not differ significantly and were generally low.Source: §4.1 · p. 6, Figure 4 · p. 7
4.2 Objective measurements
- 22 to 1 incorrect selections for Dwell Time and 8 to 1 for Head and Gaze with EPS active; Nod’s median stayed 0 in both conditions despite a significant error reduction.The comparison is shown in Fig. 5a.Source: §4.2 · p. 6, Figure 5 · p. 8
- 435 to 925 median points for Dwell Time and 745 to 819 for Head and Gaze with EPS active; Nod showed no significant difference, with medians of 787 and 789.The comparison is shown in Fig. 5b.Source: §4.2 · p. 8, Figure 5 · p. 8
- EPS classification was high across methods, with more balanced correct- and incorrect-selection accuracy for Dwell Time and Head and Gaze than for Nod.Nod had very high correct accuracy but lower accuracy for incorrect selections; macro and micro accuracy indicated consistency across methods with variation in that balance.Source: §4.2 · p. 8
- EPS performance was good for most participants, several achieved perfect accuracy, and some had lower performance, particularly in correct accuracy.Gaze and Head had more outliers than Dwell Time; only a few participants made incorrect selections with EPS active for Nod.Source: §4.2 · p. 8
5 Discussion
The EPS reduced errors across interaction methods and improved task performance where errors were more likely, but its benefit varied by method and individual. Participants received it positively, while the discussion highlights trust and adaptability as considerations.
5 Discussion
- The evaluation used ten rounds per method, split between EPS and non-EPS conditions, and assessed both performance and questionnaire responses.EPS performance was assessed using correct and incorrect accuracy metrics.Source: §5 · p. 8
- Table 3 defines correct and incorrect accuracy as class-specific rates, with macro accuracy averaging them and micro accuracy weighting by sample count.Source: Table 3 · p. 9
- Trust remains important because lost trust can be difficult to rebuild, as Rittenberg et al. [2024] highlight.Source: §5 · p. 9
- With EPS enabled, incorrect selections fell across all three methods; task ratings improved for Dwell Time and Gaze and Head, while the benefit was minimal for Nod.The discussion characterizes the EPS as most effective for methods prone to incorrect selections and less effective for inherently more robust methods.Source: §5 · pp. 9–10
- Correct accuracy was consistently high, but Nod’s small number of incorrect selections limited meaningful assessment of incorrect accuracy.The discussion also reports that overall accuracy was slightly below the authors’ previous proof-of-concept result.Source: §5 · p. 10
- The EPS was positively received, but participants rated it as less useful for Nod, where errors were rare.The discussion notes that participants still appreciated the system even when their accuracy scores were comparatively low.Source: §5 · p. 10
- Performance varied among participants, suggesting a uniform approach may be insufficient; future versions could adjust thresholds or detection strategies dynamically.The authors connect these differences to individual viewing behavior and interaction styles.Source: §5 · p. 10
6 Limitations
The evidence comes from a controlled VR visual-search game, so transfer to other tasks and environments remains uncertain.
6 Limitations
- The findings concern a visual-search task in a controlled VR game and may not transfer directly to other applications or interaction contexts.Source: §6 · p. 10
- Dwell time may be less effective in real-world settings with more unintended fixations and environmental distractions.Source: §6 · p. 10
- Applying the approach to other tasks or augmented reality may require adjustments and a new training dataset; broader validation is needed.Source: §6 · p. 10
7 Conclusion
The study concludes that real-time anomaly-based error prevention can support gaze interaction in VR, with benefits varying across methods and participants.
7 Conclusion
- Real-time anomaly-based EPS reduced errors and improved performance for error-prone methods, while benefits were limited for Nod; participant variability points to adaptive approaches and evaluation beyond VR.Source: §7 · p. 10