Source-linked AI summary
GazeIntent: Adapting dwell-time selection in VR interaction with real-time intent modeling
Anish S. Narkar, Jan J. Michalak, Candace E. Peacock, Brendan David-John
TL;DR
Gaze-only selection must balance fast selection against unintended selections, and prior intent models lacked comprehensive cross-task evaluation. GazeIntent uses temporal intent predictions to adapt dwell thresholds; the model reached F1 = 0.94, and returning users preferred personalized models. The approach was evaluated across tasks and users, though the study’s participant pool limits generalizability.
Problem
Method
GazeIntent uses gaze-based temporal intent predictions to scale dwell thresholds, evaluated with new and returning users across additional tasks.
Source: §1 · p. 2, §4.3 · p. 10Results
F1 = 0.94; returning users preferred personalized GazeIntent models, and the system was evaluated on tasks with selection frequencies differing from training.
Source: §2 · p. 4, §5 · p. 13Takeaways & Limitations
GazeIntent’s dwell-scaling architecture can accommodate intent models that output values between zero and one, and its thresholds can be adapted to user preferences and practice effects.
Source: §5.1 · p. 14, §6 · p. 15The studies involved 25 right-handed participants aged 20–35, limiting generalizability to larger and more diverse user groups.
Source: §5.1 · p. 13
Abstract
from arXiv · showhide
The use of ML models to predict a user's cognitive state from behavioral data has been studied for various applications which includes predicting the intent to perform selections in VR. We developed a novel technique that uses gaze-based intent models to adapt dwell-time thresholds to aid gaze-only selection. A dataset of users performing selection in arithmetic tasks was used to develop intent prediction models (F1 = 0.94). We developed GazeIntent to adapt selection dwell times based on intent model outputs and conducted an end-user study with returning and new users performing additional tasks with varied selection frequencies. Personalized models for returning users effectively accounted for prior experience and were preferred by 63% of users. Our work provides the field with methods to adapt dwell-based selection to users, account for experience over time, and consider tasks that vary by selection frequency
1 INTRODUCTION
Gaze-only selection avoids reliance on hand gestures, but must distinguish intended selections from glances. GazeIntent uses temporal intent predictions to adapt dwell-time thresholds for gaze-only selection.
1 INTRODUCTION
- Gaze-only selection is a hands-free alternative to gesture interaction, whose range, fatigue, and sensing accuracy can be constrained by the environment.The work focuses on eye movements alone for selection in virtual environments.Source: §1 · p. 1
- Dwell thresholds balance selection speed against accidental selections, while prior intent models have not been comprehensively evaluated across training and deployment tasks.Earlier gaze-only interaction risked selecting objects users meant only to glance at.Source: §1 · p. 2
- GazeIntent scales dwell time using temporal intent predictions to mitigate spurious positive predictions and enable faster selection than a fixed dwell time.The evaluation trained on controller-selection data and deployed the approach in gaze-only tasks.Source: §1 · p. 2
- The paper reports an intent model with F1 = 0.94 and says returning users preferred personalized models over a general model.The introduction also reports preference over state-of-the-art intent-based dwell selection and accommodation of new selection tasks.Source: §1 · p. 2
2 RELATED WORKS AND MOTIVATION
Prior work explored gaze-based prediction and dwell-threshold adaptation, but real-time deployment across users, tasks, and selection frequencies remains insufficiently evaluated. The paper motivates testing these factors together.
2 RELATED WORKS AND MOTIVATION
- OptiDwell optimizes dwell thresholds per user and interface over five sessions, using logged false selections to track accuracy.The approach accounts for changes in user behavior over time.Source: §2 · pp. 2–3
- Gaze models have predicted interaction events and cognitive states, but a model for navigation lostness showed reduced performance in a different virtual environment.The related studies motivate evaluating interaction intent across cognitive states, tasks, and environments.Source: §2 · p. 3
- Earlier selection-intent models were not deployed in real time, and the pantry model reported AUC-PR=0.12, with user-varying top features suggesting personalization may help.Bednarik et al. evaluated offline; David-John et al. identified differences in top-ranked features between users.Source: §2 · p. 3
- Isomoto et al. paired intent classification with a static dwell threshold, but users had to wait through the threshold before the model could confirm selection.That sequence creates a lower bound on interaction speed, particularly relevant for tasks with frequent selections.Source: §2 · pp. 3–4
- User practice can increase selection speed, while ordinary training-test splits do not preserve the temporal order of that changing behavior.The authors used forward-chaining training and validation to model behavior as users became familiar with a task.Source: §2 · p. 4
- The paper identifies three evaluation gaps: different cognitive states, new users, and task selection frequencies that differ from training.Existing work often lacked real-time evaluation; this study trains on 1–9 selections per minute and deploys on tasks with 20–40 selections per minute.Source: §2 · p. 4
3 TRAINING DATASET & MODEL DEVELOPMENT
The authors collected VR gaze and selection data during arithmetic tasks, processed it into temporal features, and developed an intent model for deployment. Model testing found strong performance, greater predictive influence from continuous features, and no significant performance difference under the tested cognitive-load conditions.
3.1 Data Collection
- Eighteen participants completed arithmetic divisibility tasks with varying divisors and difficulty to provide the intent-model training data.Participants selected answers on a VR grid using a hand controller; the study also included color memorization in later trials.Source: §3.1 · pp. 4–5
3.2 Data Pipeline
- The data pipeline processed gaze signals, detected fixation and saccade events, extracted continuous and event-based features, and labeled windows near controller-trigger selections as positive.Training used gaze data from active divisibility tests in sliding windows; the feature set included displacement, velocity, acceleration, and event-derived measures.Source: §3.2 · pp. 5–6
3.3 Model Training and Selection
- Model development used temporal splits, walk-forward validation, and Shapley analysis to select model configurations and features.The first stage trained 288 configurations; later stages refined the top models and feature sets.Source: §3.3 · p. 7
3.4 Results
- Most trained models reached F1 ≥0.9, compared with F1 = 0.8 for the best prior gaze-based selection intent model.The optimal model used 17 samples per window, overlap 2, an order-1 length-11 Savitzky–Golay filter, and a one-second label interval.Source: §3.4 · pp. 7–8
- Continuous features had mean influence 1.644 ± 0.89, versus 0.05 ± 0.009 for event-based features.The feature-influence table summarizes the continuous features as primarily affecting model predictions.Source: §3.4 · p. 7, Table 1 · p. 8
- Models using only continuous, continuous-plus-event-booleans, or top-30 features showed no practical performance difference from the complete feature set; event-only models performed worst.The continuous-plus-event-booleans set marginally outperformed the other feature sets.Source: §3.4 · p. 8
- Tests found no significant model-performance difference between subsets with and without induced cognitive load (p > 0.05).Load was induced through color memorization and arithmetic complexity.Source: §3.4 · p. 8
4 REAL-TIME MODEL DEPLOYMENT
GazeIntent maps recent intent predictions to a dwell-time scaling factor and applies it to a static threshold. The end-user evaluation compared static dwell, static dwell plus intent, and general and personalized GazeIntent across several gaze-only tasks.
4.1 Dwell Time Scaling
- The scaling factor uses the last four intent predictions and task- or preference-set weights to reduce the static dwell threshold.The factor ranges from 0 to 1: 0 leaves the threshold unchanged, while 1 reduces it to zero for immediate selection.Source: §4.1 · p. 9
4.2 Implementation Details:
- The real-time implementation maintained gaze samples, feature frames, and the last four predictions, updating the model input as new samples arrived.It ran in Unity with Barracuda inference and used 17×11 feature frames.Source: §4.2 · p. 9
4.3 End User Study
- Sixteen participants, including eight returning from the initial study, tested four gaze-only tasks and four selection methods.Tasks included circle selection, arithmetic with same or different stimuli, and a sliding puzzle; GI-P was available only to returning participants.Source: §4.3 · pp. 9–10
4.3.3 Pilot Study.
- The pilot selected static dwell thresholds of 0.3 seconds for Circle, 1.5 seconds for Arithmetic, and 1.2 seconds for Sliding Puzzle.Participants preferred the third tested threshold configuration.Source: §4.3.3 · p. 10
4.3.4 Metrics.
- The study measured selection counts for Circle and Sliding Puzzle, completion time for Arithmetic, and post-task experience and method preferences.Quantitative analysis modeled task, method, and returning-user interactions; no significant effects or interactions were found for completion time.Source: §4.3.4 · pp. 10–11, §4.3.5 · p. 11
4.3.5 Results.
- Returning users made significantly more selections with GI-G (μ=192 selections, p<0.001) and GI-P (μ=181 selections, p<0.001) than with S+I (μ=46 selections) or Baseline (μ=69 selections) in sliding-puzzle tasks.No other pairwise relationships were found.Source: Methods · p. 11
Methods:
GazeIntent was perceived as the fastest selection method overall, while method preferences varied by user group and task. Selection counts also differed between tasks and, for the sliding puzzle, between returning and new users.
Methods:
- Selection counts differed significantly between tasks (ps < 0.005), reflecting the speed-focused Fitt’s Law-style task versus the puzzle’s move-planning requirement.Source: Methods · p. 11
- Sliding-puzzle selections were higher for returning users (μ=88) than new users (μ=52); no other significant relationships between user groups were found.The difference was significant at ps < 0.005.Source: Methods · p. 11
- Subjective false-positive and selection-accuracy measures showed no significant three-way interaction among task, method, and user group; perceived speed also had no significant three-way interaction.The method-by-returning-user interaction for perceived speed approached significance (p = 0.055).Source: Methods · p. 11
- GazeIntent was perceived as fastest overall; returning users ranked GI-G = GI-P > Baseline > S+I, while new users ranked GI-G > Baseline > S + I.The figure shows subjective speed perceptions across tasks; *** marks significantly different groups at p < .001.Source: Figure 3 · p. 12, Methods · p. 12
- Returning users most preferred GI-P for the Fitt’s Law-style task (five of eight) and Arithmetic task (four of eight), but Static Dwell for the sliding puzzle (seven of eight).Across tasks, returning users ranked S + I least preferred.Source: Methods · p. 12
- Five of eight new users preferred GI-G for Arithmetic; Static Dwell was preferred for the other two tasks, and S+I was least preferred across tasks.GI-G was the second most preferred method in those other two tasks.Source: Methods · p. 12
5 DISCUSSION
GazeIntent performed well on unseen tasks, while discussion results also indicate benefits from personalization and additional user data over time. The authors note limits in the study’s scope and identify directions for broader evaluation and adaptation.
5 DISCUSSION
- The N=16 arithmetic-task study found LSTM intent models were not impacted by cognitive-load changes; continuous features outperformed event-based features, and eleven gaze features supported real-time use with F1 = 0.94.Source: §5 · pp. 12–13
- Returning users preferred personalized models for Fitts’ Law-style and Arithmetic tasks, while Static was preferred for Sliding Puzzle; new users preferred GI-G only for Arithmetic.Four of the five returning users who preferred GazeIntent favored their personalized model.Source: Figure 4 · p. 13, §5 · p. 13
- On Circle Selection, GI-G selection counts were 56% and 273% higher than baseline and S+I, respectively; on Sliding Puzzle, they were 94% and 177% higher, respectively.Both tasks differed from the Arithmetic training task and produced more frequent selections.Source: §5 · p. 13
- As models trained on successive temporal splits of user data, F1 increased and performance variability decreased; 62% of returning users preferred GazeIntent on similar Division tasks.The temporal splits used the first 20%, 40%, 60%, 80%, and 100% of training data.Source: §5 · p. 13, Figure 5 · p. 14
5.1 Limitations
- The 25 participants were right-handed and aged 20–35, limiting generalizability to larger and more diverse user groups.The data collection and end-user study shared this participant scope.Source: §5.1 · p. 13
- One-second labels may miss individual decision-to-hand-movement lead times; cognitive load was varied narrowly, and fusing continuous and discrete features remained the primary model limitation.Source: §5.1 · p. 14
5.2 Future Work
- Personalized models had marginally lower offline F1, although most returning users preferred them; the pilot-set thresholds could be adapted for practice effects or task changes.Source: §5.2 · p. 14
- Future work proposes free-viewing and mixed-reality evaluation, threshold optimization, continual learning, task-switching tests, and evaluation with audio or tactile stimuli.Source: §5.2 · pp. 14–15
6 CONCLUSION
GazeIntent uses temporal intent predictions to adapt dwell-time thresholds for gaze selection. The authors report effective interaction across tasks without retraining the underlying model, with thresholds potentially adaptable to user preferences and practice effects.
6 CONCLUSION
- GazeIntent applies thresholds to temporal model predictions to adapt dwell time, supporting interaction across tasks without retraining the underlying intent model.Source: §6 · p. 15