PaperlayerView exact revision on arXiv

Source-linked AI summary

GazePointAR: A Context-Aware Multimodal Voice Assistant for Pronoun Disambiguation in Wearable Augmented Reality

Jaewook Lee, Jun Wang, Elizabeth Brown, Liam Chu, Sebastian S. Rodriguez, Jon E. Froehlich

arXiv:2404.08213v1cs.HC

TL;DR

Voice assistants often lack the spatiotemporal context needed to resolve pronouns. GazePointAR combines gaze, pointing, and conversation history in a wearable-AR assistant; participants appreciated its naturalness, while identifying limits including unsupported multiple or past referents.

  • Problem

    Current voice assistants lack users’ spatiotemporal context and do not well support pronoun disambiguation, despite pronouns’ prominence in human speech.

    Source: §1 · p. 2

  • Method

    GazePointAR uses eye gaze, pointing gestures, and conversation history with real-time scene analysis to replace spoken pronouns in queries.

    Source: §1 · p. 2

  • Results

    Participants appreciated GazePointAR’s naturalness and human-likeness, and favored pronouns for referents with difficult, long, or unknown names, though pronoun use sometimes felt less natural.

    Source: §4.5 · p. 9

  • Takeaways & Limitations

    Future context-aware voice assistants should support natural gaze and gesture input alongside speech so users can ask spontaneous queries.

    Source: §8 · p. 14

    GazePointAR did not collect and store gaze over time and could not support queries with multiple or past referents.

    Source: §8 · p. 14

Abstract

from arXiv · show

Voice assistants (VAs) like Siri and Alexa are transforming human-computer interaction; however, they lack awareness of users' spatiotemporal context, resulting in limited performance and unnatural dialogue. We introduce GazePointAR, a fully-functional context-aware VA for wearable augmented reality that leverages eye gaze, pointing gestures, and conversation history to disambiguate speech queries. With GazePointAR, users can ask "what's over there?" or "how do I solve this math problem?" simply by looking and/or pointing. We evaluated GazePointAR in a three-part lab study (N=12): (1) comparing GazePointAR to two commercial systems; (2) examining GazePointAR's pronoun disambiguation across three tasks; (3) and an open-ended phase where participants could suggest and try their own context-sensitive queries. Participants appreciated the naturalness and human-like nature of pronoun-driven queries, although sometimes pronoun use was counter-intuitive. We then iterated on GazePointAR and conducted a first-person diary study examining how GazePointAR performs in-the-wild. We conclude by enumerating limitations and design considerations for future context-aware VAs.

1 INTRODUCTION

Current voice assistants lack spatiotemporal context and poorly support pronouns. GazePointAR uses gaze, pointing, and conversation history to resolve them, and the authors study its use in lab and real-world settings.

1 INTRODUCTION

  • Although pronouns are prominent in human speech, current voice assistants lack spatiotemporal context and may produce unnatural dialogue or fail to answer queries such as “What is that?”Prior context-aware prototypes explored gaze, pointing, or touch, but used Wizard-of-Oz setups, tightly controlled studies, one added modality, or smartphones.Source: §1 · p. 2
  • GazePointAR replaces spoken pronouns with phrases generated from real-time scene analysis informed by gaze and pointing, then sends the modified query to GPT-3.For example, looking at a juice bottle can yield a phrase describing its type, brand, flavor, and calories.Source: §1 · p. 2
  • The authors evaluated GazePointAR in a three-part lab study with 12 participants, comparing it with Google Voice Assistant and Google Lens and exploring usability and user-generated queries.A follow-up diary study involved five days and 48 queries by the first author.Source: §1 · p. 2
  • In open-ended lab trials, participants mainly used gaze for diverse queries and praised its ability to incorporate gaze, while reporting limitations including single-shot gaze and multiple-pronoun handling.In the five-day diary study, the first author found GazePointAR more natural and robust for complex-to-describe objects than a traditional VA, but encountered recognition and privacy concerns.Source: §1 · p. 2
  • The paper contributes a functional wearable-AR voice assistant, findings from two user studies, and design considerations for future systems supporting spontaneous natural queries.The stated design concerns include continuous gaze tracking and AI explainability.Source: §1 · p. 2

2 RELATED WORK

The related work situates GazePointAR within research on pronouns, multimodal interaction, voice assistants, augmented reality, and wearable AR.

2 RELATED WORK

  • The section reviews pronoun use in speech before discussing multimodal interaction, voice assistants, augmented reality, and wearable AR.It narrows the broad multimodal literature to voice assistants and augmented reality.Source: §2 · p. 2

2.1 Pronoun Usage in Speech

  • Pronouns are frequent in speech, and people use gaze, pointing, and conversational context to resolve their referents.Prior work also links pronouns with naturalness and expressivity in speech.Source: §2.1 · pp. 2–3

2.2 Multimodal Interaction

  • Multimodal interaction research combines modalities such as gaze, gestures, and speech, including to address limits of gaze-alone input.The paper focuses this background on voice assistants and augmented reality.Source: §2.2 · p. 3

2.2.1 Multimodal Interaction with Voice Assistants.

  • Voice-assistant research has combined speech with gestures, gaze, pens, and other inputs, while AR voice-assistant prototypes have explored gaze, touch, or gestures.The paper positions its work as a fully functional multimodal assistant supporting ambiguous speech.Source: §2.2.1 · p. 3

2.2.2 Multimodal Interaction in Augmented Reality.

  • AR pronoun-disambiguation systems have used gaze, touch, or pointing, each with tradeoffs in speed, accuracy, and hands-free use.GazePointAR combines gaze with pointing gestures and evaluates the combination in a wearable-AR voice assistant.Source: §2.2.2 · p. 3

2.2.3 Other Uses of Gaze, Pointing, and Speech in Wearable AR.

  • Wearable-AR research has used gaze for adaptive information displays, machine learning for hand gestures, and glasses for captioning, translation, and speech augmentation.The authors describe GazePointAR as influenced by this work on wearable-AR interaction and context.Source: §2.2.3 · p. 3

3 GAZEPOINTAR PROTOTYPE 1

GazePointAR is a wearable-AR voice assistant designed to resolve spoken pronouns using gaze, pointing, and conversation history. Its prototype applies a pronoun taxonomy and a computer-vision pipeline to transform referents into query phrases.

3 GAZEPOINTAR PROTOTYPE 1

  • GazePointAR is a fully functional context-aware assistant for AR glasses that uses eye gaze, pointing gestures, and conversation history to support pronoun disambiguation.The prototype was implemented for Microsoft HoloLens 2.Source: §3 · p. 3, §3.2 · p. 4

3.1 Taxonomy of Pronoun Use and Resolution

  • The taxonomy covers 13 frequently spoken pronouns across nominal and adverbial demonstratives and third-person pronouns, with referents resolved through gaze, pointing, or conversation history.Anaphoric and pleonastic “it” require conversational history, while demonstratives can be resolved by distance and gesture or gaze.Source: §3.1 · pp. 3–4

3.2 System Implementation

  • When a query contains a taxonomy pronoun, GazePointAR captures the user’s field of view and sends the image to object, text, and celebrity-recognition models.OCR detections are linked to parent object or face boxes when their overlap meets an empirically tuned 70% threshold.Source: §3.2 · p. 4
  • For singular pronouns, gaze or pointing within a parent box yields a phrase combining that object with its OCR text; otherwise, the system selects nearby text results.For plural pronouns, the system expands input coordinates into half-image-sized boxes and applies a 70% overlap test.Source: §3.2 · pp. 4–5
  • The system combines the spoken query, its generated referent phrase, and up to five recent query-answer pairs before sending the modified query to GPT-3.Gaze and pointing coordinates are projected to image pixels; the response is displayed and read aloud.Source: §3.2 · pp. 4–5

4 STUDY 1: THREE-PART LAB EVALUATION OF GAZEPOINTAR

The three-part lab study examined initial use of GazePointAR, comparison with commercial voice systems, and the context-sensitive queries participants wanted to try. Twelve participants completed a counterbalanced within-subjects evaluation.

4 STUDY 1: THREE-PART LAB EVALUATION OF GAZEPOINTAR

  • The within-subjects study with 12 participants asked how users perceive and use a context-aware assistant, how it compares with traditional VAs, and what queries users want to perform.The study aimed to observe reactions to a functional AR query system rather than quantitatively measure performance.Source: §4 · p. 5

4.1 Participants

  • All 12 participants had prior experience with voice assistants and AI chat systems, while nine had not used AR headsets or glasses.Participants were recruited through mailing lists, social media, and snowball sampling; people with specified visual or auditory disabilities, seizure histories, or limited English fluency were screened out.Source: §4.1 · p. 5

4.2 Procedure

  • After tutorials, participants compared the systems on a recipe task, completed three context-sensitive GazePointAR tasks, then brainstormed and attempted their own queries after viewing scenario videos.The recipe task used each system’s input modality; participants also completed usability questionnaires and interviews, and ranked the systems on perceived intelligence, helpfulness, naturalness, and preference.Source: §4.2 · p. 6

4.3 Data and Analysis

The analysis combined interviews, observations, and questionnaires, using thematic coding for qualitative data and nonparametric tests for Part 1 comparisons.

4.3 Data and Analysis

  • Three sources—interview transcripts, study-session observations, and post-task questionnaires—formed the analyzed dataset.Source: §4.3 · p. 6
  • The team iterated on a 34-code codebook, then collaboratively discussed themes after the first author coded participant quotes.Source: §4.3 · p. 6
  • Part 1 comparisons used Friedman and Holm-corrected Wilcoxon signed-rank tests; SUS responses were rescaled from 0–40 to 0–100.Source: §4.3 · p. 6

4.4 Findings

Findings compare query behavior and reactions across voice assistants, then report how participants used GazePointAR for context-sensitive and self-generated queries, including successes and limitations.

4.4.1 Part 1: Comparing VAs.

  • 1.7 (Google VA) and 1.8 (GazePointAR) mean overall-preference ranks beat Google Lens’s 2.6; Google VA scored 80.0 on SUS versus GazePointAR’s 62.1.Task completion averaged 26.3 seconds with Google VA, 37.4 with GazePointAR, and 60.7 with Google Lens.Source: §4.4.1 · pp. 6, 8
  • Google VA’s familiar, explorable results were valued but demanded specificity and pronunciation; Lens drew criticism for extra steps and poor matches, while GazePointAR’s single answer limited exploration despite simpler, human-like interaction.Source: §4.4.1 · pp. 6–7
  • Google Lens averaged 1.3 words per query, GazePointAR 6.3, and Google VA 8.4; all GazePointAR users said “this” with gaze, while Lens users photographed the jar.Source: §4.4.1 · pp. 7–8
  • Participants ranked GazePointAR highest for perceived intelligence (mean rank 1.5), Google VA highest for helpfulness (1.3), and GazePointAR and Google VA highly for naturalness (1.6 and 1.7, respectively).Seven participants felt GazePointAR was most natural. P12 distinguished speaking naturally to another person from speaking naturally to a machine, favoring Google VA for the latter.Source: §4.4.1 · p. 8

4.4.2 Part 2: Context-sensitive Queries.

  • Across Part 2, participants favored gaze+speech over pointing; “this” felt natural for math, while two-item comparisons exposed one-pronoun support and uncertainty about indicating multiple referents.One participant also used conversation history to follow a price query with a cost-difference question.Source: §4.4.2 · pp. 8–9
  • For celebrity queries, participants used “this,” “her,” or “she,” and seven specifically said pronouns were helpful when asking about an unfamiliar person in a photo.Source: §4.4.2 · p. 9

4.4.3 Part 3: Design Probe and Co-design.

  • Participants saw broad potential for GazePointAR, especially when a visual referent was unknown or difficult to vocalize, and showed interest in hazardous-object and recycling scenarios.Source: §4.4.3 · p. 9
  • Across 32 self-generated queries, all used gaze and 13 received satisfactory answers; failures included missing recognition or online information, and queries with multiple or past pronouns.Most queries sought object or person information (23/32); others included translation, comparison, and action checks.Source: §4.4.3 · p. 9

4.5 Study 1 Summary

Participants valued GazePointAR’s natural, human-like interaction and mostly relied on gaze, though pronoun use could feel counter-intuitive. They also identified needs for better gaze capture, explanations, and answer options.

4.5 Study 1 Summary

  • Participants appreciated GazePointAR’s simplicity, naturalness, and human-likeness, mostly using gaze for hands-free interaction and occasionally pointing or conversation history.They especially preferred pronouns when referents had difficult-to-pronounce, long, or unknown names.Source: §4.5 · p. 9
  • Pronoun-driven queries could feel less natural than naming a referent directly.For example, participants contrasted asking “What can I make with this?” with naming Rao’s Marinara sauce.Source: §4.5 · p. 9
  • Participants wanted support for multiple pronouns, more answer options and explanations, and more robust machine-learning models.They also suggested capturing gaze over time, communicating which images and contextual inputs informed answers, and displaying explorable search results.Source: §4.5 · pp. 9–10
  • Explicit gazing could tire users, while Figure 7 contrasts recreated answerable queries with examples that returned unsatisfactory responses.Source: §4.5 · p. 10, Figure 7 · p. 10

5 GAZEPOINTAR PROTOTYPE 2

GazePointAR prototype 2 replaced its object recognizer, redesigned phrase generation with prompt engineering, and updated its language model. The object-recognition change increased runtime, while the redesigned generator enabled multiple-pronoun support.

5 GAZEPOINTAR PROTOTYPE 2

  • The second prototype incorporated three changes before a five-day first-person diary study of version 2 in daily life.The changes were the object-recognizer replacement, phrase-generator redesign, and API update.Source: §5 · p. 10
  • The prototype replaced Google Cloud Vision’s Object Localization with YOLOv8 trained on MS COCO.The earlier model was chosen for rapid prototyping but could classify objects it could not precisely identify as “packaged goods.”Source: §5 · p. 10
  • The reported setup increased ML-services runtime to 3.75 ± 0.31 seconds (+11.28%) and overall runtime to 7.94 ± 0.38 seconds (+5.73%), relative to its prior runtime.The initial model categorized objects as “packaged goods” when it could not precisely identify them, confusing GPT-3 and participants; this iteration used YOLOv8 trained on MS COCO [51].Source: §5 · p. 10
  • The revised prompt-engineered phrase generator preserved the original query and enabled GazePointAR v2 to support multiple pronouns.It combined object, celebrity-recognition, and OCR results as parent and child layers, and prompted GPT to briefly explain answers.Source: §5 · p. 10
  • The prototype updated its chat-completions API to use GPT-3.5-turbo.The paper describes it as trained on more up-to-date data and more efficient than GPT-3.Source: §5 · p. 10

6 STUDY 2: GAZEPOINTAR DEPLOYMENT

A first-person, five-day diary study examined GazePointAR v2 as an always-available multimodal wearable assistant in everyday settings. It found natural, companion-like interactions alongside limited answer success and practical constraints involving scene recognition, gaze history, public use, and fatigue.

6.1 Procedure

  • Across five days, the lead researcher used GazePointAR v2 for four hours daily in indoor and outdoor settings, documenting interactions with video and written notes.The HoloLens required an internet connection, supplied through mobile hotspots or public Wi-Fi.Source: §6.1 · p. 11

6.2 Findings

  • 20 of 48 queries received satisfactory answers; prompt engineering appeared to prioritize gaze targets, consider similar objects, and produce concise answers with justifications.Even when unable to answer, the system often appeared to interpret the user's input and intentions accurately.Source: §6.2 · p. 11
  • GazePointAR felt companion-like and handled a clothing-brand recommendation, but public speaking drew self-consciousness and extended gaze caused eye fatigue.The bulky headset attracted attention, particularly in quiet public places.Source: §6.2 · p. 11
  • 21 queries sought referent information, 11 recommendations, 9 directions, 4 personal information, and 3 past actions.The researcher relied on gaze as hands-free and avoided pointing in public; first- and second-person pronouns and full-sentence queries were common, with 31 queries containing more than one pronoun.Source: §6.2 · pp. 11–12
  • GazePointAR answered menu and clothing queries, but failures included unrecognized equipment, lost-key queries needing stored history, and comparisons involving multiple pronouns.The reported failure causes included missing object categories and gaze-capture limits; the system did not retain information over time.Source: §6.2 · p. 12

6.3 Study 2 Summary

  • The study identified boundaries in real-world use: time-dependent and shifting-referent queries need gaze history, while broader object-category coverage was suggested for improvement.The researcher also reported fatigue from extended dwelling and avoided pointing in public because it felt awkward and physically effortful.Source: §6.3 · p. 12

7 DISCUSSION

GazePointAR’s use of gaze, gestures, conversation history, and an LLM was associated with more natural interaction and human-like assistant qualities. The discussion identifies design challenges involving gaze over time, answer choice, explainability, query flexibility, object recognition, and the scope of the in-the-wild study.

7 DISCUSSION

  • Across both studies, users valued more natural query formation, always-available interaction, and GazePointAR’s human-like assistant qualities.The discussion also includes diary-study examples of queries answered accurately and inaccurately.Source: §7 · p. 12, Figure 9 · p. 13, Figure 10 · p. 13

Capturing gaze information over time

  • Capturing gaze only once after a query can leave some queries unanswerable; continuous tracking could support gaze shifts and multiple referents over time.The discussion notes privacy concerns and the challenge of representing temporal gaze data for an LLM.Source: §7 · p. 12

Ensuring user autonomy in choosing an answer

  • GazePointAR selects and reads one answer, while participants expressed differing preferences for exploring options and receiving concise answers.The discussion proposes an optional Google-like interface after a concise verbal answer.Source: §7 · p. 12

7 DISCUSSION

Enhancing explainability

  • The authors’ initial explainability step prompted an LLM to explain responses; future systems could also show captured images, inputs, ML results, and predicted referents.The discussion suggests a concise explanation first, with more information available on request.Source: §7 · p. 13

Supporting instinctive queries

  • Pronouns can facilitate interaction but may complicate query formation, so GazePointAR v2 uses prompts to process original queries, including ambiguous ones.Participants sometimes preferred explicit queries over pronouns.Source: §7 · p. 13

Enhancing machine learning capabilities

  • Object-recognition failures were more apparent in the diary study, where many real-world objects were outside YOLOv8’s categories.The discussion identifies personal and online information as potentially useful for recommendation queries, alongside privacy risks.Source: §7 · p. 13

Designing a more robust study

  • The first-person diary study involved only the lead researcher, so its findings may be subjective; future work should include more in-the-wild participants.Source: §7 · p. 13

8 CONCLUSION

The conclusion presents GazePointAR as a context-aware multimodal voice assistant for wearable AR and summarizes users’ appreciation of its naturalness and human-likeness. It also identifies system limitations and calls for future assistants to support spontaneous speech, gaze, and gesture input.

8 CONCLUSION

  • GazePointAR is a context-aware multimodal voice assistant for wearable AR that can answer pronoun-driven ambiguous queries.Source: §8 · p. 13
  • Users appreciated its naturalness, human-likeness, and ability to refer to objects that are difficult to pronounce or describe.Source: §8 · p. 13
  • The conclusion lists limitations in gaze collection, autonomy, explainability, multiple or past referents, and object-recognition categories.Source: §8 · p. 14
  • Future context-aware assistants should support natural gaze and gesture alongside speech, allowing users to ask queries spontaneously.Source: §8 · p. 14

A DEMOGRAPHIC QUESTIONNAIRE

The appendix introduces a demographic questionnaire covering participant characteristics and familiarity with voice assistants, AR devices, and AI chat systems.

A DEMOGRAPHIC QUESTIONNAIRE

B QUERIES

The appendix lists tables of user-spoken queries from three lab-study parts and the first-person diary study.

B QUERIES

  • Table 1 is titled as a collection of user-spoken queries from Part 1 of the study.Source: Table 1 · p. 17
  • Table 2 is titled as a collection of user-spoken queries from Part 2 of the study.Source: Table 2 · p. 18
  • Table 3 is titled as a collection of user-spoken queries from Part 3 of the study.Source: Table 3 · p. 19
  • Table 4 is titled as a collection of user-spoken queries from the first-person diary study.Source: Table 4 · p. 20
Loading 2404.08213v1…