PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Activation: Gaze Invocation with Visible Status for an Embodied AR Assistant in Co-Located Collaboration

Chenrui Ma, Yoshio Ishiguro, Qing Zhang

arXiv:2610.09284v1cs.HC

TL;DR

In co-located AR, nearby speech may be directed either to an assistant or to another person, and wake-word sensitivity trades missed commands against accidental activations. The paper implements gaze-based invocation with an embodied assistant, visible status, and lifecycle turn management. In the evaluation, 23 utterances began after release, highlighting that users may miss the assistant’s status once attention returns to their work.

  • Problem

    In co-located AR, the system must distinguish speech to an assistant from speech to a nearby person, while wake-word sensitivity trades missed commands against accidental activations.

    Source: §1 · p. 1

  • Method

    The embodied assistant displays activation and listening status as sustained gaze opens a local channel, speech and playback retain it, and inactivity releases it.

    Source: §1 · p. 2, §3.2 · p. 3

  • Results

    23 of 496 assistant-directed utterance attempts began after the assistant had returned to Idle, while 6 began before gaze acquisition completed.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Embodied visible status can clarify entry, but release needs cues that remain perceptible after users look back to their work.

    Source: §8 · p. 10

    The collaborative tabletop task, HoloLens 2 implementation, and one conversational service bound generalization; the single-session study also cannot show learning effects or changes through repeated use.

    Source: §7.4 · p. 9

Abstract

from arXiv · show

Wake words face an inherent trade-off: higher sensitivity reduces missed commands but increases accidental activations. In co-located augmented reality (AR), the system must also determine whether the user is speaking to the assistant or to a nearby person. We introduce a new perspective on this problem: combining gaze-based address with an embodied assistant, visible listening status, and turn management across activation, continued interaction, and release. In our implementation, users look at an assistant anchored in the scene and see activation progress and listening status on its body. Sustained gaze opens a local interaction channel; speech and playback keep it open when attention returns to the task; and inactivity closes it. We evaluated this design with 25 participants. After selecting the assistant's placement and dwell duration, each participant worked with a partner to plan a trip while using the assistant. The study recorded 500 interaction outcomes, including 496 assistant-directed utterance attempts. Gaze acquisition completed before speech for 490 of these attempts (98.8\%): 467 began while the channel remained open, whereas 23 began after it had been released. The other 6 attempts began before acquisition completed. Among 102 reviewed attempts in which gaze left after acquisition but before speech, the configured retention policy kept the channel open for 79 and released it before 23. The results show that an embodied target with visible status can support clear entry into an assistant interaction while revealing a different problem at release: after users look back to their work, they may not see that the assistant has stopped listening. We contribute the implemented gaze-invocation design and design implications for communicating assistant state after visual attention moves elsewhere.

1 INTRODUCTION

The paper addresses ambiguous speech in co-located AR by combining gaze-based address with an embodied assistant whose visible status supports interaction beyond initial activation. Its evaluation examines the full lifecycle, including cases where users spoke after the channel had been released.

1 INTRODUCTION

  • Wake-word sensitivity trades fewer missed commands for more accidental triggers, while co-located AR also requires distinguishing speech to the assistant from speech to a nearby person.Existing work addresses parts of this ambiguity, but does not jointly examine continued availability, interaction ending, and communicating release during collaboration.Source: §1 · p. 1
  • The design separates interaction into acquisition by sustained gaze, retention through speech or assistant playback, and release after inactivity.Participant-selected dwell duration balances responsiveness against incidental activation, and visible opening status is easier to see than closing status after attention shifts away.Source: §1 · p. 2
  • 23 of 496 assistant-directed utterance attempts began after the assistant had returned to Idle, while 6 began before gaze acquisition completed.First-person video, a visible gaze cursor, and assistant status cues supported reconstruction of these transitions.Source: §1 · p. 2
  • The paper contributes an embodied gaze-invocation design with participant-specific placement and dwell calibration, and reports lifecycle-level findings that activation-only evaluations overlook.The assistant serves as both gaze target and status display; the implementation was deployed on HoloLens 2.Source: §1 · p. 2

2 RELATED WORK

Prior work uses gaze, embodiment, and multimodal cues to address activation and addressee ambiguity, but the paper distinguishes invocation from referential gaze and follows interaction through retention and release. It relates the design to conversational turn management while treating lifecycle status as a joint concern.

2.1 Wake-Word Invocation and Other Address Cues

  • Look and Talk and Look to Wake primarily examine gaze-triggered interaction; this work follows gaze-based assistant interaction through acquisition, retention, and release.Speak-to-shoulder detects addressees, GazePointAR [26] resolves spoken references, and Persistent Assistant connects repeated queries.Source: §2.1 · p. 2

2.2 Gaze and Embodiment as Spatial Address

  • Gaze serves both as a spatial cue for addressing an assistant and as a referential cue, while dwell trades faster activation against more incidental looks.The study uses a participant-selected, fixed dwell; an embodied assistant provides a stable target and a place to display status, which may no longer be visible after attention shifts.Source: §2.2 · p. 2

2.3 Conversational Coordination and Control of the Assistant’s Turn

  • Turn-taking research links gaze, speech, feedback, and timing; this design applies those elements by using speech and playback to retain an open channel and inactivity to close it.The assistant displays the controller’s current state.Source: §2.3 · pp. 2–3

3 GAZE INVOCATION WITH VISIBLE ASSISTANT STATUS

The design treats invocation as a lifecycle: sustained gaze opens a local channel, speech and playback retain it as attention shifts, and inactivity releases it. An embodied assistant combines the gaze target with visible activation, listening, and response status.

3.1 Design Requirements

  • The design requirements are a stable embodied target with adjustable dwell, retention without continuous gaze, and visible feedback for activation, listening, response, and release.Together, these requirements frame invocation as a lifecycle rather than a single trigger.Source: §3.1 · p. 3

3.2 Interaction Lifecycle

  • Acquisition opens the channel after sustained gaze; retention keeps it open as attention moves elsewhere; release returns the controller to Idle.The four states are Idle, Gaze arming, Eligible, and Release timing; only Eligible and Release timing have an open channel.Source: §3.2 · p. 3
  • Dwell filters shorter incidental looks, but its duration sets a responsiveness–accidental-activation trade-off, so the threshold is adjustable.Speech or playback postpones release after gaze departure; new activity restarts the post-activity wait, and returning gaze pauses the pending countdown.Source: §3.2 · p. 3
  • The channel closes only after the applicable countdown completes with gaze absent and no new activity, so the user may be looking elsewhere when release occurs.Speech and playback retain an already-open channel but cannot open one themselves.Source: §3.2 · p. 3

3.3 Embodied Target and Visible Status

  • The assistant is an adjustable, wearer-facing gaze target whose body displays dwell progress, listening-ready status, and response activity.Its approximately 20 cm miniature scale was intended to limit competition with the collaborator and tabletop materials; playback adds an animated ellipsis without adding a controller state.Source: §3.3 · p. 3
  • Status colocated with the gaze target is inspectable during acquisition, but may be out of view during retention and release.The evaluation therefore follows the interaction through release rather than treating successful activation as the endpoint.Source: §3.3 · p. 3

4 IMPLEMENTATION

The implementation keeps time-critical gaze decisions and visible state on HoloLens 2 while routing conversational processing to a remote service. A local controller gates media and applies the acquisition, retention, and release policy.

4.1 Runtime Deployment and Data Paths

  • The deployed prototype places gaze control and embodied feedback on the headset while using a remote conversational service for processing.The system was implemented as a Unity application for HoloLens 2; a persistent WebSocket carries audio and optional camera frames, and the controller decides locally whether to admit media.Source: §4 · p. 3, §4.1 · p. 4
  • A separate workstation companion path carries configuration and activation commands, while structured controller records return as telemetry; user media bypasses the workstation.This separation supports live configuration and logging without placing the workstation in the user-media path.Source: §4.1 · p. 4

4.2 Local Interaction Control and Media Gating

  • The local controller uses gaze intersection with the assistant’s collider to apply lifecycle transitions and control both the embodied cue and media gate.A closed gate suppresses outgoing media and discards incoming response audio; speech activity and playback postpone release, while microphone packets are withheld during playback, preventing user barge-in.Source: §4.2 · pp. 4–5
  • After a final speech callback, the earliest return to Idle with gaze absent and no new activity is approximately 1.5+Tfollow+Tleave seconds; ongoing playback extends this interval.The service’s speech-activity window lasts 1.5 seconds, after which the controller applies Tfollow and then Tleave.Source: §4.2 · p. 5

4.3 Runtime Configuration and Service Interface

  • Experimenters can change spatial, gaze, display, dwell, and leave-wait parameters at runtime without rebuilding or restarting the headset application.The communication layer presents stable session, media-gating, activity, and response-audio operations while keeping service-specific details internal.Source: §4.3 · p. 5

5 EXPERIMENTAL SETUP AND CALIBRATION

The study calibrated assistant placement and dwell for each participant before testing the configured gaze interaction. Placement, gaze-to-speech timing, and explicit dwell choices informed the setup used in later tasks.

5.1 Participants, Ethics, and Procedure

  • The formal cohort comprised 25 participants; a 7-person engineering pilot informed refinement, but its data were excluded from analyses and figures.All formal participants completed the tasks in the same order.Source: §5.1 · p. 5, Figure 5 · p. 6

5.2 Apparatus and Experimental Configuration

  • The HoloLens 2 prototype used a persistent gaze cursor distinct from assistant state cues, with fixed 1.0 s follow-up and leave waits for all sessions.The 1.5 s speech-activity window remained unchanged; camera transmission was disabled, but collaborator-speech leakage was not independently measured.Source: §5.2 · p. 5

5.3 Task 1: Assistant Placement

  • Task 1 let participants reposition the assistant to address it without obstructing collaborative work, using a fixed 0.8 s dwell and visible readiness cues.Audio interaction was disabled during placement familiarization.Source: §5.3 · p. 5

5.4 Task 2: Gaze-to-Speech Timing

  • Task 2 estimated speech onset from local microphone RMS history after service notification, retaining gaze-to-speech intervals only when they fell between 100 and 3000 ms.The retained dataset contained 178 measurements; analysis used the first 7 per participant to give each equal weight.Source: §5.4 · p. 5

5.3 Task 1: Assistant Placement

  • Most participants placed the assistant near the task materials and below eye level, with greater horizontal than distance variation.Final placement distance had a mean/median of 0.685/0.68 m; elevation had a mean/median of −20.3°/−19.9°.Source: §5.3 · p. 5, Figure 6 · p. 6

5.4 Task 2: Gaze-to-Speech Timing

  • Gaze-to-speech timing varied substantially across the 175 equal-participant-weight retained observations: mean 1155.8 ms, median 936 ms, and IQR 536.5–1800 ms.Rejected episodes were discarded before logging, so the measurements do not describe every gaze episode observed.Source: §5.4 · p. 6, Figure 7 · p. 7

5.5 Task 3: Dwell Selection

  • Final dwell choices ranged from 100–1800 ms (median 800 ms); 19 participants shortened the calculated starting dwell, 5 increased it, and 1 kept it.The starting value was calculated from Task 2 measurements, then participants adjusted it in 100 ms steps; it was not treated as an optimized threshold.Source: §5.5 · p. 7, Figure 8 · p. 7

6 COLLABORATIVE INTERACTION TEST AND EVENT ANAL-

The configured assistant was evaluated during collaborative travel planning, with event outcomes reconstructed from first-person video and runtime records. Most intended utterances began while the channel was ready, while reviewed gaze departures also revealed post-release starts.

6.1 Collaborative Planning Task

  • Task 4 recorded 20 chronological events per participant, yielding 496 assistant-directed utterance attempts and 4 other channel-open events.Participants planned a trip with a researcher acting as a natural conversational collaborator while attention and speech shifted among materials, collaborator, and assistant.Source: §6.1 · p. 7

6.2 Event Recording and Critical-Incident Coding

  • Two researchers reviewed each complete first-person recording and reached consensus on final event codes, using visible gaze and assistant cues to reconstruct state and addressee.Participants clarified ambiguous addressee intent when needed.Source: §6.2 · p. 7, Table 1 · p. 7
  • The analysis classified channel state at speech onset and reviewed gaze departures, but did not assess ASR content, response quality, task success, or isolated component effects.The evaluation examined outcomes within one integrated acquisition–retention–release policy rather than comparing gaze invocation with wake words.Source: §6.2 · p. 7

6.3 Outcomes Across Acquisition, Retention, and Release

  • 98.8% of intended attempts had gaze acquisition complete before speech; 94.2% began with the channel Ready, while 29 began with it closed.The 29 closed-channel starts comprised 23 post-release events and 6 premature-speech events; these outcomes do not measure ASR accuracy, response quality, or task success.Source: Figure 9 · p. 8, Table 2 · p. 8, §6.3 · p. 8
  • Among 102 reviewed attempts where gaze left after acquisition but before speech, retention kept the channel open for 79 (77.5%) and released it before 23 (22.5%).These percentages apply only to the reviewed gaze-departure sequences, not to all interactions.Source: §6.3 · p. 8
  • The four other channel-open outcomes were 1 accidental activation and 3 over-retention events, not estimates of false-activation or over-retention probabilities.The fixed 20-event sample did not enumerate all incidental looks or collaborator-directed utterance opportunities.Source: §6.3 · p. 8

6.4 Post-Study Interview and Willingness

  • Of 25 participants, 21 reported willingness to use gaze-based invocation, 4 were neutral, and none selected an unwilling option.Open-ended comments were descriptive context, not a systematically coded qualitative dataset; an early-stop report was not aligned to a specific runtime record.Source: §6.4 · pp. 8–9

7 DISCUSSION

The discussion finds that visible embodiment makes gaze acquisition legible, while retention accommodates attention shifts and release remains difficult to communicate after attention moves away. The findings are bounded by the study’s sampling, configuration, exposure, and participant and task scope.

7.1 Acquisition through an Embodied Assistant with Visible Status

  • 490 of 496 utterance attempts (98.8%) followed completed gaze acquisition, with participants seeing progress and readiness on the same embodied target.The anchored character served as addressee, interaction target, and status display; the measure does not establish response correctness or task success.Source: §7.1 · p. 9

7.2 Retention, Release, and Later Intended Speech

  • 79 of 102 reviewed cases (77.5%) remained open after gaze left, allowing participants to inspect materials or formulate questions without continuous fixation.The other 23 cases ended in release before speech: 3 before an initial question and 20 before a follow-up.Source: §7.2 · p. 9

7.3 Design Lessons for Gaze-Activated AR Assistants

  • Embodiment can colocate addressee and status, but once attention moves, release feedback should reach users through a peripheral visual, audio, or haptic channel.The discussion also recommends exposing and evaluating controller transitions, with adaptive retention as a future direction.Source: §7.3 · p. 9

7.4 Scope and Limitations

  • The evidence is limited to a fixed sample of 20 events per participant and one integrated configuration, without wake-word or component-ablation comparisons.Dwell estimates excluded episodes outside 100–3000 ms, and the gaze cursor may have influenced behavior. The single-session sample and educated, XR-experienced participants, plus one task, device, and service, constrain generalization; researchers reached consensus rather than reporting inter-rater reliability.Source: §7.4 · p. 9

8 CONCLUSION

The paper presents an embodied AR assistant that uses gaze to open an interaction channel and speech or playback to retain it as attention returns to the task. The collaborative evaluation found consistent entry within the tested configuration, while also identifying release communication as a challenge after users look elsewhere.

8 CONCLUSION

  • Gaze opens the channel through an embodied assistant with visible status, while speech and playback keep it open as attention returns to the task.The design used participant-selected placement and dwell.Source: §8 · p. 10
  • Participant-selected placement and dwell provided consistent entry into assistant interaction within the evaluated configuration.Source: §8 · p. 10
  • After activation, users looking back to task materials could miss that the assistant had stopped listening, making release cues perceptible beyond its body a lifecycle design need.The conclusion identifies embodied status as clarifying entry and retention as supporting natural visual-attention shifts.Source: §8 · p. 10
Loading 2610.09284v1…