PaperlayerView exact revision on arXiv

Source-linked AI summary

EgoVoice: Proactive Spoken Assistance from Egocentric Multimodal Streams

Heeseung Kim

arXiv:2610.12248v1cs.CLcs.CVcs.SD

TL;DR

EgoVoice addresses the lack of a single model for deciding when to speak and what to say from continuous egocentric streams. It introduces a training and evaluation framework, and both trained variants outperform baselines on onset F1 and content relevance.

  • Problem

    Existing systems do not combine deciding when to speak and what to say from continuous egocentric streams, despite the need for guidance when users cannot pause to ask.

    Source: §1 · p. 1

  • Method

    EgoVoice introduces a framework for training and evaluating proactive egocentric spoken assistants.

    Source: §1 · p. 1

  • Results

    Both EgoVoice-SFT and EgoVoice-DPO outperform all baselines on onset F1 and content relevance.

    Source: §6 · p. 9

  • Takeaways & Limitations

    EgoVoice is an initial step toward always-on spoken assistants that proactively guide users through real-world tasks.

    Source: §6 · p. 9

    Because the dataset covers a specific set of procedural assembly tasks, generalization to cooking, outdoor navigation, or social interaction is not guaranteed.

    Source: Limitations · p. 10

Abstract

from arXiv · show

Wearable augmented reality (AR) assistants are moving toward continuous real-world interaction, where they perceive the user's activity through first-person video and audio and provide timely spoken guidance without being explicitly asked. While proactive video assistants, spoken dialog systems, and egocentric task understanding have each advanced rapidly, existing systems do not address the joint problem of deciding when to speak and what to say from continuous first-person streams. We introduce EgoVoice, a framework for training and evaluating proactive egocentric spoken assistants. From HoloAssist video recordings of real human instructors, we construct clean audio streams through source separation and speech resynthesis, and convert each video session into a format where the model must decide at each moment whether to remain silent or provide spoken guidance. We fine-tune an omni-modal LLM with our data, and further improve its proactive intervention behavior with direct preference optimization. Experiments across closed and open-source models show that existing systems rarely produce well-timed, meaningful proactive interventions, while EgoVoice yields clear improvements in intervention timing, content relevance, and human preference over the zero-shot backbone.

1 Introduction

EgoVoice targets proactive spoken guidance from continuous egocentric streams, a joint capability not addressed by existing work. It builds training and evaluation around HoloAssist recordings and combines supervised fine-tuning with preference optimization.

1 Introduction

  • Proactive assistance matters when users are occupied, uncertain, or unaware they need guidance and cannot pause to ask for help.The assistant must decide whether to speak, when to speak, and what to say from first-person video and audio.Source: §1 · p. 1
  • EgoVoice converts HoloAssist recordings into clean, time-aligned supervision and trains MiniCPM-o 4.5 with SFT and DPO for proactive spoken guidance.The framework evaluates intervention timing, content relevance, and overall assistance quality.Source: §1 · pp. 1–2
  • Across closed-source, open-source, and trained models, existing systems rarely time proactive interventions well, while EgoVoice improves timing and content relevance over baselines.EgoVoice-SFT retains an edge in content relevance, while EgoVoice-DPO refines intervention timing.Source: §1 · p. 2
  • The work contributes a HoloAssist-based dataset and benchmark, an evaluation protocol, and comparisons across baselines and trained models.The dataset contains time-aligned intervention labels and spoken guidance targets.Source: §1 · p. 2

2 Related Work

Prior work advances multimodal spoken interaction, egocentric task understanding, and proactive streaming assistance, but does not combine them into one model for continuous spoken guidance.

Spoken Dialog and Omni-Modal Models

  • Omni-modal and full-duplex models support multimodal speech interaction, but are designed for reactive dialogue rather than proactive intervention during physical tasks.MiniCPM-o 4.5 is close in modality coverage but is a general-purpose assistant, not an egocentric task-guidance model.Source: §2 · p. 2

Egocentric Task Understanding and Datasets

  • Egocentric datasets support activity understanding and task assistance, but none targets continuous spoken guidance during ongoing tasks.HoloAssist includes real instructor speech, but its mono-channel audio and pitch-shifted speaker identities require processing for voice-interaction training.Source: §2 · pp. 2–3

Proactive Assistance in Streaming Video

  • Proactive video systems address speak-or-stay-silent decisions, but prior approaches produce text, predict only when to speak, or rely on cascaded pipelines.ProAssist and ESTP study egocentric streaming video, while Mirai and ProMemAssist demonstrate wearable spoken-assistance prototypes using cascaded API calls.Source: §2 · p. 3

3 Method

EgoVoice adapts streaming MiniCPM-o 4.5 for egocentric assistance, using processed HoloAssist recordings to train and evaluate when to listen and when to provide spoken guidance.

3.1 Data Construction

  • HoloAssist’s real instructor speech is behavior-aligned and retains ambient sound, unlike text-derived instructions whose playback timing can mismatch continuous user actions.The corpus is processed with source separation and speech resynthesis to create clean, synchronized streams.Source: §3.1 · p. 4
  • HoloAssist provides naturally timed instructor interventions and background sounds, but its mixed, pitch-shifted audio requires processing before model training.The processed corpus contains 1,249 training sessions (91.7 hours) and 170 held-out benchmark sessions (12.9 hours), from 1,673 original sessions.Source: §3.1 · p. 4

3.2 Evaluation Protocol

  • Onset timing uses matched utterances and an asymmetric window of −2.5 seconds early to +1.5 seconds late, with precision, recall, and F1 macroaveraged across sessions.Models receive task names and step-by-step recipes as procedural context.Source: §3.2 · p. 4
  • End-vs-action measures whether an utterance finishes before the user’s next action, and is reported only for directives linked to an action within five seconds.That subset contains 942 of 2,067 reference utterances across 170 benchmark sessions.Source: §3.2 · pp. 4–5
  • Content relevance is cosine similarity for temporally matched text, while an LLM judge rates session correctness, promptness, efficiency, and overall helpfulness.Metrics are evaluated on both model-generated text and ASR transcriptions of spoken output, representing intended text and what users hear.Source: §3.2 · p. 5

3.3 Training

  • The streaming model processes one-second video-and-audio chunks, choosing LISTEN or SPEAK and generating speech on SPEAK chunks.SFT trains the LLM and speech-token decoder with separate text/decision and speech-token losses; gradients from the speech loss do not flow through the LLM backbone.Source: §3.3 · pp. 5–6
  • DPO refines SFT behavior using preferred and rejected candidate interventions scored by onset F1, while excluding speech tokens from its likelihood.The reference model is the frozen SFT model; LoRA is applied to the LLM backbone because preferences concern intervention decisions and textual guidance.Source: §3.3 · p. 6

4 Experiments

The experiments compare EgoVoice with closed-source, open-source, cascaded, and zero-shot systems, using MiniCPM-o 4.5 as the backbone for EgoVoice’s trained variants.

Baselines

  • Closed-source comparisons include Gemini Live and two OpenAI Realtime API snapshots, evaluated without background audio to avoid false triggering.These systems accept streaming video and audio and produce text and speech.Source: §4 · p. 6
  • ProAssist is the text-only prior-work baseline, while its ASR/TTS cascade retains the same intervention decisions and assumes ideal zero latency between components.The end-to-end voice baseline is zero-shot MiniCPM-o 4.5, which also serves as EgoVoice’s backbone.Source: §4 · pp. 6–7

Our Models

  • The trained systems are EgoVoice-SFT and EgoVoice-DPO; SFT uses five epochs, while DPO further trains SFT for one epoch.The DPO checkpoint is selected based on validation performance.Source: §4 · p. 7

5 Results

Across 170 benchmark sessions, off-the-shelf systems struggle to intervene proactively, while EgoVoice improves timing and content relevance. Ablations and analyses show complementary SFT/DPO strengths alongside remaining timing and recall failures.

5.1 Model Comparison

  • Closed-source APIs reach F1 scores of at most 0.03, while zero-shot MiniCPM-o 4.5 reaches F1 = 0.113 but has the lowest Overall judge score (1.41).The ProAssist speech cascade leaves onset F1 unchanged at 0.195 and ends after the linked action in 62.0% of matched interventions, versus 24.3% for SFT and 28.2% for DPO.Source: §5.1 · pp. 7–8
  • EgoVoice-SFT scores higher than ProAssist on F1 (0.274 vs 0.195) and cosine similarity (0.684 vs 0.594); DPO raises SFT’s F1 by 0.050 (p = 0.0007) but lowers similarity to 0.649.The difference in overall judge score between SFT and DPO is not significant (p = 0.42).Source: §5.1 · pp. 7–8

5.2 Ablation Studies

  • Removing procedural context causes the largest SFT degradation across metrics except end-vs-action; removing gradient detachment slightly raises F1 but lowers judge scores.The authors conclude procedural context matters for both what guidance says and when it is delivered.Source: §5.2 · p. 8
  • All three tested DPO losses improve F1 over SFT; length-normalized DPO is selected for the most balanced timing, content, and end-vs-action performance.Increasing β above 0.10 prompts more low-content speech, while selecting preference pairs by both F1 and cosine similarity degrades both metrics versus SFT; F1-only selection works better.Source: §5.2 · p. 8

5.3 Human Evaluation

  • Both EgoVoice variants are preferred to MiniCPM-o 4.5 (p < 0.05); DPO’s slight numerical edge over SFT is not significant (p = 0.60).The table reports pairwise human preference results.Source: §5.3 · pp. 8–9, Table 3 · p. 9

5.4 Analysis

  • EgoVoice-DPO reaches Kendall’s τb = 0.868 and SFT 0.841, versus 0.850 for ProAssist and 0.749 for MiniCPM-o 4.5, indicating closer alignment with instructor task order.This analysis covers sessions with at least two matched intervention pairs.Source: §5.4 · p. 9
  • Both EgoVoice models deliver guidance more often than the backbone and narration close to instructors’ levels; excess brief acknowledgments account for most of their remaining distribution gap.70.7% of EgoVoice-DPO’s brief acknowledgments contain three words or fewer.Source: §5.4 · p. 9
  • EgoVoice-DPO still finishes after the linked action begins in 28.2% of matched interventions and misses most instructor intervention moments at recall 0.285.SFT’s late-finish rate is 24.3%; the authors describe these failures as far from solved.Source: §5.4 · p. 9

6 Conclusion

EgoVoice provides a framework and dataset for training and evaluating proactive egocentric spoken assistants. The authors report that SFT and DPO outperform baselines on onset F1 and content relevance.

6 Conclusion

  • The HoloAssist-derived spoken interaction dataset uses source separation and speech resynthesis, with an evaluation protocol covering timing, content relevance, and assistance quality.Source: §6 · p. 9
  • EgoVoice-SFT and EgoVoice-DPO, trained on MiniCPM-o 4.5 with supervised fine-tuning and direct preference optimization, outperform all baselines on onset F1 and content relevance.Source: §6 · p. 9
  • The authors position EgoVoice as an initial step toward always-on spoken assistants that proactively guide users through real-world tasks.Source: §6 · p. 9

Limitations

The authors identify limits in task coverage, model behavior and timing, evaluation realism, DPO content quality, and privacy.

Limitations

  • Training data covers specific procedural assembly tasks, so generalization to cooking, outdoor navigation, or social interaction is not guaranteed; performance also remains substantially below instructor behavior.The authors say more diverse data and larger-scale training would be needed to approach human-level proactive assistance.Source: Limitations · p. 10
  • MiniCPM-o 4.5 can produce overly long or repetitive responses and language-switching artifacts, while its one-second chunks may limit sub-second timing precision.Fine-tuning reduces these artifacts, but they remain in both EgoVoice variants.Source: Limitations · p. 10
  • Prerecorded evaluation fixes user actions independently of model advice, making scores a proxy for interactive performance; the protocol also excludes either speaker interrupting the other.Handling barge-in in both directions is left to future work.Source: Limitations · p. 10
  • DPO improves intervention timing but slightly reduces content quality, suggesting the reward signal needs refinement.Source: Limitations · p. 10
  • Always-on recording can capture bystanders without consent, and sending audio or video to external servers risks exposing sensitive information.The authors identify bystander-consent mechanisms, on-device processing, and audio redaction as needed mitigations.Source: Limitations · p. 10

Ethics Statement

The dataset uses synthetic resynthesized speech and anonymized sources, while the authors acknowledge privacy risks from always-on recording and restrict the released models to research use.

Ethics Statement

  • HoloAssist is released under CDLAv2, and the replacement speech is synthetic, sourced from speakers identified only by numeric IDs, and not linkable to named people.The authors say the separated background contains no residual speech and that future dataset versions will preserve this property.Source: Ethics Statement · p. 10
  • Human-evaluation workers were paid per completed task, and the study collected no personally identifying information beyond platform requirements.Source: Ethics Statement · p. 10
  • The authors flag risks to users and bystanders from continuous first-person recording and release the models for research, not monitoring-tool deployment.They discuss possible mitigations in the Limitations section.Source: Ethics Statement · p. 10

A.1 Data Construction Details

The construction pipeline separates background audio, normalizes and resynthesizes dialogue, then filters sessions for speech residue and synthesis quality.

A.1 Data Construction Details

  • SAM-Audio extracts prompted speech from the mono mixture, and the residual is retained as background audio; Silero VAD removes sessions where that background still contains speech.Speech intervals use metadata timestamps with a ±0.2-second conditioning margin.Source: §A.1 · p. 14
  • A two-stage LLM pipeline normalizes transcripts for spelling, punctuation, fillers, and proper nouns, removing task-irrelevant opening and closing utterances flagged in the metadata.Those removed utterances account for 9.0% of all utterances.Source: §A.1 · p. 14
  • The construction filters include TTS/WER quality and a minimum 60-second duration, while the evaluation split retains surviving sessions from the original HoloAssist evaluation split without further filtering.Source: Table 5 · p. 14
  • Each utterance is synthesized 20 times; candidates exceeding the original duration are discarded, and the lowest-WER candidate is selected for manual review when needed.Sessions with all selected utterances at WER = 0.0% are automatically retained; sessions with any mispronunciation are discarded after checking.Source: §A.1 · p. 14
  • 141 sessions were discarded at source separation because residual speech remained, the pipeline’s largest reduction.Source: §A.1 · p. 14, Table 5 · p. 14

A.2 Baseline Configurations

The baselines include real-time spoken APIs, a video-text proactive model, and the untuned MiniCPM-o backbone; their configurations and speaking behavior differ substantially.

A.2 Baseline Configurations

  • Gemini Live was evaluated with proactive audio enabled, JPEG video at 1 fps and 768×768 resolution, and a 15-second tail wait after video transmission.Source: §A.2 · pp. 14–15
  • Naive TTS can outlast the user’s available action window; moving it earlier can collide with prior speech or precede relevant context, motivating duration-constrained resynthesis.Source: Figure 4 · p. 15
  • OpenAI Realtime used the alloy voice, server-side VAD, 1-fps video at 768 px, and 16-bit PCM audio at 24 kHz.Source: §A.2 · p. 15
  • Gemini Live and OpenAI Realtime averaged 0.25, 0.79, and 0.94 utterances per session, respectively, in a setting where users are largely silent.Both APIs rely on voice activity detection to trigger responses; background noise was removed to prevent false triggers, a favorable evaluation condition.Source: §A.2 · p. 15
  • ProAssist is an 8B streaming egocentric video-text model that produces text rather than speech and does not accept audio input.Its cascade variant uses ASR transcripts of resynthesized user speech and synthesizes ProAssist text output with IndexTTS2.Source: §A.2 · p. 15
  • MiniCPM-o 4.5 averaged 4.85 utterances per session, but only 30.1% of its 825 benchmark utterances were proactive guidance.The remainder included 21.0% third-person scene narration and 49.0% acknowledgements, greetings, or hallucinated text.Source: §A.2 · p. 15

A.3 Training Hyperparameters

The appendix details evaluation procedures and supplementary results, including how intervention timing, utterance structure, and failure modes vary across systems.

A.3 Training Hyperparameters / A.4 Evaluation Details

  • The appendix points to Table 6 for SFT and DPO hyperparameters and evaluates action-linked directives against the nearest subsequent action within five seconds.The action-linked subset contains 942 of 2,067 reference utterances across 170 sessions; judge scores use four 1-to-5 axes, and utterance types are majority-classified over five calls.Source: §A.3 · p. 15, §A.4 · p. 16, Table 6 · p. 16

A.5 Human Evaluation

  • Human evaluators compare anonymous assistant overlays on the same egocentric clip, with randomized A/B sides and at least 50% playback required for both clips.The study recruits 100 workers per comparison and retains 252–267 valid judgments per comparison after excluding workers who fail the attention test.Source: §A.5 · p. 18, Figure 8 · p. 19

A.6.1 Results on the text-llm Channel

  • Across text-llm and text-asr, relative system rankings are largely consistent; the ASR control has onset F1 0.195 in both cases, while cosine drops by 0.009.Text-llm metrics are often slightly higher because internally generated text avoids TTS or ASR noise; onset-based matching also selects different intervention pairs across channels.Source: §A.6.1 · pp. 18, 20

A.6.2 End-of-Utterance Decomposition

  • EgoVoice-SFT and EgoVoice-DPO average 2.99 and 2.93 seconds per utterance, close to the instructor’s 3.05, while longer baseline turns account for late endings.All systems start speaking 3.4–5.9 seconds before the action on average; cascade, MiniCPM-o 4.5, and gpt-realtime-2 utterances last 6.38, 9.87, and 20.39 seconds, respectively.Source: §A.6.2 · p. 20, Table 8 · p. 20

A.6.3 Utterance Statistics

  • EgoVoice averages 8.1 and 7.8 words per utterance against the instructor’s 8.3, but speaks about half as often; baselines average 16.6–51.6 words.EgoVoice also has a similar share of brief acknowledgements, whereas baselines almost never produce them; the appendix links long turns to lower efficiency and late endings.Source: §A.6.3 · p. 20, Table 9 · p. 21

A.6.4 Significance Tests

  • Onset F1 separates every tested system pair, while judge scores distinguish both EgoVoice variants from the zero-shot backbone but not from ProAssist or each other.Cosine similarity is not tested because models are scored on different matched pairs, leaving no common set for a paired test.Source: §A.6.4 · p. 21

A.6.5 Qualitative Examples and Failure Modes

  • Qualitative examples show both successful instruction timing and two measured errors: overlong speech that continues into the user’s action, and guidance for an earlier task step.The late-ending examples last 5.8 and 4.0 seconds versus instructor durations of 1.6 and 1.4 seconds; 47 of 1,090 utterances also fail to stop, and language switching appears in 13.Source: §A.6.5 · pp. 21, 25, Figure 10 · p. 22, Figure 11 · p. 23

A.7 Artifacts and Licenses / A.8 AI Assistant Usage

  • The released dataset replaces original voices with zero-shot TTS from unrelated reference speakers, and the paper reports AI-assistant use for proofreading, analysis, plotting, and an initial figure draft.Artifacts and licenses are listed in Table 11; the source HoloAssist recordings are released under CDLAv2.Source: §A.7 · p. 25, §A.8 · p. 25
Loading 2610.12248v1…