PaperlayerView exact revision on arXiv

Source-linked AI summary

Vinci2: Providing Proactive Assistance in Continuous Egocentric Videos

Gong Sitong, Tianyu Yan, Caixin Kang, Bo Zheng, Xiang Ruan, Huchuan Lu, Kaipeng Zhang, Yoichi Sato, Yifei Huang

arXiv:2607.11523v1cs.CVcs.AI

TL;DR

Proactive egocentric assistants must decide when intervention is warranted using evolving context, rather than merely reacting to prompts or predefined events. Vinci2 introduces EgoServe and the training-free EgoMemo agent, which establishes strong baselines on EgoServe and competitive or state-of-the-art results on five existing benchmarks.

  • Problem

    Existing egocentric assistants lack mechanisms to reason over long-horizon observations and determine whether the current situation warrants proactive intervention.

  • Method

    Vinci2 combines EgoServe, a benchmark spanning four temporal horizons and ten categories, with EgoMemo, a training-free agent using retrieval-augmented reasoning over complementary memories.

  • Results

    EgoMemo establishes strong baselines on EgoServe and achieves competitive or state-of-the-art results on five existing benchmarks.

  • Takeaways & Limitations

    EgoServe provides a large-scale benchmark for evaluating proactive assistance in continuous egocentric video, including whether agents should intervene or remain silent.

  • Takeaways & Limitations

    EgoServe scores remain moderate across methods because proactive assistance requires timely intervention, correct service selection, and helpful response generation.

Abstract

from arXiv · show

When should an intelligent assistant speak up without being asked? Continuous egocentric video offers rich, evolving context that enables a new form of assistance: one that is proactive rather than merely reactive. Yet existing approaches either wait passively for user queries or treat every detected event as requiring a response, without considering the user's history, current activity, or whether assistance would actually be welcome. We reframe proactive assistance as a context-dependent decision problem: the agent must not only perceive what is happening, but reason over accumulated temporal context to determine when and whether to intervene. To this end, we present Vinci2, a proactive egocentric assistance system that advances the on-device assistant Vinci from reactive response toward proactivity. On the evaluation side, we present EgoServe, the first large-scale benchmark for proactive assistance in continuous egocentric video. EgoServe comprises over 3,000 service instances organized along 4 temporal memory horizons, ranging from immediate safety alerts to long-term habit coaching, across 10 service categories. On the modeling side, we propose EgoMemo, a training-free, memory-augmented agent that maintains three complementary memory representations: multi-scale temporal summaries, a semantic knowledge graph, and visual embedding archives. At each timestep, EgoMemo performs retrieval-augmented reasoning to determine whether assistance is warranted and, if so, produces contextually grounded responses. Experiments demonstrate that EgoMemo establishes strong baselines on EgoServe while remaining competitive on existing egocentric benchmarks. Our benchmark and code are publicly available at \href{https://sitonggong.github.io/EgoServe-page/}{Vinci2}.

1 Introduction

The introduction frames proactive egocentric assistance as a context-dependent decision problem: an assistant must understand evolving first-person video and decide when help is warranted. Vinci2 addresses this gap with EgoServe, a benchmark spanning temporal horizons and service categories, and EgoMemo, a training-free memory-augmented agent using retrieval-augmented reasoning.

  • Motivation: Proactive assistance requires reasoning over evolving user context to offer help at the right moment without an explicit request.The introduction positions continuous egocentric video as enabling comprehension and reasoning over first-person experiences.
  • Problem: Existing assistants are reactive when prompted or semi-proactive when monitoring user-specified instructions for predefined events.These paradigms constrain assistance to explicit prompts or the scope of an initial instruction.
  • Evaluation gap: Existing benchmarks evaluate clip comprehension, streaming understanding, or instruction-driven task completion, but not autonomous proactive intervention decisions.The introduction identifies the absence of a benchmark for deciding whether assistance should be provided during continuous video.
  • EgoMemo: EgoMemo is a training-free, memory-augmented agent that uses multi-scale temporal summaries, a semantic knowledge graph, and visual embedding archives for long-horizon retrieval.It applies retrieval-augmented reasoning to determine whether a response is warranted and generate grounded assistance.
  • EgoServe: EgoServe is introduced as the first proactive-assistance benchmark, comprising over 3,000 service instances across 4 temporal horizons and 10 service categories.The benchmark covers assistance needs ranging from immediate safety alerts to long-term habit coaching.
  • Results: EgoMemo establishes strong baselines on EgoServe while remaining competitive on existing egocentric benchmarks.The reported results support the feasibility of proactive assistance through retrieval-augmented reasoning.

2 Related Works

Prior egocentric video research primarily studies offline understanding and retrieval, while newer streaming systems enable real-time conversation. Both paradigms answer questions about observed content rather than deciding whether and when to proactively intervene, a gap EgoServe addresses.

  • Egocentric Video Understanding: Offline egocentric benchmarks evaluate episodic memory, cognitive capabilities, cross-view understanding, and video-language retrieval using pre-recorded footage.Examples include EgoSchema, EgoThink, EgoExoLearn, EgoExoBench, EgoVLP, and LaViLa.
  • Egocentric Video Understanding: Existing offline and streaming approaches answer questions about observed content without modeling whether and when to proactively intervene.EgoServe fills this gap by evaluating proactive intervention decisions.
  • Egocentric Video Understanding: Streaming systems enable real-time video conversation, low-latency perception-reasoning, and online video comprehension.Examples include VideoLLM-online, Flash-VStream, StreamChat, Dispider, and OVO-Bench.

3 EgoServe Benchmark

EgoServe frames proactive assistance as a joint intervention-and-response task over continuous egocentric video, evaluated across temporal horizons and service contexts. The benchmark requires timely, correctly categorized, and contextually relevant assistance.

  • Task formulation: At each timestep, the agent makes a binary intervention decision and generates a service response when intervention is selected.This formalizes proactive assistance as joint decision and generation over sequential video clips.
  • Task formulation: A correct proactive response must occur near the ground-truth trigger, match its service category, and address the service need using observed context.Response relevance and grounding are assessed through LLM-based evaluation against reference responses.
  • Data construction: The benchmark is built from three egocentric video datasets spanning diverse scenarios and temporal scales, including multi-day recordings for long-horizon reasoning.EgoLife contributes recordings across participants and their first five days, supporting connections between temporally distant events.
  • Service taxonomy: EgoServe organizes services by four temporal memory horizons and application context, yielding four major categories and ten subcategories.The horizons span instant, short-term, episodic, and long-term reasoning requirements.
  • Data construction: A semi-automated annotation pipeline uses category-specific foundation-model prompts grounded in existing human annotations to identify service timing and responses.For long-term services, cues accumulate across intervals and are combined with future timeline annotations to simulate cross-session reasoning.

4 Methodology

EgoMemo is a training-free, memory-augmented agent that continuously builds structured memory from egocentric video and uses retrieval-augmented reasoning to decide whether and how to intervene. Its three complementary memory representations support temporal, semantic, and visual retrieval, which are unified into context for a shared proactive or reactive reasoning pipeline.

  • Agent overview: EgoMemo continuously processes incoming clips, maintains structured long-term memory, and autonomously decides whether to provide proactive interventions or respond to user queries.Reactive mode uses the same retrieval pipeline with a user query as the trigger and sets d_t = 1 by default.
  • Memory construction: Clip captions are incrementally organized into clip-, activity-, and session-level temporal summaries that preserve detail while encoding broader activity and session context.Only newly accumulated segments trigger summarization at the next level, and all levels are indexed with dense text embeddings for similarity search.
  • Retrieval and reasoning: At each timestep, the agent generates a retrieval query and performs parallel temporal, graph-based semantic, and visual similarity retrieval constrained to past observations.Temporal memory supplies fine-grained or long-range context, graph expansion captures differently worded or distant events, and visual retrieval matches multimodal embeddings.
  • Retrieval and reasoning: Retrieved graph and visual indices are reconstructed into query-focused captions, aggregated with temporal results, and passed to a reasoner that outputs an intervention decision d_t and response r_t.The unified formulation uses no architectural modification between proactive assistance and reactive question answering.
  • Memory construction: An evolving knowledge graph extracts and merges entities and relations from captions, while a visual embedding archive stores keyframe representations with source captions and timestamps.Visual retrieval complements text-based memory by recovering visually relevant moments that may lack lexical overlap with a query.

5 Experiments

Experiments show that EgoMemo substantially improves proactive assistance on EgoServe through structured memory and retrieval, while generalizing competitively across streaming and offline egocentric benchmarks. Ablations and qualitative examples further indicate that its temporal, semantic, and visual pathways provide complementary evidence, though proactive assistance remains difficult.

  • EgoServe: EgoMemo achieves 8.0 overall F1 on EgoServe, nearly doubling GPT-5-mini’s 4.7 and substantially outperforming baselines on long-term services.Memory Link reaches 4.9 and Routine Optimization 11.8, versus 0.0 for both baselines in each category.
  • EgoServe ablations: EgoMemo’s performance falls from 8.0 to 7.0 without multi-scale summaries and to 6.6 without caption reconstruction, showing both components are important.Memory Link drops from 4.9 to 1.9 and Routine Optimization from 11.8 to 5.7 without the temporal hierarchy.
  • EgoServe ablations: Removing semantic graph retrieval lowers F1 to 6.5 and Memory Link to 0.0, while removing multi-scale temporal retrieval lowers F1 to 6.8.Caption reconstruction and graph semantic retrieval have the largest individual effects, while temporal, semantic, and visual retrieval contribute complementary evidence.
  • Qualitative analysis: Qualitative cases show EgoMemo triggering an immediate knife-scrubbing safety alert from current-clip evidence and a Day 2 routine intervention from recurring cross-session phone-recording patterns.The examples illustrate assistance based on both immediate observations and retrieved long-term memory.
  • Cross-benchmark generalization: On ESTP-Bench, EgoMemo scores 27.6 on explicit proactive tasks and 34.7 on implicit proactive tasks, surpassing EyeWO’s 23.6 on explicit tasks but trailing its 52.5 on implicit tasks.It reaches 32.4 versus 25.1 on TRU and 35.8 versus 20.8 on EOSC; EgoMemo is training-free, unlike EyeWO.
  • Cross-benchmark generalization: EgoMemo obtains the best real-time perception score of 75.15 and leads two of three offline QA datasets, scoring 74.8 on EgoSchema and 68.0 on QAEgo4D.The streaming-first architecture generalizes across streaming and offline settings without architectural modification.

6 Conclusion … Output Schema and Quality Control

Vinci2 contributes EgoServe, a large-scale benchmark for proactive assistance in continuous egocentric video, and EgoMemo, a training-free memory-augmented agent for retrieval-augmented reasoning. The supplementary materials detail benchmark construction across multiple datasets, temporal horizons, and quality-controlled service annotations.

  • 6 Conclusion: EgoServe contains over 3,000 service instances across 10 categories and 4 temporal memory horizons, while EgoMemo maintains three complementary memory representations.The representations are multi-scale temporal summaries, an evolving knowledge graph, and a visual embedding archive.
  • A. EgoServe Benchmark: 3.4k service instances span ∼128h of video from three source datasets, covering egocentric perspective, multi-day temporal span, proactive evaluation, and streaming inference.EgoServe is described as the first benchmark to cover these four properties simultaneously.
  • A.1 Annotation Details: Annotation uses category-specific prompts to generate candidate service instances from human annotations, followed by manual verification before final benchmark inclusion.The complete prompt templates are provided in the supplementary material.
  • EgoLife Subset: EgoLife processes long-term services with streaming cue capture, carrying accumulated cross-segment events between turns to build richer multi-day context.This design grounds services in genuine multi-day patterns rather than single-interval observations.
  • HoloAssist Subset: HoloAssist uses structured procedural annotations in a single-stage pipeline to generate primarily Instant and Short-Term services from task-oriented interactions.Its annotations include step boundaries, instructor corrections, error flags, and chronological interaction timelines.
  • CaptainCook4D Subset: CaptainCook4D generates Tool Use, Next-Step Guidance, and Error Recovery instances by mapping procedural step transitions and error labels to proactive feedback.The selected 87 videos contain explicit step-error annotations across 24 recipes.
  • Output Schema and Quality Control: Each service instance follows a structured JSON schema containing a temporal trigger window, category label, observation context, and proactive response.Long-Term outputs additionally include linked past events and cross-session reasoning chains, while prompts enforce mutual exclusion rules.
  • Output Schema and Quality Control: Generated responses are contextually grounded in observed video content, referencing current objects, actions, or states and incorporating past events for Episodic and Long-Term services.The supplied passage describes visualizations spanning 11 service subcategories and emphasizes direct anchoring to video evidence.

A.2 Manual Verification

EgoServe annotations undergo joint manual verification, with acceptance requiring consensus and rejection for utility, temporal-alignment, or visual-grounding failures. Of 4,038 candidates, 3,437 remain after verification, for an 85.1% overall acceptance rate, while verification also corrects category misassignments.

  • A.2 Manual Verification: Two trained annotators jointly review each candidate and retain it only when both reach consensus.They watch the corresponding source video segment together and discuss acceptance; disagreement leads to discarding the candidate.
  • A.2 Manual Verification: Candidates are rejected for insufficient practical utility, temporal misalignment, or inadequate visual grounding.Annotations relying solely on speech transcripts or dialogue without visual observations are excluded.
  • A.2 Manual Verification: 4,038 candidates generated by the annotation pipeline were reduced to 3,437 after verification, yielding an 85.1% overall acceptance rate.Safety Alert increased from 195 to 241 because genuine safety events were reassigned from other categories after Gemini misclassification.
  • A.2 Manual Verification: Among Long-Term services, Memory Link achieved 100.0% retention, Routine Optimization reached 85.9%, and Habit Coaching had the lowest rate at 80.3%.The lower Habit Coaching rate is attributed to cross-session behavioral patterns producing responses with insufficient practical utility.

B. Methods … C. Experiments

EgoMemo builds streaming, multi-scale temporal memory and adapts retrieval-augmented reasoning across proactive, online, and offline egocentric benchmarks. Experiments evaluate the resulting memory hierarchy and caption reconstruction through EgoServe ablations.

  • B. Methods: EgoMemo constructs clip-, activity-, and session-level captions that preserve fine-grained actions while progressively summarizing broader temporal spans.Clip-level captions retain timestamped descriptions; activity- and session-level captions capture medium- and long-horizon context.
  • B.1 Details on Multi-Scale Temporal Memory Construction: Named entities extracted from captions form an evolving knowledge graph, while caption embeddings are indexed for structure-aware semantic retrieval.The memory combines entity-based graph structure with dense embedding similarity search.
  • B.1 Details on Multi-Scale Temporal Memory Construction: Streaming construction summarizes only newly accumulated segments, keeping computational cost and latency bounded as videos extend across multiple days.Caption generation, summarization, entity extraction, and embedding indexing operate incrementally rather than reprocessing the full history.
  • B.2 More Details on Different Benchmarks: At each reasoning step, the agent decides whether retrieval is needed by assessing whether the available streaming context is sufficient to answer.This decision is made by the reasoning LLM through a structured prompt.
  • C. Experiments: EgoServe experiments report Precision (P) and Recall (R), including ablations that remove session-level memory or VLM-based caption reconstruction.The w/o MS variant uses only clip-level captions, while w/o Recons. removes caption reconstruction.
  • EgoServe (EgoLife Subset): For EgoLife, reasoning combines the current clip caption with the latest activity summary and restricts historical retrieval to temporally resolved ranges when necessary.Relative scopes such as “last one hour” are converted into absolute recording intervals before retrieval.
  • B.2 More Details on Different Benchmarks: OVO-Bench realtime questions use the last 10 seconds of clip captions plus all activity- and session-level summaries without retrieval.This configuration directly supplies current and summarized context for realtime perception.
  • Offline Egocentric Benchmarks: Offline benchmark captioning conditions the VLM on frames, questions, and answer options, while inference retrieves fine-grained evidence for up to 3 rounds when summaries are insufficient.For very short videos without activity summaries, all clip-level captions are provided directly to the reasoner.

C.1 Detailed Results on EgoServe Benchmark

EgoMemo achieves the strongest average Precision and Recall on EgoServe, while ablations show that temporal retrieval and caption reconstruction are important for long-term coverage and intervention quality. Performance varies by subset: GPT-5-mini leads CaptainCook4D overall, whereas simplified retrieval variants perform best on HoloAssist.

  • EgoServe benchmark: EgoMemo achieves the highest average Precision (9.4) and Recall (17.8) on EgoServe, balancing proactive coverage with intervention quality.Across models, Recall exceeds Precision; GPT-5-mini shows the opposite tradeoff with Recall (14.8) and Precision (5.3).
  • Ablation analysis: Removing multi-scale temporal retrieval causes the largest Long-Term Recall drop, with RO decreasing from 18.8 to 7.8.Removing caption reconstruction produces the largest average Precision decline, from 9.4 to 6.8.
  • Subset results: On CaptainCook4D, GPT-5-mini achieves the best overall score (8.4), exceeding EgoMemo’s overall score (7.2) in structured cooking tasks.The passage attributes the subset’s generally higher scores to the structured procedural nature of cooking tasks.
  • Subset results: On HoloAssist, EgoMemo leads Next-Step Guidance with 36.3, while w/o GSR and w/o MS achieves the highest overall score (9.0).The ablation result suggests graph-based semantic retrieval and high-level captions provide limited benefit for short, self-contained procedural videos.

C.2 Retrieval Time Analysis

The retrieval-time analysis measures single-retrieval processing latency on 30 EgoSchema test videos, while separating EgoMemo’s asynchronous memory-construction cost from downstream query latency. VideoAgent requires 67.25 seconds per minute because of iterative multi-round reasoning, whereas memory construction takes 77.33 seconds per minute without increasing retrieval latency.

  • C.2 Retrieval Time Analysis: 30 EgoSchema test videos were sampled to measure each method’s average processing time per video minute under single retrieval.In this setting, the model decides whether retrieval is needed and performs at most one retrieval pass.
  • C.2 Retrieval Time Analysis: 67.25 seconds per minute is required by VideoAgent, attributed to its iterative multi-round agent reasoning process.
  • C.2 Retrieval Time Analysis: 77.33 seconds per minute is required for EgoMemo’s memory construction, but asynchronous streaming prevents this cost from increasing per-query retrieval latency.Memory construction and downstream reasoning operate asynchronously in the streaming pipeline.

C.3 Evaluation Protocol

The evaluation protocol matches predicted and ground-truth service events using subtype-specific temporal tolerances and one-to-one greedy assignment. It reports macro-averaged detection F1 alongside LLM-judged dialogue Rationality and Effectiveness, using two independent judges for robustness.

  • Temporal matching: Predicted and ground-truth events are matched separately by service subtype using greedy nearest-neighbor assignment within dataset-specific temporal tolerances.The tolerances are δ = 60 s for EgoLife, δ = 10 s for HoloAssist, and δ = 25 s for CaptainCook4D.
  • Temporal matching: Each prediction matches at most one ground-truth event, enforcing a one-to-one assignment.Distance is zero when the prediction center falls inside the tolerated ground-truth interval and otherwise equals the gap to its nearest boundary.
  • Detection metrics: Detection is evaluated with globally aggregated subtype-level Precision, Recall, and F1, using macro-averaged F1 over active subtypes as the primary metric.Active subtypes contain at least one ground-truth event or prediction, preventing dominant subtypes from overshadowing rare ones.
  • LLM-as-judge scoring: Matched prediction–ground-truth dialogue pairs are scored by an LLM judge on Rationality and Effectiveness using a 1–5 scale.The protocol evaluates generated service-dialogue quality beyond detection accuracy.
  • LLM-as-judge scoring: GPT-4o and DeepSeek-R1 produce highly consistent relative rankings despite different absolute scores, with mean overall scores of 2.81 and 2.38, respectively.DeepSeek-R1 is systematically stricter, and both judges identify the GPT-5-mini base configuration as best-performing.

C.3 More Qualitative Results

Fig. 7 qualitatively demonstrates EgoMemo’s proactive assistance across four service categories, showing both streaming intervention decisions and generated content. The examples range from recent-observation reminders to broader-context guidance and long-term habit coaching, while remaining silent when no actionable opportunity is detected.

  • Qualitative examples: Fig. 7 presents four qualitative EgoMemo examples spanning Resource Reminder, Task Reminder, Next-Step Guidance, and Habit Coaching.Each example shows the streaming decision process—Trigger or Keep Silent—and the generated service content.
  • Recent-context services: Resource Reminder and Task Reminder cases use recent clip-level observations to prompt checking a faucet and completing or tidying unfinished flower-arranging materials.The Resource Reminder occurs on Day 1, while the Task Reminder occurs on Day 4.
  • Broader-context services: Next-Step Guidance demonstrates broader-context assistance when the user places a green box on the table during a meal and receives a natural follow-up suggestion.The example is triggered on Day 2.
  • Broader-context services: Habit Coaching retrieves repeated phone-use evidence across earlier social gatherings and synthesizes the cross-day pattern into a suggestion to reduce screen time.The example occurs on Day 5 when the user checks their phone.
  • Streaming behavior: EgoMemo remains silent during intervals without actionable service opportunities, avoiding unnecessary interruptions.The streaming timeline explicitly shows this behavior.
Loading 2607.11523v1…