PaperlayerView exact revision on arXiv

Source-linked AI summary

Companion-style QA Assistance in Ego-Vision

Hangyu Qin, Junbin Xiao, Shenglang Zhang, Angela Yao

arXiv:2609.06721v1cs.CVcs.AI

TL;DR

Companion-style QA requires models to answer practical questions from long egocentric streams while resolving deictic references and chained conversational context. The paper introduces BuddyVQA and MyBuddy, which combines multimodal reasoning with filtering and multi-level memory. MyBuddy improves foundation-model performance on BuddyVQA and generalizes across streaming and common VideoQA benchmarks.

  • Problem

    Existing VideoQA benchmarks largely overlook ego-deictic expressions, interactively chained questions, and long-term context in egocentric streaming assistance.

  • Method

    MyBuddy uses multimodal chain-of-thought reasoning, question filtering, and multi-level memory to answer companion-style questions from historical QA and visual content.

  • Results

    MyBuddy achieves superior performance on BuddyVQA and existing long streaming VideoQA benchmarks, with ablations supporting its design components.

  • Takeaways & Limitations

    BuddyVQA and MyBuddy support evaluation and assistance for egocentric companion QA involving visual references, conversational dependencies, and streaming context.

  • Takeaways & Limitations

    Low-Active Memory consolidation assumes users are not continuously asking questions, so rapid query sequences may limit background consolidation opportunities.

Abstract

from arXiv · show

AI companions are envisioned as always-on assistants that support users in daily life. With this regard, we introduce BuddyVQA, a benchmark for companion-style question answering (QA) on egocentric streaming video. BuddyVQA contains 21.6K questions linked to 6K highlight moments across 1,012 long, egocentric videos. It features two key characteristics that are common in daily first-person QA assistance but are largely overlooked in existing VideoQA benchmarks: ego-deictic expressions and interactively chained questions (e.g., "Where is it?", "How to get there?"). These require models to infer a user's in-situation intent by resolving visual pronouns in the context of egocentric visual and QA contents, with both grounded in a long-form streaming setting. To tackle the challenges, we propose MyBuddy, a companion-style QA assistant that highlights a multimodal chain-of-thought reasoning mechanism to infer the final answer based on the historical QA and visual content. An additional question filter and multi-level memory are designed to facilitate efficient QA and visual information retrieval under streaming QA settings. Experiments show that MyBuddy significantly enhances the performance of foundation models on BuddyVQA. Moreover, these gains generalize to other streaming and common video QA benchmarks, demonstrating the applicability and effectiveness of our approach. Our code and dataset are available at https://github.com/QHUni/BuddyVQA

1 Introduction

The paper defines companion-style QA as egocentric, streaming assistance that must maintain visual context, resolve deictic references, and track conversational dependencies. It introduces BuddyVQA and MyBuddy to address these challenges, with MyBuddy achieving state-of-the-art results and generalizing across VideoQA settings.

  • Companion-style QA requires a visual assistant to maintain context over time, understand vague intentions, and provide practical answers during real-world interaction.
  • Existing VideoQA and streaming VQA settings overlook or only partially cover shared egocentric views, visual deictics, conversational continuity, and long-term memory.
  • BuddyVQA contains 21.6K questions linked to 6K moments across 1,012 long egocentric videos, emphasizing ego-deictics and interactively chained questions.
  • Existing foundation models rarely exceed 50% accuracy on BuddyVQA and frequently misinterpret references or miss historical information.
  • MyBuddy is a training-free, model-agnostic framework that uses visually grounded chain-of-thought reasoning and multi-level memory for companion-style VQA.
  • MyBuddy achieves state-of-the-art results on BuddyVQA and generalizes to other streaming and offline VideoQA datasets.

2 Related Works

Related work spans offline VideoQA, streaming VideoQA, egocentric VQA, long-video modeling, and multimodal dialogue. These approaches provide relevant capabilities but generally do not jointly address egocentric deictics, chained dependencies, and efficient long-term companion interaction.

  • Offline VideoQA processes complete third-person clips with fixed temporal scopes, whereas streaming VQA processes video online without knowing future questions.
  • Existing streaming VQA approaches use memory buffers to retain question-agnostic historical information under online processing constraints.
  • BuddyVQA differs from prior streaming work by focusing on shared egocentric vision and cross-modal chained questions with ego-deictics.
  • Earlier egocentric benchmarks study episodic memory, procedures, and long-range reasoning, while newer studies broaden embodied assistance but omit ego-deictic and chained QA.
  • Long-video systems reduce processing costs through token compression, sparse sampling, hierarchical memory, or retrieval-based reasoning, while streaming systems process frames incrementally.
  • Video-grounded dialogue methods often re-encode prior dialogue or treat history as undifferentiated text, making continuous interaction increasingly expensive.

3 Dataset: BuddyVQA

BuddyVQA is constructed from daily egocentric activities and targets visual deictics, interactive chaining, and realistic assistance across past, present, and future-oriented questions. Its statistics show frequent implicit references and dependencies extending across multiple conversational turns.

  • Dataset Construction: BuddyVQA uses EgoSchema videos and annotates highlight moments with ego-deictic questions grounded in shared camera perspective.
  • Dataset Construction: Interactively chained questions reference previous QA pairs within the same highlight moment and conversational context, preserving temporal coherence.
  • Dataset Construction: Gemini-2.5-Pro detects 2–18-second highlight segments and generates question drafts from their visual and temporal context.
  • Dataset Construction: Annotators label questions as Recall, Recognition, Scrutinization, or Advisory, covering past, current, and future time points.
  • Dataset Analysis: 21,574 questions span 6,002 highlight moments in 1,012 videos, with an average of 2.76 answers per question.
  • Dataset Analysis: 87.6% of questions contain a deictic pronoun, 79.1% are chained, and 57.6% of chained questions cannot be answered correctly without base QAs.
  • Dataset Analysis: 37.8% of chained questions follow 3–5 intervening QAs, 19.3% follow 6–8, and only 12.7% are directly consecutive.
  • Dataset Analysis: BuddyVQA combines chained questions, diverse ego-deictics, and multiple candidate answers to model realistic uncertainty in companion assistance.

4 Method: MyBuddy

MyBuddy addresses companion-style QA challenges by combining multimodal reasoning with retrieval from multi-level visual memory and historical QA. Its memory hierarchy and question-filtering strategy support efficient reasoning over recent, mid-term, and long-term streaming context.

  • Multimodal CoT Reasoning: MyBuddy uses multimodal chain-of-thought reasoning to resolve deictic references and connect chained questions with visual and historical QA context.The framework associates ego-gaze and hand-pointing regions with relevant historical interactions before answer generation.
  • Multi-Level Heterogeneous Memory: The system maintains Active, Semi-Active, and Low-Active memories at different temporal granularities and semantic abstractions.Active memory retains recent frame embeddings, Semi-Active memory aggregates older visual content, and Low-Active memory stores summarized long-term knowledge.
  • Multi-Level Heterogeneous Memory: Semi-Active Memory incrementally clusters older visual features with density-based DBSCAN instead of repeatedly reclustering all historical memories.Cluster representatives provide compact retrieval indices while member features preserve fine-grained visual evidence.
  • Multi-Level Heterogeneous Memory: Low-Active Memory uses overlapping summaries to retain recurring activities, persistent context, objects, actions, spatial relations, and temporal anchors.Its summarization runs during idle intervals, supporting long-term inquiries without competing with online question answering.
  • Efficiency: Memory maintenance is decoupled from online QA through asynchronous consolidation, clustering, and summarization during idle intervals.This design is intended to improve memory and time efficiency while avoiding additional response latency from those maintenance operations.
  • Historical QA and Question Filter: The question filter organizes historical QA pairs by temporal recency, searches short-term candidates first, and falls back to long-term candidates for distant dependencies.Questions without relevant historical pairs are treated as independent, while retrieved pairs support chained-question reasoning and reduce historical context.

5 Experiments

Experiments show that MyBuddy improves companion-style QA across BuddyVQA, especially for chained and ego-deictic questions, while maintaining performance over longer streams and generalizing to other video QA benchmarks. Ablations attribute these gains to historical QA, hierarchical rephrasing, embodied ego-cues, memory, and question filtering.

  • BuddyVQA comparison: 65.1% accuracy makes MyBuddy (GPT-5.5) the strongest BuddyVQA system, exceeding ReKV (LLaVA-OV) by 12.4%.MyBuddy (Qwen3.5) and MyBuddy (Gemini-3.1-pro) also reach 60.7% and 63.2%.
  • BuddyVQA comparison: MyBuddy reaches 63.7% on Advisory and 63.0% on Scrutinization, surpassing ReKV by 18.8% and 18.6%, respectively.These categories challenge models with subjective and context-dependent queries.
  • BuddyVQA comparison: MyBuddy remains stable from 64.6% to 63.8% as video length grows from 60s to 180s, unlike degrading baselines.ReKV falls from 54.0% to 48.6%, while InternVL-3.5 falls from 58.3% to 40.2%.
  • Chained and ego-deictic questions: MyBuddy exceeds ReKV by 11.9% on chained questions and 14.9% on ego-deictic questions across backbones.The larger deictic gains support combining ego-cues with history-aware reasoning.
  • Generalization: On VStream-QA, MyBuddy (GPT-5.5) achieves 66.1% on RVS-Ego and 58.9% on RVS-Movie, with 3.0–3.4s latency versus Flash-VStream’s 2.4s.These results indicate a speed–accuracy trade-off on 30–60-minute videos.
  • Generalization: MyBuddy remains competitive on standard VideoQA, reaching 82.9% on EgoSchema and outperforming listed baselines on Video-MME with Gemini-3.1-pro.On EgoSchema, 82.9% matches LVAgent and exceeds GPT-5.5’s 79.8%.
  • Ablation study: Removing historical QA lowers chained accuracy from 62.8% to 44.5%, while removing hierarchical rephrasing lowers chained and deictic accuracy by 10.1% and 12.6%.Removing embodied ego-cue extraction reduces deictic accuracy by 9.1%.
  • Ablation study: The Question Filter reduces inference cost by 1.6s and 20.9GB while increasing accuracy by 6.4% and chained accuracy by 10.7%.The reported gains are attributed to reduced noise and context length.

6 Conclusion

BuddyVQA captures interactive chaining and ego-deictic references in timestamp-specific egocentric QA, while MyBuddy combines multi-level memory with cross-modal chain-of-thought reasoning for streaming assistance.

  • BuddyVQA captures Interactive Chaining and Ego-Deictics in timestamp-specific, first-person companion-style QA.
  • MyBuddy uses multi-level memory and cross-modal chain-of-thought reasoning to retain video representations and interpret user intent from multimodal historical contexts.
  • MyBuddy achieves superior performance over strong competitors on BuddyVQA and existing long streaming VideoQA benchmarks.The paper states that ablations further demonstrate contributions from its designs.

7 Limitations and Future Work

The framework's current limitations concern asynchronous memory consolidation during rapid questioning and its single-user interaction assumption, while the associated data and method are intended for public release.

  • Asynchronous low-active-memory summarization assumes users are not continuously issuing queries, limiting background consolidation when questions arrive rapidly.The paper notes that summarization may then need to run alongside QA, potentially introducing latency.
  • BuddyVQA currently assumes a single primary user, whereas multiple people or agents make referential interpretation more complex.The authors identify person tracking, speaker-aware dialogue modeling, and social interaction understanding as future directions.
  • The authors state that the BuddyVQA dataset and MyBuddy method will be made publicly available with access instructions for reproducibility.

A.1 Video Activities

BuddyVQA covers diverse indoor and outdoor daily activities drawn from 1,012 EgoSchema videos, reflecting situations common in assistive contexts.

  • The dataset selects 1,012 EgoSchema videos sourced from Ego4D's diverse daily-life recordings.The source scenarios include household, outdoor, workplace, leisure, and construction activities.
  • BuddyVQA spans indoor activities such as cooking and cleaning and outdoor activities such as driving, shopping, gardening, and picnics.The activities are described as common in daily assistive contexts.

A.2 Automatic Question Candidate Generation

BuddyVQA combines automatically generated questions with human annotation and filtering to produce natural, temporally grounded, visually evidence-based companion QA.

  • Automatic filtering removes questions irrelevant to their video segments, near-duplicates, or answerable without visual information.
  • Twenty-five volunteers from diverse countries and occupations annotated QA pairs from generated highlight moments and draft questions.
  • Annotators were restricted to video content before each highlight's ending timestamp, and answers had to use information available by that time.
  • Annotators could use ego-deictic expressions and colloquial forms, while chained questions had to relate closely to a prior QA pair within the same highlight moment.Chained questions could differ in subject or phrasing and could not loop back as chain bases.
  • The annotation guidelines were designed to make human-authored questions and answers natural, temporally grounded, and well-defined.

A.4 Interactively Chaining Analysis

BuddyVQA questions form referential, temporal, and inferential chains that require models to connect current questions with prior answers and visual context. These dependencies test ambiguity resolution, temporal continuity, and higher-level interpretation.

  • Chained relationship types: Referential chains account for 41.2% of chained questions and use pronouns or vague expressions grounded by previous answers.Examples include “it,” “there,” and “that.”
  • Chained relationship types: Temporal chains account for 23.5% and require reasoning over earlier events or actions.They involve tracking event sequences and temporal continuity across observations.
  • Chained relationship types: Inferential chains account for 21.8% and ask for higher-level interpretation or evaluation based on earlier answers.For example, a model may judge whether an action was performed correctly after identifying it.
  • Reasoning demands: Referential chains test multimodal alignment and memory retrieval, whereas temporal and inferential chains require temporal indexing, continuity, causality, or abstraction.The three chain types therefore span grounding, sequence tracking, and interpretation demands.

B.1 Study of MyBuddy Design

The MyBuddy design study examines how question dependency, memory selection, rephrasing, embodied cues, and modular additions affect streaming QA efficiency and accuracy. Results favor selective multi-level context and explicitly grounded deictic-question processing.

  • Question dependency: Treating every question as dependent reduced accuracy from 65.1% to 56.5% and increased inference time by 32.7%.Accumulated historical QA introduced irrelevant context that interfered with grounding the current visual scene.
  • Question dependency: Treating every question as independent reduced overall accuracy from 65.1% to 38.2% by omitting essential contextual cues.The study therefore contrasts contamination from excessive history with information loss from no history.
  • Memory and routing: Combining working and episodic memory, then expanding to active, semi-active, and low-active memory, provided a more favorable efficiency-context balance.A lightweight zero-shot LLM routing strategy also outperformed a trained binary classifier.
  • Hierarchical rephrasing: Hierarchical rephrasing increased disambiguation accuracy from 62.4% to 86.7% while introducing limited latency.The module uses chain-of-thought reasoning to resolve deictic pronouns and vague expressions before answer generation.
  • Hierarchical rephrasing: Ego-deictic questions showed the largest relative improvement, while vague temporal, spatial, and action-related questions also benefited from clearer contextual grounding.These gains support more reliable temporal and procedural reasoning.
  • Ablation study: A controlled ablation improved accuracy from 50.5% with raw frames and historical QA to 52.2% after retrieval and rephrasing, then to 56.1% after adding the Question Filter.The study attributes progressively higher performance to the successive Buddy components.
Loading 2609.06721v1…