PaperlayerView exact revision on arXiv

Source-linked AI summary

MA-EgoQA: Question Answering over Egocentric Videos from Multiple Embodied Agents

Kangsan Kim, Yanlai Yang, Suji Kim, Woongyeong Yeo, Youngwan Lee, Mengye Ren, Sung Ju Hwang

arXiv:2603.09827v2cs.CVcs.AI

TL;DR

Question answering across multiple long-horizon egocentric streams remains underexplored, especially when answers require integrating experiences across agents. MA-EgoQA benchmarks this setting, and EgoMAS combines shared memory with agent-wise dynamic retrieval. Current models struggle on the benchmark, while EgoMAS reaches 41.41% accuracy, 4.48% higher than the Gemini-2.5-Flash baseline.

  • Problem

    Multi-agent contextual question answering remains underexplored, while systems must interpret long egocentric histories and integrate information across agents to answer queries.

    Source: §1 · p. 2

  • Method

    MA-EgoQA benchmarks multi-agent egocentric question answering across five categories, while EgoMAS uses shared memory and agent-wise dynamic retrieval.

    Source: §1 · p. 3, §8 · p. 15

  • Results

    Current LLMs and video LLMs struggle with multiple egocentric streams; EgoMAS (Gemini-2.5-Flash) achieves 41.41% accuracy, 4.48% higher than the Gemini-2.5-Flash baseline.

    Source: §1 · p. 3, §6.2 · p. 12

  • Takeaways & Limitations

    The findings underscore the complexity of multi-agent egocentric understanding and identify MA-EgoQA as a promising direction for research on multiple agents.

    Source: §8 · p. 15

    MA-EgoQA is based only on EgoLife and does not include other scenarios in different environments.

    Source: C Limitation and Future Work · p. 20

Abstract

from arXiv · show

As embodied models become powerful, humans will collaborate with multiple embodied AI agents at their workplace or home in the future. To ensure better communication between human users and the multi-agent system, it is crucial to interpret incoming information from agents in parallel and refer to the appropriate context for each query. Existing challenges include effectively compressing and communicating high volumes of individual sensory inputs in the form of video and correctly aggregating multiple egocentric videos to construct system-level memory. In this work, we first formally define a novel problem of understanding multiple long-horizon egocentric videos simultaneously collected from embodied agents. To facilitate research in this direction, we introduce MultiAgent-EgoQA (MA-EgoQA), a benchmark designed to systemically evaluate existing models in our scenario. MA-EgoQA provides 1.7k questions unique to multiple egocentric streams, spanning five categories: social interaction, task coordination, theory-of-mind, temporal reasoning, and environmental interaction. We further propose a simple baseline model for MA-EgoQA named EgoMAS, which leverages shared memory across embodied agents and agent-wise dynamic retrieval. Through comprehensive evaluation across diverse baselines and EgoMAS on MA-EgoQA, we find that current approaches are unable to effectively handle multiple egocentric streams, highlighting the need for future advances in system-level understanding across the agents. The code and benchmark are available at https://ma-egoqa.github.io.

1 Introduction

MA-EgoQA studies question answering over long egocentric video streams from multiple embodied agents, where answering queries requires integrating experiences across agents. It introduces a benchmark and EgoMAS, a shared-memory baseline with selective retrieval, and reports that existing models struggle with the task.

1 Introduction

  • Multi-agent systems can decompose and execute tasks in parallel and use collective reasoning, but require robust communication and strategies for sub-task generation and allocation.These challenges matter for agents operating at scale in real-world environments.Source: §1 · p. 2
  • Contextual understanding and QA remain underexplored despite their importance for integrating agent experiences and retrieving relevant episodes to answer human managers’ queries.Prior work has predominantly focused on goal-directed algorithms, task allocation, and completion.Source: §1 · p. 2
  • Multi-agent video QA must locate query-relevant events across massive, potentially day-long streams and integrate events across agents, beyond the few-hour video scale current video LLMs can process.The streams arise because embodied agents continuously capture egocentric video during operation.Source: §1 · p. 2
  • MA-EgoQA contains 1.7k question-answer pairs from six people sharing a house for seven days, covering five categories unique and fundamental to multi-agent scenarios.Questions and answers are generated with GPT-based pipelines, refined through LLM filtering, and validated by human annotators.Source: §1 · p. 3
  • EgoMAS aggregates information from all agents into shared memory and selectively retrieves relevant information from appropriate agents, outperforming naive concatenation in token efficiency and QA performance.The model is presented as a baseline illustrating the potential of multi-agent video QA.Source: §1 · p. 3
  • EgoMAS outperforms the baseline by 4.48% with the same Gemini-2.5-Flash [9] backbone, while Qwen3VL-8B-Thinking [2] EgoMAS is comparable to the Gemini baseline at a 1M-token context.The results highlight both the difficulty of multi-agent egocentric video understanding and the promise of specialized approaches.Source: §1 · p. 3

2 Related Work

Prior work studies multi-agent cooperation [42] [43] [14] [5] and egocentric QA, while MA-EgoQA evaluates QA over multiple super-long, temporally aligned egocentric videos.

2.1 Multiple Embodied Agents System

  • Prior work studies multi-agent planning and cooperation [42], constrained communication [43], teamwork prompting [14], and collaboration benchmarks [5], not long-term multi-agent egocentric-memory QA.These studies primarily optimize action execution and cooperation strategies.Source: §2.1 · p. 3

2.2 Egocentric Video Understanding Benchmarks

  • Egocentric benchmarks cover episodic memory, long-term context, reasoning, and planning, but their video samples remain shorter than one hour.Ego4D provides 3,670 hours of egocentric video, while the per-sample duration in these benchmarks remains under one hour.Source: §2.2 · p. 4
  • EgoLife captures seven days of experiences from six people, but EgoLifeQA assumes single-agent memory; MA-EgoQA evaluates QA over multiple super-long, temporally aligned egocentric videos.MA-EgoQA is described as the first benchmark to evaluate this multi-video QA setting.Source: §2.2 · p. 4

3 MA-EgoQA Benchmark

MA-EgoQA defines question answering over multiple embodied agents’ egocentric video streams and benchmarks questions that require information across agents and timestamps. It organizes these questions around skills relevant to multi-agent interactions and long-horizon reasoning.

3.1 Task Definition

  • The task takes a user query and N agents’ video streams, requiring a temporally and contextually aligned global understanding and retrieval of relevant cross-agent events.Queries require information from more than two agents and may refer to different timestamps across their streams.Source: §3.1 · p. 5

3.2 Benchmark Categories

  • Categories cover cross-stream conversations and group behavior (SI), role and decision coordination toward goals (TC), and agents’ beliefs, perceptions, misunderstandings, and intentions (ToM).SI also includes meaningful conversational information; TC covers responsibilities and action sequencing.Source: §3.2 · pp. 5–6
  • Temporal Reasoning tests concurrent activities and cross-agent event ordering, while Environmental Interaction tracks object use, including frequency, first use, and agent involvement.Environmental Interaction questions support tracking shared environmental states and planning actions toward shared goals.Source: §3.2 · p. 6
  • Social Interaction and Task Coordination also include multi-span questions requiring reasoning across non-contiguous temporal windows, extending the benchmark to long horizons.The benchmark provides examples for each category.Source: §3.2 · p. 6

3.3 Benchmark Statistics

  • MA-EgoQA contains 1,741 questions based on six agents’ videos over seven days, with questions distributed across categories, days, and agents.Question wording ranges from common What and Who questions to Why, How, When, and Which formulations.Source: §3.3 · p. 6, Figure 3 · p. 6
  • The benchmark construction pipeline generates question-answer pairs through category-specific processes, then refines them with LLM filtering and manual checks.Figure 4 depicts this construction pipeline.Source: Figure 4 · p. 7

4 Benchmark Construction

MA-EgoQA uses category-specific generation strategies to create multi-agent questions, then filters candidate pairs and verifies the remainder with human reviewers. Its construction combines open-ended and template-based generation with checks for trivial, single-agent-answerable, or invalid questions.

4.1 Data Generation

  • For SI, TC, and ToM, GPT-based generation produces open-ended candidates for filtering; TR and EI use contextualized templates, with all questions offering five options.The prompts and templates for each generation stage are provided in Supplementary Sections E and F.Source: §4.1 · p. 7
  • For SI, TC, and ToM single-span questions, GPT-4o uses five-minute segments of all-agent captions and transcripts to generate questions grounded in more than two agents.The process generated 33.4k SI, 31.6k TC, and 34.1k ToM samples.Source: §4.1 · p. 7
  • Multi-span questions group semantically similar single-span pairs across timestamps, then GPT-5 synthesizes questions and answers while false options are generated separately.The method generated 15.9k SI and 16.3k TC multi-span samples.Source: §4.1 · pp. 7–8
  • TR templates generate comparison and concurrency questions from multi-agent captions at 30-second, 10-minute, or one-hour scales; EI templates use object interactions over six-hour or one-day intervals.For TR, comparison samples use different timestamps across agents, while concurrency samples align captions at one timestamp; GPT-5 generates five-option questions for both categories.Source: §4.1 · p. 8

4.2 LLM Filtering and Human Verification

  • GPT-5 filtering removes questions answerable without context or from a single agent’s memory, while Gemini-2.5-Flash and Claude-Sonnet-4 flag invalid questions, answers, or options.In zero-shot filtering, a question is discarded if GPT-5 answers correctly in more than two of three trials; either external model can trigger removal.Source: §4.2 · pp. 8–9
  • Four human verifiers reviewed 3,436 candidates, selecting 1,741 high-quality samples after automatic filtering.Verifiers had access to captions, transcripts, and videos from all agents and used the data-generation guidelines.Source: §4.2 · p. 9

5 Egocentric Video Reasoning in Multi-Agent System

EgoMAS is a training-free centralized multi-agent system for egocentric reasoning that combines shared event memory with agent-wise dynamic retrieval.

5 Egocentric Video Reasoning in Multi-Agent System

  • EgoMAS integrates fragmented events across agents into shared memory and dynamically selects agent memories and adapts queries for fine-grained multi-perspective reasoning.It is designed as a simple baseline to stimulate future research on MA-EgoQA.Source: §5 · pp. 9–10

5.1 Event-based Shared Memory

  • Every 10 minutes, agents caption their preceding observations; a centralized manager organizes key events into global memory with When, What, Where, Who, and How fields.This event-based structure aligns agent perspectives while preserving details for reasoning, rather than producing a flat textual condensation.Source: §5.1 · p. 10

5.2 Agent-wise Dynamic Retrieval

  • For a query, EgoMAS retrieves top-n shared memories by BM25, generates agent-specific subqueries, retrieves and filters agent memories, then conditions its response on both result sets.Memories scoring below threshold τ are filtered out.Source: §5.2 · p. 10

6 Experimental Setup and Results

The evaluation compares language, video-language, retrieval, and EgoMAS systems on MA-EgoQA, finding the benchmark difficult for current models while EgoMAS improves performance over the baselines. Results also show the importance of retrieval and leave a substantial gap to the oracle setting.

6.1 Experimental Setup

  • Text-only baselines receive chronologically concatenated captions, video LLMs receive uniformly sampled frames across agents, and RAG methods use frames and captions as memories.The evaluated models include proprietary and open-source LLMs, video LLMs, and BM25, DPR, and VideoRAG retrieval baselines.Source: §6.1 · pp. 10–11

6.2 Evaluation Results

  • EgoMAS significantly outperforms all baselines: its Gemini-2.5-Flash version reaches 41.41% accuracy, 4.48% higher than the Gemini-2.5-Flash baseline.Qwen3VL-8B-based EgoMAS models also surpass Gemini-2.5-Flash and GPT-5.Source: Table 2 · p. 11, §6.2 · p. 12
  • Gemini-2.5-Flash is the strongest model at 36.93% average accuracy, while many baselines perform only marginally above random chance.Only proprietary models in the all-caption-concatenation group exceed 30%; smaller open-source models remain below 27% despite 128k-to-1M-token contexts.Source: Table 2 · p. 11, §6.2 · p. 12

6.1 Experimental Setup

  • EgoMAS is evaluated with four backbones at n = 20, k = 5, and τ = 10; the oracle answers using the same context used to generate each question.For single-agent EgoLife baselines that do not support multiple agents, EgoRAG uses one person's videos and the other models use an LLM router to choose whose memory to consult.Source: §6.1 · p. 11

6.2 Evaluation Results

  • RAG baselines including BM25 and WorldMM-8B outperform other non-retrieval open-source models with shorter contexts, while all-frame concatenation has the lowest average performance among baseline categories.The paper attributes all-caption concatenation's weakness to irrelevant information and all-frame concatenation's weakness to missing transcripts and irrelevant frames.Source: §6.2 · p. 12

7 Analysis

Analysis shows that MA-EgoQA is difficult because it requires reasoning across multiple video spans and agents, while theory-of-mind questions are especially challenging. EgoMAS combines shared memory with agent-wise dynamic retrieval, achieving practical latency and succeeding in a qualitative case where baselines fail.

7.1 Analysis on the MA-EgoQA Benchmark

  • Restricting EgoMAS to one agent’s memory substantially reduces performance, showing that correctly answering MA-EgoQA queries requires access to multiple agents’ memories.The comparison is described in Fig. 7.Source: §7.1 · p. 13, Figure 7 · p. 13
  • Accuracy falls on multi-span queries and as the number of required agents increases, exposing limits in long-horizon retrieval and multi-agent knowledge fusion.The authors argue that systems need stronger retrieval across events and a coherent global understanding beyond simple memory sharing.Source: §7.1 · p. 13, Figure 8 · p. 13
  • ToM has the lowest accuracy across most models and remains behind other categories even with EgoMAS, reflecting the difficulty of inferring latent mental states [26].The paper attributes this challenge to ambiguity, recursive mental-state reasoning, and goal–belief–action inference [26].Source: §7.1 · p. 13

7.2 Analysis on EgoMAS

  • 1.3 seconds per query: EgoMAS has the highest accuracy among retrieval-based models, whose latency is notably lower than that of non-retrieval models.Latency was averaged over 100 randomly selected MA-EgoQA samples.Source: §7.2 · pp. 13–14
  • Using shared memory and agent-wise dynamic retrieval together achieves the highest accuracy, supporting cross-agent memory aggregation and retrieval of relevant knowledge.Without shared memory, the ablation uses 10-minute captions as the retrieval source; without dynamic retrieval, it answers using shared memories only.Source: §7.2 · p. 14, Table 4 · p. 14
  • Event-based 4W1H memory outperforms the alternative strategies [45], while lightweight BM25 offers competitive retrieval performance against dense retrievers [24] [47].NV-Embed-v2 achieves the highest accuracy among the compared retrievers but has 7B parameters and substantial computational overhead.Source: §7.2 · p. 14, Table 5 · p. 14
  • In the case study, EgoMAS answers correctly by locating target events in shared memory and retrieving details dynamically, while the compared baselines fail.Gemini-2.5-Flash and VideoChat-Flash receive excessive information; WorldMM retrieves agent memories iteratively without aggregating them.Source: Figure 9 · p. 14, §7.2 · p. 15

8 Conclusion

MA-EgoQA benchmarks question answering over multiple long-horizon egocentric streams across five categories of multi-agent reasoning. Its evaluations show that current LLMs and video LLMs struggle in this setting, while the training-free EgoMAS baseline achieves superior performance to recent frontier models despite using a much smaller model.

8 Conclusion

  • EgoMAS combines shared memory with agent-wise dynamic retrieval and outperforms recent frontier models with significantly long context despite using a much smaller model.It is a simple, training-free baseline.Source: §8 · p. 15
  • MA-EgoQA is a benchmark for question answering over multiple long-horizon egocentric video streams from embodied agents, spanning five categories of multi-agent reasoning challenges.Source: §8 · p. 15
  • Evaluations show that current LLMs and video LLMs struggle with multi-agent egocentric settings, particularly in theory of mind and multi-agent reasoning.The results underscore the complexity of the scenario and identify MA-EgoQA as a promising direction for multiple agents.Source: §8 · p. 15

Supplementary Materials · A Additional Analysis on EgoMAS · A.1 Ablation on Hyperparameters

EgoMAS performs best with shared-memory retrieval size n around 20–60, is robust to agent-wise retrieval size k, and loses accuracy at high score thresholds τ. The supplement also outlines additional analyses, experimental details, benchmark materials, and input-modality evaluations.

Supplementary Materials

  • The supplementary materials include additional analyses, experimental setup details, and a discussion of limitations and future work.These are presented in Sections A, B, and C, respectively.Source: Supplementary Materials · p. 19
  • The supplement provides benchmark category examples, benchmark-generation prompts, and question templates for the TR and EI categories.These materials appear in Sections D, E, and F, respectively.Source: Supplementary Materials · p. 19

A Additional Analysis on EgoMAS

  • The supplement also presents additional MA-EgoQA performance analysis across different input context modalities.This analysis is identified as Section H.Source: Supplementary Materials · p. 19

A.1 Ablation on Hyperparameters

  • Shared-memory retrieval size n is optimal around 20–60; values that are too small or large make it harder to find meaningful context.The ablation varies n, k, and τ.Source: §A.1 · p. 19, Figure S1 · p. 19
  • EgoMAS shows robust performance across different agent-wise dynamic retrieval sizes k.The ablation studies k alongside shared-memory retrieval size n and score threshold τ.Source: §A.1 · p. 19, Figure S1 · p. 19
  • High score thresholds τ decrease accuracy by filtering out most retrieved data.The ablation studies τ alongside shared-memory retrieval size n and agent-wise dynamic retrieval size k.Source: §A.1 · p. 19, Figure S1 · p. 19

A.2 Sensitivity Analysis to the Number of Available Agents · B Additional Details on Experimental Setups

EgoMAS accuracy improves as more agent views are available, underscoring the benchmark’s multi-agent demands. The accompanying setup details specify how caption- and frame-based baselines and retrieval systems were configured.

A.2 Sensitivity Analysis to the Number of Available Agents

  • EgoMAS accuracy rises from 31.99% to 35.55% as more agents are available, indicating that MA-EgoQA requires integrating multiple agents.Fig. S2 reports accuracy with limited agents.Source: Figure S2 · p. 19, §A.2 · p. 19

B Additional Details on Experimental Setups

  • All Caption Concat baselines use captions from every agent as response-generation context.Gemini-2.5-Flash, Llama-3.1-Nemotron-8B, and Qwen2.5-7B-Instruct-1M use a 10-minute window per agent; other models use a 1-hour window, with captions truncated if needed to fit context limits.Source: B Additional Details on Experimental… · p. 20
  • All Frame Concat baselines use uniformly sampled frames from agents’ videos across seven days.The sample counts are 10k frames for VideoChat-Flash, 4k for VideoXL-2, and 1.9k for Qwen2.5VL-7B, selected to avoid processing errors.Source: B Additional Details on Experimental… · p. 20
  • VideoRAG retrieves five 30-second clips by cosine similarity using InternVideo2 embeddings of each clip and its caption.It supplies each retrieved clip’s caption and four uniformly sampled frames to the response generator.Source: B Additional Details on Experimental… · p. 20
  • BM25 and DPR each retrieve five 30-second clips, providing captions and four uniformly sampled frames per clip to the responder.Source: B Additional Details on Experimental… · p. 20
  • Qwen3-VL-8B-Inst generates responses for all RAG baselines except Ego-R1.Source: B Additional Details on Experimental… · p. 20

C Limitation and Future Work … G Prompts for EgoMAS

The supplementary sections describe MA-EgoQA and EgoMAS limitations and future directions, then provide benchmark samples, generation prompts, question templates, and EgoMAS prompts.

C Limitation and Future Work

  • MA-EgoQA uses only EgoLife because it is the only publicly available long-term, simultaneous multi-agent egocentric video dataset, limiting evaluation across other scenarios.The authors call for more datasets and evaluation across varied videos to test generalization.Source: C Limitation and Future Work · p. 20
  • EgoMAS remains around 42.4% behind oracle performance; stronger retrieval improves results, motivating advanced strategies such as combining text embeddings with lexical matching.The authors identify more effective retrieval as a promising direction for future work.Source: C Limitation and Future Work · p. 20

D Benchmark QA Samples

  • The QA samples cover social interaction, task coordination, and theory of mind, with examples about a power-bank request, cross-situation coffee discussions, charcoal-lighting roles, and interpreting coffee instructions.The samples present questions with answer options.Source: D Benchmark QA Samples · pp. 21, 22, 23, 25
  • Temporal-reasoning samples ask about event sequences and concurrent activities, while the supplement also includes environmental-interaction samples.The sequence example draws on events from multiple agents; the concurrency example asks what Jake did while Katrina and Alice worked on flowers.Source: D Benchmark QA Samples · pp. 26–27, Table S8 · p. 27

E Benchmark Generation Prompts

  • The supplement provides a system prompt for social interaction, task coordination, and theory-of-mind QA, plus category prompts for all five benchmark categories.The category prompts cover social interaction, task coordination, theory of mind, temporal reasoning, and environmental interaction.Source: Figure S3 · p. 28, Figure S4 · p. 29, Figure S5 · p. 29

F Question Templates

G Prompts for EgoMAS

H Analysis on Input Context Modality

Video frames provide visual knowledge needed for some MA-EgoQA questions, but their benefit depends on the model and question category. The findings support selecting modalities and frames adaptively.

H Analysis on Input Context Modality

  • WorldMM-8B consistently improves with video frames alongside text captions, showing that MA-EgoQA questions can require visual knowledge unavailable in transcripts.Examples include identifying actions such as grabbing tissues and states such as being in another room upstairs.Source: H Analysis on Input Context Modality · p. 34, Table S11 · p. 34, Table S12 · p. 34
  • For EgoMAS, adding video frames improves SI, TR, and EI performance but lowers TC and ToM performance.The passage attributes possible drops to distraction from unnecessary frames and recommends adaptive modality use and effective frame selection.Source: H Analysis on Input Context Modality · p. 34

Related papers

Loading 2603.09827v2…