Source-linked AI summary
MA-EgoQA: Question Answering over Egocentric Videos from Multiple Embodied Agents
Kangsan Kim, Yanlai Yang, Suji Kim, Woongyeong Yeo, Youngwan Lee, Mengye Ren, Sung Ju Hwang
TL;DR
Existing models struggle to answer questions requiring integration of multiple long-horizon egocentric streams, a capability important for multi-agent embodied systems. The paper introduces MA-EgoQA and EgoMAS, which uses shared memory and agent-wise dynamic retrieval; evaluations show the benchmark is difficult while the specialized baseline improves performance. MA-EgoQA is currently based on one multi-agent video dataset, and EgoMAS remains substantially below oracle performance.
Problem
Question answering over multiple embodied agents remains underexplored because systems must integrate long-horizon event histories, environmental states, and inter-agent communication across massive video streams.
Method
The paper introduces MA-EgoQA, a five-category benchmark, and EgoMAS, a baseline using shared memory across agents with agent-wise dynamic retrieval.
Results
Current models struggle on MA-EgoQA, while EgoMAS with Gemini-2.5-Flash reaches 41.41% accuracy, 4.48% above the same-backbone baseline.
Takeaways & Limitations
MA-EgoQA highlights the difficulty of system-level understanding across multiple egocentric streams and supports specialized retrieval-based approaches such as EgoMAS.
Takeaways & Limitations
MA-EgoQA uses only the EgoLife dataset, and EgoMAS remains around 42.4% below oracle performance.
Abstract
from arXiv · showhide
As embodied models become powerful, humans will collaborate with multiple embodied AI agents at their workplace or home in the future. To ensure better communication between human users and the multi-agent system, it is crucial to interpret incoming information from agents in parallel and refer to the appropriate context for each query. Existing challenges include effectively compressing and communicating high volumes of individual sensory inputs in the form of video and correctly aggregating multiple egocentric videos to construct system-level memory. In this work, we first formally define a novel problem of understanding multiple long-horizon egocentric videos simultaneously collected from embodied agents. To facilitate research in this direction, we introduce MultiAgent-EgoQA (MA-EgoQA), a benchmark designed to systemically evaluate existing models in our scenario. MA-EgoQA provides 1.7k questions unique to multiple egocentric streams, spanning five categories: social interaction, task coordination, theory-of-mind, temporal reasoning, and environmental interaction. We further propose a simple baseline model for MA-EgoQA named EgoMAS, which leverages shared memory across embodied agents and agent-wise dynamic retrieval. Through comprehensive evaluation across diverse baselines and EgoMAS on MA-EgoQA, we find that current approaches are unable to effectively handle multiple egocentric streams, highlighting the need for future advances in system-level understanding across the agents. The code and benchmark are available at https://ma-egoqa.github.io.
1 Introduction
Multi-agent embodied systems require models to integrate long-horizon egocentric video and inter-agent context for question answering. MA-EgoQA benchmarks this capability and EgoMAS provides a shared-memory retrieval baseline, while existing models remain challenged.
- Multi-agent systems can execute decomposed tasks in parallel and use collective reasoning, but require robust communication and sub-task coordination.
- Question answering over embodied agents remains underexplored despite the need to interpret event histories, environmental states, and inter-agent communication.
- Long-horizon QA must locate query-relevant events across massive streams generated by agents operating for days, exceeding current video LLM processing limits of a few hours.
- 1.7k question-answer pairs in MA-EgoQA cover five multi-agent categories using videos from six people living together for seven days.The dataset uses GPT-based generation, LLM filtering, and human validation.
2 Related Work
Prior embodied-agent research emphasizes cooperation and egocentric video benchmarks, but existing QA settings do not evaluate temporally aligned reasoning across multiple agents. MA-EgoQA addresses this gap with a multi-agent, super-long video QA benchmark.
- Existing embodied-agent systems study cooperation, planning, communication limits, and partial observability across diverse environments.
- Ego4D established a large egocentric video resource, while EgoSchema targets minute-level understanding and long-term context.
- EgoLife provides seven consecutive days of synchronized egocentric video from six people in a shared house, breaking prior length barriers.
- EgoLifeQA assumes a single agent, whereas MA-EgoQA evaluates QA over multiple super-long, temporally aligned egocentric videos.
3 MA-EgoQA Benchmark
MA-EgoQA evaluates reasoning over multiple egocentric streams through five categories designed around social, cooperative, mental-state, temporal, and environmental interactions. Its questions span agents, days, and multiple timestamps.
- Task Definition: MA-EgoQA defines multi-agent video QA as answering queries from continuously recorded streams whose information must be integrated across agents.
- Benchmark Categories: The benchmark includes Social Interaction, Task Coordination, Theory of Mind, Temporal Reasoning, and Environmental Interaction categories.
- Task Coordination: Task Coordination evaluates role assignment, responsibility division, action sequencing, and decision-making during shared-goal execution.
- Theory of Mind: Theory of Mind covers agents’ beliefs, misunderstandings, perceptual access, and intentions underlying actions.
- Temporal and Environmental Reasoning: Temporal Reasoning aligns timelines across streams through concurrency and cross-agent event-order comparisons, while Environmental Interaction tracks distributed object usage.
- Benchmark Statistics: 1,741 questions require evidence from six agents over seven days and are distributed across categories, days, agents, and question types.Social Interaction and Task Coordination additionally include multi-span questions spanning non-contiguous temporal windows.
4 Benchmark Construction
The benchmark combines category-specific question generation with semantic multi-span synthesis or templates, then removes trivial and single-agent-solvable samples through model filtering and human verification.
- Question Generation: GPT-based generation uses open-ended sampling for Social Interaction, Task Coordination, and Theory of Mind, but templates for Temporal Reasoning and Environmental Interaction.
- Single-span Generation: Five-minute multi-agent windows supply captions and transcripts to GPT-4o for single-span questions in three categories.
- Multi-span Generation: Semantically similar QA pairs are grouped through embeddings and graph connectivity before GPT-5 synthesizes multi-span questions and answers.False options are generated separately; 15.9k Social Interaction and 16.3k Task Coordination multi-span samples were generated.
- Template-based Generation: Temporal Reasoning uses captions from 30-second, 10-minute, and 1-hour windows, while Environmental Interaction uses object- and interval-based templates.
- Filtering: LLM filtering removes questions answerable without context or from one agent’s memory, and cross-model validation checks correctness and non-triviality.Zero-shot filtering discards samples answered correctly in more than two of three trials.
- Human Verification: Four human verifiers review the remaining candidates using captions, transcripts, and videos from all agents.
5 Egocentric Video Reasoning in Multi-Agent System
EgoMAS addresses multi-agent egocentric reasoning with shared event memory and dynamic agent-wise retrieval. It retrieves relevant system-level context, generates agent-specific requests, filters agent memories, and conditions final responses on both context levels.
- EgoMAS is a training-free centralized multi-agent baseline designed for multi-agent egocentric reasoning.It integrates fragmented events across agents through shared memory and dynamically selects agent memories for fine-grained reasoning.
- Shared Memory Construction: A centralized manager converts ten-minute agent captions into event-based 4W1H shared memory.The representation records When, What, Where, Who, and How for key events across agents rather than producing a flat condensation.
- Agent-wise Dynamic Retrieval: Given a query, EgoMAS retrieves the top-n memories from shared memory using BM25 ranking.The BM25 score s(m, q) ranks memory m against query q.
- Agent-wise Dynamic Retrieval: EgoMAS generates agent-specific retrieval requests containing an agent identifier and sub-query, then searches each agent’s memory.This enables retrieval to adapt across individual agent perspectives.
- Agent-wise Dynamic Retrieval: A threshold τ filters low-scoring agent memories before final response generation.The response is conditioned on both retrieved system-level context and aggregated agent-level results.
6 Experimental Setup and Results
The experiments compare proprietary and open-source language and video models, retrieval baselines, single-agent baselines, and EgoMAS on MA-EgoQA. Current models perform poorly overall, while retrieval improves efficiency and EgoMAS achieves the strongest reported accuracy.
- Experimental Setup: The evaluation covers proprietary and open-source LLMs, video LLMs, retrieval baselines, single-agent baselines, and four EgoMAS backbones.Text-only models receive chronological captions, while video LLMs receive chronologically concatenated, uniformly sampled frames.
- Evaluation Results: 36.93% average accuracy is achieved by Gemini-2.5-Flash, the strongest model among the reported current-model results.Many baselines perform only marginally above random chance, underscoring the benchmark’s difficulty.
- Evaluation Results: All Frame Concat Baselines achieve the lowest average performance among baseline categories.The passage attributes this pattern to absent transcript information and many irrelevant frames.
- Evaluation Results: 41.41% accuracy is achieved by EgoMAS with Gemini-2.5-Flash, 4.48% higher than the Gemini-2.5-Flash baseline.EgoMAS also outperforms all baselines, while Qwen3VL-8B EgoMAS models surpass Gemini-2.5-Flash and GPT-5 in the reported comparison.
7 Analysis
Analysis shows that MA-EgoQA challenges models through multi-agent memory fusion, long-horizon retrieval, and latent-state inference, while EgoMAS improves efficiency and performance through retrieval and memory aggregation.
- Multi-agent reasoning: Restricting EgoMAS to one agent’s memory causes a substantial performance drop, confirming that MA-EgoQA requires multi-agent memory.
- Multi-agent reasoning: Performance decreases as the number of agents required by a query increases, indicating limitations in current multi-agent knowledge fusion.
- Long-horizon reasoning: Queries requiring multiple temporal spans achieve lower accuracy than single-timestamp queries, exposing long-horizon retrieval and event-linking difficulties.
- Category analysis: Theory of mind is consistently the lowest-accuracy category because it requires inferring latent goals and beliefs rather than extracting observable cues.
- EgoMAS efficiency: EgoMAS achieves the highest accuracy among retrieval-based models with 1.3 seconds per query, supporting its practical efficiency.
- EgoMAS ablations: Combining shared memory and agent-wise dynamic retrieval achieves the highest accuracy in the EgoMAS ablation.
- EgoMAS ablations: Event-based 4W1H memory outperforms alternative memory structures by abstracting and fusing events across agents.
- Case study: Excessive context and unaggregated agent memories cause competing baselines to miss relevant events, whereas EgoMAS locates them successfully.
8 Conclusion
The paper introduces MA-EgoQA for evaluating question answering over multiple long-horizon egocentric streams and proposes EgoMAS as a training-free baseline. Experiments show that current models struggle with this setting, especially theory of mind and multi-agent reasoning.
- MA-EgoQA benchmarks question answering over multiple long-horizon egocentric streams across five core multi-agent reasoning categories.
- EgoMAS combines shared memory with agent-wise dynamic retrieval as a simple training-free baseline.
- Current LLMs and video LLMs struggle with multi-agent egocentric understanding, particularly theory of mind and multi-agent reasoning.
Supplementary Materials
The supplementary materials add analyses, experimental details, limitations, benchmark-generation resources, and hyperparameter sensitivity results for EgoMAS.
- The supplement includes additional analyses, experimental setups, limitations and future work, benchmark details, prompts, question templates, and context-modality analysis.
- EgoMAS uses three hyperparameters: shared memory retrieval size n, agent-wise dynamic retrieval size k, and score threshold τ.
- Shared memory retrieval performs best around 20 to 60 items, while performance is robust to k and declines with a high τ.
A.2 Sensitivity Analysis to the Number of Available Agents
Sensitivity analysis finds that EgoMAS accuracy improves when more agent views are available, supporting the benchmark’s dependence on multi-agent information.
- Using more agents improves accuracy from 31.99% to 35.55%, demonstrating sensitivity to the number of available agent views.
- The results indicate that MA-EgoQA requires models to incorporate information from multiple agents.
C Limitation and Future Work
MA-EgoQA is limited by its reliance on a single multi-agent egocentric dataset, while future work targets broader evaluation and stronger retrieval. The benchmark’s supporting materials also specify category-focused question-generation procedures and perspective selection.
- Limitations: MA-EgoQA uses only EgoLife, limiting evaluation to one shared-house scenario despite its 266 hours of video from six people.The authors attribute this scope to EgoLife being the only publicly available dataset with long-term, simultaneously captured multi-agent egocentric videos.
- Future Work: EgoMAS remains about 42.4% below oracle performance, motivating stronger retrieval methods in future work.The authors specifically suggest hybrid retrieval combining text embeddings with lexical matching.
- Benchmark Construction: The question-generation materials require multi-person memories, five answer options, a single correct answer, timestamped rationales, and timestamp-name evidence lists.These requirements are stated for generating questions from captions and subtitles of multiple people.
- Category Design: Task Coordination prompts focus on goal-driven collaboration, including how people organize, contribute to, or discuss structured tasks.This category is explicitly separated from informal social interaction.
- Category Design: Theory-of-Mind prompts target questions about what people know, believe, see, or intend when mental states differ across individuals or from reality.The category instructions distinguish mental-state understanding from merely observing events.
H Analysis on Input Context Modality
Input modality affects MA-EgoQA performance unevenly. Visual frames consistently help WorldMM-8B, whereas EgoMAS benefits in some categories but declines in others, making selective modality use important.
- Cross-Model Findings: WorldMM-8B consistently improves when video frames accompany text captions, indicating that some MA-EgoQA questions require visual information.Examples include identifying object placement, grabbing tissues, brush changes, and being upstairs in another room.
- EgoMAS Findings: EgoMAS improves with video frames in SI, TR, and EI but drops in TC and ToM.The category-dependent pattern shows that additional frames are not uniformly beneficial for the model.
- Implications: The analysis recommends adaptive modality use and effective frame selection because unnecessary frames can distract the model.This issue is also observed in the All Frame Concat baselines.