Source-linked AI summary
Agentic Very Long Video Understanding
Aniket Rege, Arka Sadhu, Yuliang Li, Kejie Li, Ramya Korlakai Vinayak, Yuning Chai, Yong Jae Lee, Hyo Jin Kim
TL;DR
Very long egocentric video understanding requires systems that can interpret and recall continuous visual and audio experience across days or weeks despite limited context windows and weak multi-hop reasoning. EGAgent addresses this with temporally structured entity scene graphs and agentic visual and audio search, achieving state-of-the-art EgoLifeQA performance and competitive Video-MME (Long) results.
Problem
Always-on personal AI assistants need to interpret and recall continuous egocentric video over days or weeks, but existing methods face context-window and compositional multi-hop reasoning limitations.
Method
EGAgent equips a planning agent with a temporally structured entity scene graph and visual, audio, and graph-search tools for cross-modal reasoning.
Results
EGAgent achieves state-of-the-art performance on EgoLifeQA and competitive performance on Video-MME (Long).
Takeaways & Limitations
Structured representations of inter-entity relationships support detailed and temporally coherent analysis of longitudinal video understanding tasks.
Takeaways & Limitations
Entity-scene-graph quality depends on upstream perception and language models, while transcript-based experiments used manually annotated speaker diarization.
Abstract
from arXiv · showhide
The advent of always-on personal AI assistants, enabled by all-day wearable devices such as smart glasses, demands a new level of contextual understanding, one that goes beyond short, isolated events to encompass the continuous, longitudinal stream of egocentric video. Achieving this vision requires advances in long-horizon video understanding, where systems must interpret and recall visual and audio information spanning days or even weeks. Existing methods, including large language models and retrieval-augmented generation, are constrained by limited context windows and lack the ability to perform compositional, multi-hop reasoning over very long video streams. In this work, we address these challenges through EGAgent, an enhanced agentic framework centered on entity scene graphs, which represent people, places, objects, and their relationships over time. Our system equips a planning agent with tools for structured search and reasoning over these graphs, as well as hybrid visual and audio search capabilities, enabling detailed, cross-modal, and temporally coherent reasoning. Experiments on the EgoLifeQA and Video-MME (Long) datasets show that our method achieves state-of-the-art performance on EgoLifeQA (57.5%) and competitive performance on Video-MME (Long) (74.1%) for complex longitudinal video understanding tasks. Code is available at https://github.com/facebookresearch/egagent.
1 Introduction
EGAgent targets very long, continuous egocentric video understanding by combining temporally structured entity graphs with agentic cross-modal search and reasoning. It achieves state-of-the-art performance on EgoLifeQA while supporting structured reasoning over extended time horizons.
- Motivation: Very long video understanding extends beyond short clips to more than 50 hours of continuous egocentric video recorded across a week.EgoLife contains longitudinal first-person video from six individuals, enabling research on cross-day entity tracking, repeated behaviors, and inactivity periods.
- Method: EGAgent represents people, places, and objects as graph nodes, with temporally annotated edges encoding relationships such as interaction, use, and speech.The representation supports structured reasoning over relationships that hold during specific time intervals.
- Method: The framework combines entity-graph search and reasoning with hybrid visual search and audio-transcript search through a planning agent.This combination supports cross-modal, compositional queries spanning multiple days.
- Results: 32% and 39.7% improvements over the previous state of the art were achieved on EgoLifeQA’s RelationMap and TaskMaster categories, respectively.Both categories require multi-hop relational reasoning.
- Results: The paper also reports an ablation study of entity-graph construction and agentic tool usage for very long video understanding.The study is conducted on EgoLife and includes Appendix D analyses.
2 Related Work
Prior long-video methods address context limits through frame or token condensation, while RAG retrieves external evidence but can lose relational and temporal structure. EGAgent instead integrates a time-aware entity graph into agentic tool use for efficient longitudinal reasoning.
- Long Video Understanding with LLMs: Long-video methods reduce context demands by selecting salient frames or compressing visual tokens, using either query-dependent or query-independent strategies.These approaches aim to fit video information within language-model context limits.
- Video Understanding with Graph-based RAG: Traditional RAG retrieves relevant information but often loses relational context because it operates over isolated text chunks.Graph-based RAG methods address this by organizing extracted entities and relationships into knowledge graphs.
- Video Understanding with Graph-based RAG: EGAgent’s entity graph annotates nodes with temporal information, allowing time-aware representation and incremental construction as new data arrives.This design distinguishes it from approaches that overlook temporal relationships or build one graph for an entire video at once.
- Video Understanding with Graph-based RAG: EGAgent matches AdaVideoRAG on Video-MME (Long) while processing more than ten times fewer frames.The comparison highlights an efficiency result reported for the long-video benchmark.
- Agentic Video Understanding: EGAgent integrates a temporally annotated entity scene graph into the agent’s tool-calling loop for cross-modal search and compositional reasoning.The approach differs from systems relying on unstructured captions or repeated frame retrieval.
3 Method
EGAgent addresses very long video understanding by combining an entity scene graph with an agent that plans multi-step, cross-modal retrieval and reasoning. The framework represents temporally grounded entities and relationships, searches visual, audio, and graph data, and synthesizes accumulated evidence into an answer.
- Entity Graph Representation: The entity graph supports questions about relationships, repeated behaviors, and interactions that naive retrieval methods do not explicitly model over extended periods.Temporal edges encode the existence, sequence, and duration of relationships, supporting reasoning across long horizons.
- Entity Graph Representation: EGAgent represents people, objects, and locations as graph nodes, with relationship edges carrying temporal information for long-horizon reasoning.The graph is extracted from audio transcripts, scene descriptions, and predicted scene locations, then aggregated into a structured representation.
- Agentic Framework: A planning agent decomposes each query into sub-tasks and selects visual, audio-transcript, or entity-graph retrieval tools for each step.Sub-tasks and tool selections are stored in the framework state, while graph extraction uses an LLM over multimodal text documents.
- Agentic Framework: Retrieved data is analyzed for relevance and evidence, accumulated in working memory, and passed to a VQA agent for final cross-modal synthesis.Visual search uses hybrid semantic and attribute search, audio uses transcript search, and entity-graph retrieval uses SQL queries.
- Agentic Framework: The framework comprises a planning agent, three retriever tools, an analyzer tool, and a VQA agent for multimodal reasoning.The VQA agent conditions on the query and compact evidence accumulated in memory to generate the final answer.
4 Experiments
Experiments evaluate EGAgent on EgoLifeQA and Video-MME (Long), showing strong gains from entity-graph reasoning across benchmarks, categories, and backbones.
- EgoLifeQA contains 500 MCQs from 50 hours of egocentric video, while Video-MME (Long) contains 300 videos lasting 30–60 minutes.
- EGAgent achieves a new state-of-the-art on EgoLifeQA, with entity-graph reasoning improving Gemini 2.5 Pro by 10.7%.
- EGAgent consistently surpasses same-backbone agentic methods, exceeding EgoButler by 8.4%, VideoAgent by 15.4%, and Ego-R1 by 8.6%.
- On RelationMap and TaskMaster, EGAgent surpasses Gemini 2.5 Pro by 20.8% and 22.2%, respectively, on categories requiring multi-hop relational reasoning.
- On Video-MME (Long), EGAgent surpasses Video-RAG by 4.5% and matches AdaVideoRAG while processing over 10× fewer frames.
- Adding visual captions to entity-graph extraction increases MCQ accuracy by approximately 2.6% across GPT-4o, GPT-4.1, and Gemini 2.5 Pro.
- EGAgent answers an MCQ in two to three minutes, while BM25 search reduces token usage 3.3× at the cost of an approximately 6.8% accuracy drop.
5 Conclusion
The conclusion presents EGAgent as a framework for longitudinal video understanding that combines temporally structured entity graphs with agentic, cross-modal tools.
- EGAgent addresses always-on assistants’ challenges in processing very long egocentric video streams through entity scene graphs and specialized reasoning tools.
- Experiments on EgoLifeQA and Video-MME (Long) demonstrate state-of-the-art performance on tasks tracking entities, behaviors, and relationships over extended periods.
- The results highlight the potential of agentic planning over structured inter-entity relationships for very long video understanding as video lengths grow.
6 Limitations
EGAgent’s entity scene graph depends on upstream perception and language models, whose errors can affect extracted entities and relationships. The experiments also rely on transcripts and manually annotated speaker diarization for EgoLife.
- Entity scene graph construction depends on upstream perception and language models that may occasionally misextract entities and relationships.
- The experiments use transcripts and manually annotated speaker diarization for EgoLife.
- Off-the-shelf diarization models may introduce prediction errors that adversely affect downstream performance.
7 Ethical Considerations
The paper reports compliance with the licenses and terms of use for the EgoLife and Video-MME datasets. It also documents design, retrieval, analysis, and implementation details in supplementary materials.
- The authors state that they adhere to the MIT license and terms of use for the publicly available EgoLife dataset.
- The authors state that they use Video-MME under its custom license, unmodified and strictly for academic research.
- Appendix B provides EGAgent design details and a visual pipeline walkthrough with a qualitative example.
- Appendix F provides prompts and code snippets for planning, entity-graph extraction, temporal annotation, and search-tool queries.
C Entity Graph
The entity graph represents temporally localized relationships and supports structured retrieval within EGAgent’s pipeline. Its extracted EgoLife graph contains 13,968 relationships over seven days, primarily involving person sources.
- Entity-graph edges receive temporal annotations that enable EGAgent to localize relevant relationships for a query.
- 13,968 relationships are extracted over a 7-day period, with 13,930 having a person as the source node.
- The graph stores person, location, object, and person target nodes, emphasizing person-person and person-object interactions.
- For a sub-task, the planning agent selects a SQL query using entity types, relationship type, and a time interval within a strict-to-relaxed hierarchy.
- The tool ablation evaluates combinations of entity-graph, visual, and audio search tools, highlighting cross-modal reasoning across EgoLifeQA task types.
D Ablation Study on EgoLife
The EgoLife ablation analysis examines tool usage, oracle upper bounds, retrieval accuracy, and the EGAgent pipeline’s wall-clock latency. It also studies how individual tools affect performance.
- The ablation study evaluates the impact of tool usage on EgoLifeQA performance.
- Oracle experiments estimate upper-bound performance for EGAgent on EgoLife.
- The analysis measures search-tool retrieval accuracy, wall-clock latency, and pipeline behavior.
D.1 Ablation on tool usage
The ablation shows that different retrieval tools support different task types, while entity-graph search is crucial for robust multi-modal reasoning. Combining entity-graph, visual, and audio search yields the strongest broad performance.
- Tool contributions: Frame-only search performs poorly on entity-centric RelationMap but remains strong on visual-focused EntityLog and EventRecall.RelationMap reaches 28%, compared with 40% on EntityLog and 37.3% on EventRecall.
- Tool contributions: +13.1% on HabitInsight, +13.6% on RelationMap, and +31.7% on TaskMaster follow adding audio transcript search to frame search.EntityLog decreases slightly by 3.2%, showing that audio primarily benefits audio- and relation-dependent tasks.
- Tool contributions: Adding visual search to audio search improves EntityLog by 15.2% and EventRecall by 5.5% but reduces HabitInsight, TaskMaster, and RelationMap.The decreases are 3.3%, 4%, and 5%, respectively, relative to audio search alone.
- Tool contributions: Entity-graph search produces state-of-the-art RelationMap, TaskMaster, and EventRecall performance while remaining competitive on EntityLog and HabitInsight.Relative to EgoButler, gains are 23.2% on RelationMap, 31.8% on TaskMaster, and 11.9% on EventRecall.
- Oracle comparison: Oracle visual and audio transcript retrieval outperforms EGAgent by 6.9% with GPT 4.1 and 11.2% with Gemini.The oracle uses ground-truth relevant moments for retrieval.
D.3 Retrieval Accuracy
Retrieval quality and temporal localization are central to EGAgent’s performance. The combined tools provide strong fine-grained recall, while each individual modality has distinct temporal coverage and cost characteristics.
- Recall methodology: Recall@W measures whether any selected timestamp falls within a temporal window W around a ground-truth timestamp across 500 EgoLifeQA questions.Windows range from 10 seconds to one hour to assess recall as temporal strictness is relaxed.
- Tool retrieval behavior: Visual search shows strong recall at a 10-second window, but visual-only EGAgent reaches only 34.6% MCQ accuracy.Even perfect audio-visual retrieval saturates at 68.7% accuracy, indicating retrieval is not the only constraint.
- Tool retrieval behavior: Audio transcript search has poor fine-grained recall because relevant timestamps can be omitted when transcript evidence is temporally ambiguous.Despite this, an audio-transcript oracle is 21% better than the previous state of the art.
- Tool retrieval behavior: Entity-graph search has the weakest recall at windows of 2 minutes or less but surpasses audio transcript search at larger windows.Its broader temporal coverage reflects its lower-dimensional projection of the audio-visual space.
- Combined retrieval: The combined EGAgent tools achieve recall of 0.88 at a 10-second window, supporting temporal localization as a contributor to EgoLifeQA performance.The strict-to-relaxed strategy broadens temporal windows when initial matches fail.
- Efficiency: The analyzer dominates visual and audio-search latency, with visual analysis taking about 9.1 times longer than actual retrieval.Visual analysis processes 50 retrieved frames.
- Efficiency: The entity graph is lightweight at about 2 MB for roughly 52 hours of EgoLife video, whereas the visual embedding database is much larger.The supplied passage contrasts graph and visual-index storage without providing the complete embedding size.
- Graph quality: Entity-graph relationships achieve 94% accuracy in a manual audit of 100 sampled relationships.Failures generally involve subtle visual-captioning or caption-transcript fusion errors.
E Failure Cases
EGAgent’s failures arise from identity ambiguity, cross-modal conflicts, incomplete relational graph coverage, and ambiguous questions. These cases expose limits in grounding and evidence reconciliation.
- Identity attribution: Identity attribution can fail even when EGAgent localizes the relevant moment, especially when degraded visual signals conflict with concurrent audio.Persistent persona representations are suggested as a possible robustness improvement.
- Cross-modal conflict: Conflicting audio and visual evidence can lead EGAgent to privilege prior verbal intent over more reliable visual observations.Cross-modal consistency checks and prioritizing direct evidence are proposed mitigations.
- Graph incompleteness: The entity graph can underrepresent social or relational information relative to object interactions, leaving EGAgent unable to reconcile some cases.Richer agent and interaction representations are proposed to address missing relational attributes.
- Question ambiguity: Question ambiguity can leave multiple answers plausible, as frequent pizza consumption conflicts with positive feedback about Tremella soup.The example concerns inference of Katrina’s food preferences.
F Implementation Details
EGAgent is implemented as a LangGraph StateGraph that plans multi-step queries, routes them to visual, audio, or entity-graph tools, and accumulates evidence before answering. Its graph is extracted from fused audio-visual representations with temporally annotated relationships.
- Agent orchestration: EGAgent converts each multiple-choice question into a VeryLongVideoQA StateGraph containing the agent’s inference state.Reported accuracies come from a single run because repeated agent execution is computationally prohibitive.
- Agent orchestration: A planner decomposes each query into sub-tasks and assigns visual, entity-graph, or audio-transcript search tools to each step.The current sub-task and selected tools are stored in the graph state.
- Agent orchestration: A dispatch node executes selected tools, updates working memory, and either completes the step or returns control to the planner.Completion routes to answer generation, while incomplete plans return to planning.
- Implementation: The implementation uses LangGraph components including planner, routing, tool-search, step-completion, and answer-generation nodes.The graph routes visual, entity-graph, and audio searches back through the planner before final generation.
- Entity graph construction: The entity graph is built by fusing GPT-4.1 visual captions and audio transcripts generated at 30-second intervals.An LLM graph transformer extracts candidate entities and relationships from the fused captions.
- Entity graph construction: The system annotates extracted graph relationships with timestamps to support temporal filtering during search.The entity-graph search tool queries time, keywords, source and target entities, and relationship type.
- Entity graph construction: Relationships use coarse categories such as interacts with, uses, talks to, and mentions because finer relation types fragment retrieval and reduce recall.Coarser abstractions improve matching across noisy ASR and caption signals while retaining information for temporal localization and multi-hop reasoning.