PaperlayerView exact revision on arXiv

Source-linked AI summary

EgoLife: Towards Egocentric Life Assistant

Jingkang Yang, Shuai Liu, Hongming Guo, Yuhao Dong, Xiamengwei Zhang, Sicheng Zhang, Pengyun Wang, Zitang Zhou, Binzhu Xie, Ziyue Wang, Bei Ouyang, Zhengyu Lin, Marco Cominelli, Zhongang Cai, Yuanhan Zhang, Peiyuan Zhang, Fangzhou Hong, Joerg Widmer, Francesco Gringoli, Lei Yang, Bo Li, Ziwei Liu

arXiv:2503.03803v3cs.CV

TL;DR

Existing egocentric systems lack the long-duration, multimodal, multi-person context needed for personalized life assistance. EgoLife introduces a dataset, benchmark, and integrated EgoButler system, with evaluations showing strong egocentric performance and improved ultra-long-context QA through retrieval, while also exposing personalization and retrieval limitations.

  • Problem

    Existing egocentric datasets have short recording durations and predominantly monographic perspectives, limiting comprehensive habit and interpersonal-interaction coverage.

  • Method

    EgoLife collects a 300-hour multimodal, multiview dataset, builds EgoLifeQA, and combines EgoGPT with EgoRAG for personalized egocentric understanding and long-context question answering.

  • Results

    EgoGPT (EgoIT-99K+D1) achieves the highest average score across reported egocentric benchmarks, while EgoGPT+EgoRAG reaches 35.4 on queries spanning over 24 hours.

  • Takeaways & Limitations

    The released dataset, benchmark, and system provide resources for studying long-duration activities, interpersonal dynamics, multimodal understanding, and personalized assistance.

  • Takeaways & Limitations

    EgoGPT can misidentify people based on appearance and incompletely understand laughter and emotions, while EgoRAG lacks iterative multi-step reasoning and may fail when direct retrieval misses evidence.

Abstract

from arXiv · show

We introduce EgoLife, a project to develop an egocentric life assistant that accompanies and enhances personal efficiency through AI-powered wearable glasses. To lay the foundation for this assistant, we conducted a comprehensive data collection study where six participants lived together for one week, continuously recording their daily activities - including discussions, shopping, cooking, socializing, and entertainment - using AI glasses for multimodal egocentric video capture, along with synchronized third-person-view video references. This effort resulted in the EgoLife Dataset, a comprehensive 300-hour egocentric, interpersonal, multiview, and multimodal daily life dataset with intensive annotation. Leveraging this dataset, we introduce EgoLifeQA, a suite of long-context, life-oriented question-answering tasks designed to provide meaningful assistance in daily life by addressing practical questions such as recalling past relevant events, monitoring health habits, and offering personalized recommendations. To address the key technical challenges of (1) developing robust visual-audio models for egocentric data, (2) enabling identity recognition, and (3) facilitating long-context question answering over extensive temporal information, we introduce EgoButler, an integrated system comprising EgoGPT and EgoRAG. EgoGPT is an omni-modal model trained on egocentric datasets, achieving state-of-the-art performance on egocentric video understanding. EgoRAG is a retrieval-based component that supports answering ultra-long-context questions. Our experimental studies verify their working mechanisms and reveal critical factors and bottlenecks, guiding future improvements. By releasing our datasets, models, and benchmarks, we aim to stimulate further research in egocentric AI assistants.

1. Introduction

EgoLife addresses the lack of long-duration, multimodal, multi-person data for life-oriented egocentric assistance by introducing a 300-hour dataset, practical QA benchmark, and integrated system.

  • Existing egocentric datasets are limited by short recordings and predominantly monographic perspectives, restricting coverage of habits and social interactions.
  • Six participants lived together for one week, producing the 300-hour EgoLife dataset with egocentric multimodal video, synchronized third-person views, and rich annotations.The collection used Meta Aria glasses, 15 additional cameras, and two mmWave devices.
  • EgoLifeQA evaluates long-context, life-oriented assistance through tasks including item location, event recall, health-habit tracking, social-interaction analysis, and recommendations.Example questions concern misplaced scissors, water consumption, and later purchases or restocking.
  • EgoButler combines EgoGPT for egocentric omni-modal understanding with EgoRAG for ultra-long-context, week-level question answering.The system targets visual-audio integration, individual recognition and tracking, and retrieval over extensive temporal sequences.
  • The project releases the EgoLife dataset, EgoLifeQA tasks, and EgoButler system as a foundation for long-context understanding, multimodal integration, and personalized assistance.

2. Related Work

Egocentric vision research progressed from small foundational datasets to larger general-purpose collections and specialized resources, while video-language models increasingly address longer-duration content.

  • Early egocentric vision datasets established the field but were limited in scale, motivating larger collections such as EPIC-KITCHENS and Ego4D.
  • Specialized egocentric datasets target procedure learning, industrial tasks, collaboration, and ego-exo learning.
  • Video-language models evolved from feature extraction and pretraining toward instruction-following models with capabilities for longer-duration content.
  • Long-duration video methods commonly rely on video compression or extending model context length.

3. The EgoLife Dataset & Benchmark

EgoLife combines week-long multimodal data collection with a benchmark for long-term life-assistant question answering. Its annotation and curation pipeline converts synchronized recordings into practical, person- and habit-oriented QA tasks.

  • 3.1. Data Collection: Six volunteers recorded multimodal egocentric video in EgoHouse, supplemented by 15 GoPro cameras and millimeter-wave radars for synchronized third-person and spatial perspectives.Participants captured daily activities with Meta Aria glasses over seven days.
  • 3.1. Data Collection: The dataset spans discussions, rehearsals, cooking, decoration, shopping, and sightseeing, including an Earth Day party organized during the recording week.These activities provide varied everyday interactions across in-house and external settings.
  • 3.1. Data Collection: The collection protocol targeted informative coverage by requiring at least six hours of daily recording and managing periodic data uploads.Glasses were collected every three hours for upload and storage clearing, with participants limiting activities during that interval.
  • 3.4. Caption Annotations: Transcripts and captions were synchronized, speaker-diarized, densely narrated, converted into text, and enriched with visual frames and speech context.The annotation process included review and correction, producing time-aligned visual-audio captions.
  • 3.4. Caption Annotations: EgoLifeQA evaluates five life-assistant capabilities: entity logging, event recall, habit insight, relationship mapping, and task management.Questions are generated from visual-audio captions, filtered and refined by annotators, and finalized as multiple-choice QA with distractors.

4. EgoButler: Agentic Egocentric Life Assistant

EgoButler combines EgoGPT for continuous omni-modal egocentric video understanding with EgoRAG for hierarchical retrieval and long-context question answering. Its components support personalized responses across multimodal, identity-aware, and week-level temporal information.

  • System overview: EgoButler comprises EgoGPT for clip-level omni-modal understanding and EgoRAG for long-context question answering.
  • EgoLifeQA: EgoLifeQA contains 500 long-context questions per participant, totaling 3K questions, covering practical life-oriented assistance.Examples include breakfast recall and recommendations based on recent consumption.
  • System-I: EgoGPT: EgoGPT processes each 30-second clip using visual and audio inputs to generate continuous captions for understanding and retrieval.It also uses retrieved clues from EgoRAG to assist question answering.
  • System-I: EgoGPT: EgoGPT is fine-tuned with EgoIT-99K and personalized on EgoLife Day-1 video to support identity-aware questioning.The personalized model is EgoGPT (EgoIT-99K+D1), while EgoGPT (EgoIT-99K) is the non-personalized baseline.
  • System-II: EgoRAG: EgoRAG builds a structured, multi-level memory bank from continuously generated captions summarized hourly and daily.The memory stores clip features, textual descriptions, and timestamped summaries.
  • System-II: EgoRAG: EgoRAG first retrieves higher-level summaries to narrow the relevant time window, then selects top-k clips using visual-textual relevance.The scoring function balances visual and textual relevance with λ.
  • Integration and synergy: EgoButler combines efficient video interpretation with long-context memory to deliver personalized, context-aware responses.The hierarchical retrieval strategy is described as both contextually relevant and computationally efficient.

5. Experiments

The experiments evaluate EgoGPT and EgoRAG across egocentric understanding and long-context question answering, showing strengths in personalized multimodal processing and retrieval while exposing limitations in speech, identity recognition, and reasoning.

  • Main Results of EgoGPT: EgoGPT (EgoIT-99K+D1) achieves the highest average score across comparisons with commercial and open-source models on egocentric benchmarks.Its advantages are especially noted for RelationMap and omni-modal integration, although the tasks remain challenging.
  • The Effects of EgoRAG: EgoRAG retrieves evidence across video segments, improving long-context question answering when global context is needed.Without retrieval, segmenting ultra-long videos can produce hallucinations and incorrect answers because relevant clues may lie elsewhere.
  • The Effects of EgoRAG: 35.4 is the EgoGPT+EgoRAG score for queries spanning over 24 hours, outperforming both EgoGPT and Gemini-1.5-Pro.This result demonstrates the reported role of long-term retrieval for questions extending beyond individual segments.
  • Analysis of EgoGPT Variants: Combining visual and audio inputs yields the best performance among EgoGPT variants used to construct EgoRAG memory banks.Human captions perform best overall, audio-only models are weakest, and additional EgoLife Day-1 data consistently improves results.
  • Qualitative Results: EgoGPT provides personalized, contextually relevant captions but incompletely understands speech, laughter, emotions, and individual identities.Fine-tuning on EgoLife Day-1 can cause identity overfitting, such as confusing different people who later wear similar clothing.
  • Limitations and Future Improvements: EgoRAG uses a single-pass retrieval process and fails when direct supporting evidence is unavailable because it lacks multi-step reasoning and error tolerance.The reported improvement areas are speech comprehension, personalization, and more advanced retrieval reasoning.

6. Conclusion and Outlook

The conclusion presents EgoLife as a foundation for ultra-long egocentric video understanding and future multimodal AI assistants. Its recordings and sensing modalities support analysis of synchronized human behavior in shared everyday environments.

  • 6. Conclusion and Outlook: EgoLife combines ultra-long collaborative recordings of six participants with synchronized ego-exo views and multimodal signals for studying everyday human behavior.The dataset includes calibrated cameras, millimeter-wave radar, and WiFi alongside detailed annotations.
  • 6. Conclusion and Outlook: The project aims to inspire further research toward efficient, empathetic, and transformative multimodal AI companions.The authors describe EgoLife as an initial step toward AI that supports human life.

Contents of Supplementary Material

The supplementary material documents project execution, data collection and processing, ethical safeguards, and potential social impacts of EgoButler and EgoLifeQA.

  • Project Contributions: The supplementary material credits leadership and contributions across data collection, annotation, model development, EgoRAG construction, evaluation, and advising.It identifies project direction, dataset organization, annotation management, model training, and retrieval-system development responsibilities.
  • Ethical Protocols: The project reports permissions, institutional ethics review, informed consent, anonymization, sensitive-audio muting, screen blurring, and data-security measures.Faces, bystanders, license plates, passwords, and personal information were specifically addressed.
  • Potential Social Impact: EgoButler and EgoLifeQA could support daily activities, personal organization, contextual reminders, learning, task tracking, and skill development.The stated potential includes consistent support for elderly people and individuals with cognitive challenges.
  • Potential Social Impact: Privacy safeguards, transparency in AI decision-making, user autonomy, and trust are identified as essential considerations for positive societal reception.The supplementary discussion frames these requirements as important as egocentric AI integrates more deeply into personal spaces.

D. EgoLife Dataset Card

The EgoLife Dataset is an ultra-long, multiparticipant, multiview, and multimodal resource for studying daily human behavior and human–machine interaction. Its main session is fully annotated and synchronized, while an extension session follows the same protocol but is reserved for a separate release.

  • Dataset Description: The dataset contains ultra-long recordings from multiple participants captured through first-person and third-person perspectives with synchronized multimodal signals.It documents natural daily activities and supports behavior recognition, multimodal signal analysis, and human–machine interaction research.
  • Sessions: The main data session is fully annotated and synchronized, whereas the extension session follows the same capture protocol and will be released separately.The two sessions differ in temporal scale and annotation status.

D.1. Data Capturing

EgoLife includes two sessions with different temporal scales and annotation status, with the main session fully annotated and synchronized. The main session captures multi-day everyday activities, while the extension follows the same protocol for a separate release.

  • The main session records over 40 hours of daily activities across multiple consecutive days, primarily in a single dominant language.

D.2. Data Cleaning

The dataset was cleaned to improve data quality and protect participant privacy by removing or obscuring sensitive content.

  • Faces and license plates were blurred, while audio segments containing sensitive content were muted.

D.3. Dataset Composition

Each data instance combines first- and third-person video with synchronized radar and WiFi signals. Across two sessions, six participants contributed more than 40 hours in the main session and approximately 6 hours in the extension session.

  • Each instance includes first-person video from AI glasses and third-person video from fixed indoor cameras.
  • Synchronized multimodal signals include millimeter-wave radars and WiFi signals.
  • Video and signal fields support visual capture from both perspectives and radar/WiFi-based spatial and behavior correlation analysis.
  • More than 40 hours of multi-day activities form the main session, while the extension session contains approximately 6 hours from one day, across six participants.

D.4. Dataset Collection Process

The collection combines multimodal sensing, synchronization, privacy processing, annotation, and session-based organization during natural daily activities. Participants performed varied collaborative activities, including planning, shopping, cooking, rehearsals, decorating, games, outdoor exploration, and shared meals.

  • Collection setup: Six volunteers participated across both sessions, with diverse daily activities and interaction patterns recorded.
  • Collection setup: The equipment included head-mounted AI glasses, fixed indoor cameras, and millimeter-wave radars for first-person, third-person, spatial, and movement capture.
  • Collection setup: Participants performed typical daily activities, with natural interactions captured in various indoor settings.
  • Signal processing: Monostatic radar data were processed with CFAR detection to identify dynamic targets against background noise and static environmental signals.
  • Signal processing: WiFi transmitters and receivers collected channel state information independently using the AX-CSI platform.
  • Synchronization: Radar and WiFi signals were extracted and aligned with video, while multiple sources were synchronized using timestamps.
  • Annotation: Q&A annotations were initially generated with GPT, then refined by humans, alongside fine-grained and integrated activity and event labels.
  • Session organization: The main session spans multiple days in one dominant language, whereas the extension is single-day and multilingual.

G.1. Egocentric Datasets

Egocentric datasets have expanded from small, task-specific collections toward larger and more general daily-activity benchmarks, while adding specialized modalities and interaction settings. The EgoLife annotation pipeline combines multimodal scene descriptions with time-aligned object, action, visual, and audio information for reviewing and analyzing egocentric video.

  • EPIC-KITCHENS and Ego4D broadened egocentric research from kitchen action recognition toward larger-scale, more general daily-activity benchmarks.
  • Specialized datasets target procedure learning, collaborative tasks, multiview learning, hand-object interaction, gesture recognition, and hand-pose estimation.
  • EgoLife complements egocentric recordings with synchronized third-person video and millimeter-wave data collected across a shared living environment.
  • Each SRT entry aligns a time interval with protagonist actions, interactive objects, and merged visual-audio captions that combine visual, speech, and contextual information.The annotations are stored in SRT format so reviewers can overlay them on video for alignment and verification.
Loading 2503.03803v3…