PaperlayerView exact revision on arXiv

Source-linked AI summary

EgoLife: Towards Egocentric Life Assistant

Jingkang Yang, Shuai Liu, Hongming Guo, Yuhao Dong, Xiamengwei Zhang, Sicheng Zhang, Pengyun Wang, Zitang Zhou, Binzhu Xie, Ziyue Wang, Bei Ouyang, Zhengyu Lin, Marco Cominelli, Zhongang Cai, Yuanhan Zhang, Peiyuan Zhang, Fangzhou Hong, Joerg Widmer, Francesco Gringoli, Lei Yang, Bo Li, Ziwei Liu

arXiv:2503.03803v3cs.CV

TL;DR

Existing egocentric datasets offer relatively short recordings and mostly monographic perspectives, limiting coverage of long-term behavior and social interaction. EgoLife introduces a 300-hour daily-life dataset, a long-context QA benchmark, and the EgoButler system; personalized EgoGPT achieves the highest average score across egocentric benchmarks, while EgoGPT+EgoRAG scores 35.4 on queries spanning over 24 hours.

  • Problem

    Existing egocentric datasets have relatively short recordings and mostly monographic perspectives, with limited coverage of long-term behavior patterns and social interaction.

    Source: §1 · p. 2

  • Method

    The paper collects a 300-hour multimodal, multi-view daily-life dataset and introduces EgoLifeQA and EgoButler, pairing egocentric EgoGPT with retrieval-based EgoRAG for long-context questions.

    Source: §1 · p. 2

  • Results

    Personalized EgoGPT achieves the highest average score across egocentric benchmarks; for queries spanning over 24 hours, EgoGPT+EgoRAG scores 35.4, outperforming EgoGPT and Gemini-1.5-Pro.

    Source: §5 · p. 8

  • Takeaways & Limitations

    EgoLife provides a unique opportunity to study synchronized human behaviors and everyday dynamics in shared environments.

    Source: §6 · p. 9

    EgoRAG uses single-pass retrieval without multi-step reasoning, so it can fail when relevant information is missing from direct retrieval.

    Source: §5 · p. 9

Abstract

from arXiv · show

We introduce EgoLife, a project to develop an egocentric life assistant that accompanies and enhances personal efficiency through AI-powered wearable glasses. To lay the foundation for this assistant, we conducted a comprehensive data collection study where six participants lived together for one week, continuously recording their daily activities - including discussions, shopping, cooking, socializing, and entertainment - using AI glasses for multimodal egocentric video capture, along with synchronized third-person-view video references. This effort resulted in the EgoLife Dataset, a comprehensive 300-hour egocentric, interpersonal, multiview, and multimodal daily life dataset with intensive annotation. Leveraging this dataset, we introduce EgoLifeQA, a suite of long-context, life-oriented question-answering tasks designed to provide meaningful assistance in daily life by addressing practical questions such as recalling past relevant events, monitoring health habits, and offering personalized recommendations. To address the key technical challenges of (1) developing robust visual-audio models for egocentric data, (2) enabling identity recognition, and (3) facilitating long-context question answering over extensive temporal information, we introduce EgoButler, an integrated system comprising EgoGPT and EgoRAG. EgoGPT is an omni-modal model trained on egocentric datasets, achieving state-of-the-art performance on egocentric video understanding. EgoRAG is a retrieval-based component that supports answering ultra-long-context questions. Our experimental studies verify their working mechanisms and reveal critical factors and bottlenecks, guiding future improvements. By releasing our datasets, models, and benchmarks, we aim to stimulate further research in egocentric AI assistants.

1. Introduction

EgoLife addresses the limits of short, mostly single-person egocentric datasets by combining a long-duration multimodal dataset, life-oriented QA tasks, and an integrated assistant system. Together, these contributions target week-level understanding, interpersonal dynamics, and personalized daily-life assistance.

Motivation

  • Existing datasets such as Epic-Kitchen [4] and Ego4D [5] have relatively short recordings and predominantly single-person perspectives, limiting coverage of habits and social dynamics.The paper argues that addressing these gaps requires extended activities, multimodal data, and multiple perspectives.Source: §1 · p. 2

EgoLife dataset

  • Six participants recorded about eight hours daily for one week, producing EgoLife’s 300-hour multimodal dataset with synchronized third-person views from 15 cameras and two mmWave devices.The egocentric recordings used Meta Aria glasses [1], and the dataset includes audio transcripts and visual-audio narrations at multiple time granularities.Source: §1 · p. 2

EgoLifeQA benchmark

  • EgoLifeQA introduces long-context, life-oriented QA tasks for locating items, recalling events, tracking health habits, analyzing social interactions, and making timely recommendations.Its questions are designed to support personalized, context-aware assistance in everyday activities.Source: §1 · p. 2

EgoButler system

  • EgoButler addresses multimodal understanding, person recognition and tracking, and week-level QA through EgoGPT and the retrieval-augmented EgoRAG module.EgoGPT is a lightweight personalized vision-audio-language model fine-tuned on egocentric datasets; evaluations identify key factors and bottlenecks.Source: §1 · p. 2

Project overview

  • The project aims to lay a foundation for life-oriented AI research and plans to broaden dataset coverage across languages, locations, and activities while developing more sophisticated models.The introduction frames the goal as AI glasses that support everyday life through long-context understanding, multimodal integration, and personalized assistance.Source: §1 · p. 2

2. Related Work

Prior egocentric datasets and video-language models have expanded in scale, task coverage, and duration, but week-long multiperson behavior and ultra-long egocentric video remain open challenges addressed by EgoLife.

2.1. Egocentric Datasets & Benchmarks

  • Egocentric datasets grew from foundational collections [24–26] to larger-scale datasets and long-term benchmarks, while multiperson social dynamics and over-day habits remain underexplored.Specialized datasets address procedure learning [27] [28], collaboration [29], and multiview understanding [9], while recent benchmarks cover temporal understanding [11] and planning. EgoLife adds week-long multiperson data with multimodal sensing, multiview perspectives, and detailed annotations.Source: §2.1 · p. 3

2.2. Long-Context Video Language Models

3. The EgoLife Dataset & Benchmark

EgoLife combines a week of multimodal recordings from six participants with detailed transcripts, captions, and long-context questions designed for practical life-assistant tasks.

3.1. Data Collection

  • Six volunteers lived and recorded daily activities for seven days using Meta Aria glasses, 15 GoPros, and millimeter-wave radars.Each pair of glasses recorded at least six hours per day; activities included Earth Day party preparation, cooking, shopping, and sightseeing.Source: §3.1 · pp. 3–4

3.2. Data Cleaning; 3.3. Transcript Annotations

  • Cleaning supported synchronization and privacy, while transcript annotation refined synchronized audio into participant-specific, speaker-attributed speech tracks.The initial 50-hour transcript was generated with speech recognition and diarization, then reviewed and split into six tracks retaining speech audible to each participant.Source: §3.2 · p. 4, §3.3 · p. 4

3.4. Caption Annotations

  • Annotators narrated five-minute clips, and the resulting phrases were merged and expanded into human-verified visual-audio captions aligned with video and speech.The pipeline converted 361K brief narrations into 25K merged captions, then used representative frames and transcripts to add visual and speech context.Source: §3.4 · pp. 4–5

3.5. EgoLifeQA Annotations

  • EgoLifeQA covers five assistant capabilities: entity logging, event recall, habit insight, relationship understanding, and task assignment from prior actions.The question types target object details, past events, personal habits, interpersonal interactions and identification, and reminders based on previous actions.Source: §3.5 · p. 5
  • Annotators retained questions requiring at least five minutes of look-back, reducing roughly 100K candidates per participant to 500 curated multiple-choice questions each.Questions were AI-generated from visual-audio captions, then filtered and revised; annotators added distractors and labeled audio requirements and necessary look-back time.Source: §3.5 · p. 5

4. EgoButler: Agentic Egocentric Life Assistant

EgoButler combines EgoGPT for personalized clip-level audiovisual understanding with EgoRAG for long-context retrieval, enabling context-aware answers to questions about daily life.

4. EgoButler: Agentic Egocentric Life Assistant

  • EgoButler pairs EgoGPT (System-I) for clip-level omni-modal understanding with EgoRAG (System-II) for long-context question answering.Together, the systems combine efficient video interpretation with long-context memory to provide relevant, context-aware responses.Source: §4 · pp. 5–6

4.1. System-I: EgoGPT for Clip Understanding

  • EgoGPT captions each 30-second clip from visual and audio inputs and answers questions with retrieved EgoRAG clues; Day-1 video fine-tuning enables identity-aware questioning in EgoLifeQA.EgoIT-99K contains QA pairs derived from video annotations and is used for final-stage fine-tuning. Its audio branch encodes audio with Whisper Large v3; EgoGPT (EgoIT-99K+D1) is the personalized version, while EgoGPT (EgoIT-99K) is the non-personalized baseline.Source: §4.1 · p. 6

4.2. System-II: EgoRAG for Long-Context Q&A

  • EgoRAG builds a multi-level memory from clip captions and hourly and daily summaries, then narrows retrieval from day to hour before selecting relevant clips.A relevance score balances visual and textual relevance; a language model uses the retrieved content to generate a response, supporting contextually relevant and computationally efficient answers.Source: §4.2 · pp. 6–7

4.3. Integration and Synergy in EgoButler

  • EgoGPT continuously gathers personalized egocentric data while EgoRAG retrieves relevant clues, combining video interpretation and long-context memory for accurate, context-aware responses.Source: §4.3 · p. 7

5. Experiments

EgoGPT (EgoIT-99K+D1) achieves the highest average score across the evaluated egocentric benchmarks, while EgoRAG improves long-context question answering by retrieving evidence across video segments. Experiments also show that caption quality matters and identify limitations in speech understanding, personalization, and retrieval reasoning.

The Effects of EgoRAG

  • 35.4 is the score EgoGPT+EgoRAG achieves for queries spanning over 24 hours, outperforming both EgoGPT and Gemini-1.5-Pro.Retrieving evidence across segments mitigates errors that arise when models answer within isolated 30-minute segments lacking global context.Source: Table 6 · p. 7, §5 · p. 8

Analysis of EgoGPT Variants

  • Human-annotated captions yield the highest scores; among EgoGPT variants, audio-only performs weakest, visual-only better, and combined visual-audio inputs best, with Day-1 captions improving results.These findings indicate that audio alone is inadequate for EgoLifeQA and that participant-specific information and domain-specific generalization matter.Source: Table 7 · p. 8, §5 · p. 8

Main Results of EgoGPT

  • EgoGPT (EgoIT-99K+D1) achieves the highest average score across egocentric benchmarks, with identity recognition and omni-modal integration distinguishing it from general-purpose commercial models.The tasks remain challenging, with substantial room for improvement.Source: §5 · p. 8

Qualitative Results

  • EgoGPT (EgoIT-99K+D1) produces personalized, context-relevant captions, but struggles with laughter and emotions and may overfit identity recognition to Day-1 appearance.The qualitative comparison reports fewer hallucinations than Gemini-1.5-Pro, while the Day-1 fine-tuning example shows a person in a blue shirt later being mistaken for someone else in blue.Source: Figure 8 · p. 8, §5 · p. 9
  • EgoRAG uses a single-pass retrieval without iterative refinement or step-by-step reasoning, so it can fail when direct retrieval misses evidence and lacks error resilience.The paper identifies improved speech comprehension, personalization, and retrieval reasoning as areas for immediate improvement.Source: §5 · p. 9

6. Conclusion and Outlook

EgoLife is presented as a dataset and benchmark advancing ultra-long egocentric video understanding, with recordings and multimodal resources for studying everyday human behavior. The authors frame it as a foundation for further research toward multimodal AI assistants.

6. Conclusion and Outlook

  • EgoLife is a pioneering dataset and benchmark that marks a significant step toward ultra-long egocentric video understanding.The authors describe its potential as extending beyond its debut focus.Source: §6 · p. 9
  • Recordings of six participants, ego-exo alignment, calibrated cameras, multimodal signals, and detailed annotations enable study of synchronized behavior and everyday dynamics.The signals include millimeter-wave radar and WiFi, supporting modeling human life across temporal and spatial contexts.Source: §6 · p. 9
  • The authors hope EgoLife will inspire further research toward AI assistants that are efficient, empathetic, and transformative companions in human life.Source: §6 · p. 9

Contents of Supplementary Material … C. Potenial Social Impact

The supplementary material documents contributors’ roles across EgoLife’s development, outlines privacy and consent safeguards, and describes the potential benefits and ethical conditions of egocentric AI assistance.

A. Authorship Statement

  • The project lead oversaw EgoLife from conception through execution, coordinating casting and data collection as well as data preparation, modeling, evaluation, writing, and presentation.Jingkang Yang is identified as project lead and director of the initiative.Source: A. Authorship Statement · p. 17
  • Contributors planned casting and collection, set up and calibrated cameras, supported collaborative first- and third-person recording, and collected mmWave and Milan-subset data.The Milan subset also included WiFi collection infrastructure; contributors supported the associated data-collection setup.Source: A. Authorship Statement · p. 17
  • The team organized raw data and annotations, managed captioning and EgoLifeQA, cleaned and synchronized multiview recordings, and prepared audio transcripts and VRS data.Work also included desensitization, diarization preannotation, and reconstructing a 3D model of the EgoLife house for demonstrations.Source: A. Authorship Statement · pp. 17–18
  • Yuhao Dong and Shuai Liu led model training; Hongming Guo, Shuai Liu, and Sicheng Zhang developed EgoRAG and defined evaluation protocols, with advisors supporting training and research direction.The team integrated EgoSchema, EgoPlan, and other elements into LMMs-Eval; advising also covered long-context video language models and 3D vision.Source: A. Authorship Statement · p. 18

B. Ethical Considerations

  • Filming locations had owner permission, the institutional ethics committee reviewed and approved collection, and participants gave informed consent after project briefings.The authors state that prior communications and written or verbal agreements documented permission for locations, including the villa.Source: B. Ethical Considerations · p. 18
  • Privacy measures blurred participant and bystander faces, identifiers, and license plates, muted sensitive audio, masked private screen content, and secured and anonymized raw data.The authors state that these measures were used to protect participant identities and prevent unauthorized access.Source: B. Ethical Considerations · p. 18

C. Potenial Social Impact

  • EgoButler and EgoLifeQA could support daily activities, personal organization, contextual reminders, and learning or professional task tracking, including for people needing consistent support.The paper identifies elderly people and those with cognitive challenges as examples of people who may benefit from consistent support.Source: C. Potenial Social Impact · p. 18
  • Egocentric AI may adapt assistance for education and work, but privacy safeguards and transparent decision-making are important to user autonomy, trust, and responsible human-AI interactions.The paper frames real-time, context-aware AI companions as a potential societal benefit while emphasizing privacy-conscious interaction.Source: C. Potenial Social Impact · pp. 18–19

D. EgoLife Dataset Card

EgoLife is an ultra-long, multiparticipant daily-life video dataset combining first- and third-person recordings with synchronized multimodal signals. It supports research on human behavior, multimodal analysis, and human–machine interaction, with a fully annotated main session and a separately released extension session.

D. EgoLife Dataset Card

  • EgoLife records natural daily activities from first- and third-person perspectives and enriches the videos with synchronized multimodal signals.The recordings are ultra-long and involve multiple participants.Source: D. EgoLife Dataset Card · p. 19
  • The dataset supports research on human behavior recognition, multimodal signal analysis, and human–machine interaction.Source: D. EgoLife Dataset Card · p. 19
  • EgoLife has a fully annotated and synchronized main session and an extension session collected using the same protocol for a separate dataset release.Source: D. EgoLife Dataset Card · p. 19

D.1. Data Capturing

EgoLife’s data-capturing design reflects everyday behavior and supports multimodal synchronization under real-world conditions through two sessions with comparable activity structures but different durations and language compositions.

D.1. Data Capturing

  • The dataset includes a continuous main session of over 40 hours, primarily in one language, and an approximately 6-hour multilingual extension with similar activity types and interaction patterns.The curation emphasizes signal-based behavior modeling and multimodal synchronization in real-world conditions.Source: §D.1 · p. 19

D.2. Data Cleaning

The dataset was cleaned to improve quality and remove sensitive content.

D.2. Data Cleaning

  • All identifiable faces and license plates were blurred, while audio containing sensitive content was muted.The cleaning also removed low-quality segments.Source: §D.2 · p. 19

D.3. Dataset Composition

Each EgoLife data instance combines first-person and fixed-camera third-person video with synchronized radar and WiFi signals. The dataset includes six participants recorded across a main multi-day session and a single-day extension.

D.3. Dataset Composition

  • Each instance includes AI-glasses first-person video and fixed-camera third-person video.The video fields capture primary visual data from both perspectives.Source: §D.3 · p. 19
  • Synchronized millimeter-wave radar and WiFi signals provide fields for spatial and behavior correlation analysis.Source: §D.3 · p. 19
  • Six participants were recorded across two sessions: a main multi-day session of more than 40 hours and a single-day extension of approximately 6 hours.Source: §D.3 · p. 19

D.4. Dataset Collection Process … G. History of Egocentric Datasets

The sections describe how EgoLife data were collected, synchronized, annotated, and organized, alongside the week’s activities and the egocentric datasets used to build EgoIT.

D.4. Dataset Collection Process

  • Six volunteers took part across the main and extension sessions, using first-person glasses alongside indoor cameras and radar sensing to record daily activities and interactions.Radar data were captured as raw ADC signals and processed with CFAR to detect dynamic targets against background and static environmental signals.Source: §D.4 · pp. 19–20

D.5. Data Preprocessing

  • Radar and WiFi signals were aligned with video, and timestamps synchronized the multimodal streams for analysis.The described preprocessing also blurred faces and sensitive visuals and muted sensitive audio information.Source: §D.5 · p. 20

D.6. Annotations

  • Activities and events were annotated at fine-grained and integrated levels, with GPT-generated Q&A refined by humans for relevance.The annotations support contextual and semantic analysis of recorded scenes.Source: §D.6 · p. 20

D.7. Dataset Structure

  • The dataset separates a multi-day, primarily single-language main session from a single-day, multilingual extension session.Video and audio are stored as MP4, annotations as JSON, and radar and WiFi signals as CSV and HDF5, respectively.Source: §D.7 · p. 20

D.8. Annotations

  • The annotation types include event and activity labels as well as Q&A annotations for contextual and semantic analysis.This section describes the same GPT-first, human-refined annotation process reported in D.6.Source: §D.8 · p. 20

D.9. Cost Breakdown

  • EgoLife began in a narrow setting that sacrifices some generalizability, and finding a reliable annotation team took two months and five trials.The authors describe this as a foundation for expansion and community contributions.Source: §D.9 · p. 20

E. Daily Activities

  • The week progressed from planning, shopping, and cooking to Earth Day party preparations, the party itself, and a final day of cleanup and farewells.Activities also included dance practice, decorations, outdoor exploration, and a dog bite that led one participant to get vaccinated.Source: E. Daily Activities · pp. 20–22

F. Details of EgoIT

  • EgoIT combines egocentric datasets spanning daily life, AR, gaze, industrial tasks, question answering, procedural learning, cooking, and routine activities.Its Q&A-generation instructions call for visually grounded, answerable questions across reasoning, planning, activity localization, descriptions, and predictions.Source: F. Details of EgoIT · pp. 22–23

G.1. Egocentric Datasets · G.2. Ego-Exo Datasets · H. Annotation Examples

The sections survey the expanding range of egocentric and ego-exo datasets, then illustrate how EgoLife organizes annotations into time-aligned, multimodal captions. Together, they show task-specific resources, broader-scale collections, and annotation examples spanning actions, objects, scenes, and audio.

G.1. Egocentric Datasets

  • Early egocentric datasets grew from activity-specific recognition collections to resources for video question answering, communication, pose estimation, and other varied tasks.Examples include EgoVQA’s 600 question-answer pairs, EgoCom’s speaker labels and word-level transcriptions, and datasets for 3D pose and human-object interaction.Source: §G.1 · pp. 23–24
  • EPIC-KITCHENS [4] and Ego4D [5] marked a shift toward larger, broader egocentric datasets, inspiring derivatives for tracking, dense masks, event labels, narrations, and long-video QA.Related resources include TREK-150, VISOR [31], N-EPIC-KITCHENS, EgoClip [30], and EgoSchema [11].Source: §G.1 · p. 25
  • Procedural-learning datasets cover assembly, repair, and instructional tasks, with annotations for steps, timestamps, collaboration, and step-by-step guidance.Assembly101 provides 513 hours of multiview assembly and disassembly video; EgoProceL [27] covers 16 tasks across 62 hours.Source: §G.1 · p. 25
  • Hand-focused datasets support hand detection, gesture and pose estimation, and hand-object interaction, with annotations ranging from pixel masks to 3D poses.AssemblyHands includes 3.0M annotated images, while EgoGesture contains 24,000 gesture samples.Source: §G.1 · p. 25
  • Recent egocentric resources and benchmarks extend into video generation, planning, retrieval, spatial cognition, navigation, scene-text QA, personal assistance, and procedural-error detection.Examples include EgoVid-5M [146], EgoPlan-Bench2 [152], MomentSeeker [153], EgoTextVQA [155], EgoMe [161], and EgoOops [164].Source: §G.1 · pp. 25–26

G.2. Ego-Exo Datasets

  • Ego-exo datasets range from paired or multiview recordings of specific activities to larger collections, including 1,286 hours of simultaneous first- and third-person skilled-activity video in Ego-Exo4D.EgoTaskQA builds reasoning questions on LEMMA, while EgoExoLearn [9] records learning from exocentric demonstrations.Source: §G.2 · p. 26

H. Annotation Examples

  • EgoLife stores caption annotations in SRT files for video-aligned review, with entries combining interactive instances, protagonist actions, and merged multimodal captions.Merged captions integrate visual captioning, audio captioning, and contextual details into a coherent scene description.Source: H. Annotation Examples · p. 26
  • The examples show time-stamped entries describing actions and objects alongside first-person visual-audio narratives, including dialogue and surrounding scene details.One entry describes holding a phone and seeing Jack walk upstairs while hearing a brief exchange; another follows movement toward people gathered around a table.Source: H. Annotation Examples · pp. 26–27

Related papers

Loading 2503.03803v3…