Source-linked AI summary
HoloAssist: an Egocentric Human Interaction Dataset for Interactive AI Assistants in the Real World
Xin Wang, Taein Kwon, Mahdi Rad, Bowen Pan, Ishani Chakraborty, Sean Andrist, Dan Bohus, Ashley Feniello, Bugra Tekin, Felipe Vieira Frujeri, Neel Joshi, Marc Pollefeys
TL;DR
Interactive physical-world assistants need evidence about perception, reasoning, and collaboration beyond digital tasks, predefined instructions, or simplified simulations. HoloAssist addresses this gap with a large-scale multimodal dataset of human-guided manipulation, annotations, benchmarks, and behavioral analysis, finding proactive, concise, environment-grounded instructor guidance. Its scope includes 166 hours and 350 instructor-performer pairs, while mistake-correction annotation remains an explicitly deferred area for future work.
Problem
Existing AI assistants lack sufficient first-hand physical-world experience, and simulated agents remain separated from real-world human interaction.
Method
HoloAssist records AR-based human-human task collaboration with seven sensor modalities, manual action and conversational annotations, and benchmarks for assistant capabilities.
Results
HoloAssist reveals that human instructors intervene proactively, give concise environment-grounded guidance, and adjust it using world-model estimates and detected task-disrupting mistakes.
Takeaways & Limitations
HoloAssist provides a resource for studying and designing interactive AI assistants that collaborate with humans during real-world physical tasks.
Takeaways & Limitations
The authors defer detailed study of whether and how mistakes are corrected to future work.
Abstract
from arXiv · showhide
Building an interactive AI assistant that can perceive, reason, and collaborate with humans in the real world has been a long-standing pursuit in the AI community. This work is part of a broader research effort to develop intelligent agents that can interactively guide humans through performing tasks in the physical world. As a first step in this direction, we introduce HoloAssist, a large-scale egocentric human interaction dataset, where two people collaboratively complete physical manipulation tasks. The task performer executes the task while wearing a mixed-reality headset that captures seven synchronized data streams. The task instructor watches the performer's egocentric video in real time and guides them verbally. By augmenting the data with action and conversational annotations and observing the rich behaviors of various participants, we present key insights into how human assistants correct mistakes, intervene in the task completion procedure, and ground their instructions to the environment. HoloAssist spans 166 hours of data captured by 350 unique instructor-performer pairs. Furthermore, we construct and present benchmarks on mistake detection, intervention type prediction, and hand forecasting, along with detailed analysis. We expect HoloAssist will provide an important resource for building AI assistants that can fluidly collaborate with humans in the real world. Data can be downloaded at https://holoassist.github.io/.
1. Introduction
HoloAssist addresses the gap between digital or simulated assistance and interactive physical-world collaboration by collecting multimodal human-human task guidance. It also analyzes instructor behavior and benchmarks core assistant capabilities.
- Existing assistants often lack first-hand physical-world experience and cannot perceive world states or actively intervene during task completion.
- Simulated agents remain difficult to transfer to the real world, while their interactions are simpler than real-world human interaction.
- HoloAssist pairs an AR-equipped performer with a remote instructor who watches a real-time egocentric feed and provides verbal guidance.The capture system uses a distributed server-client setup for streaming and multimodal recording.
- 166 hours cover 350 instructor-performer pairs completing 20 object-centric manipulation tasks with seven synchronized sensor modalities and manual annotations.Annotations include text summaries, intervention types, mistakes, and action segments.
- Human instructors intervene proactively, give concise environment-grounded guidance, and adjust instructions using estimates of world state and task-disrupting mistakes.
- The paper benchmarks mistake detection, intervention type prediction, 3D hand pose forecasting, action classification, and action anticipation.These benchmarks are intended to examine modules relevant to intelligent interactive assistants.
2. Related Work
HoloAssist extends prior egocentric and embodied-agent datasets with multi-person physical task collaboration, fine-grained intervention annotations, and seven synchronized modalities. Its design targets proactive, grounded assistance beyond question answering, predefined instructions, and simplified simulation.
- Interactive AI assistants: Prior interactive assistants include procedural VR agents and newer conversational task-guidance systems, while many existing systems rely on predefined instructions or question answering.
- Egocentric video datasets: Unlike earlier egocentric datasets, HoloAssist records multi-person interactive task completion, making human interaction available for proactive and environment-grounded assistance.
- Mistake detection: HoloAssist emphasizes fine-grained mistake detection because instructors may intervene when they spot a mistake instead of waiting for a coarse action segment to finish.
- Mistake detection: HoloAssist combines mistake detection with intervention prediction to support a more comprehensive analysis of assistive task-completion interactions.
- Multimodality and interaction: HoloAssist provides seven raw sensor modalities, expanding the multimodal inputs available for modeling intentions, world states, and future actions.
- Tasks and objects: Its object-centric benchmark spans 16 objects and 20 manipulation tasks, including everyday items and rare mechanical-lab equipment.
3. HoloAssist: Human Assistance Dataset
HoloAssist is a large-scale dataset of two-person physical manipulation tasks, combining egocentric multimodal capture with instructional and procedural annotations. It covers diverse participants, objects, actions, and mistakes to support interactive-assistance research.
- Dataset and collection: HoloAssist introduces a two-person collaboration dataset for situating AI assistance in physical-world tasks.Performers complete manipulation tasks while instructors provide verbal guidance.
- Dataset and collection: 166 hours span 350 instructor-performer pairs, 20 manipulation tasks, and 2,221 sessions.The dataset includes diverse objects ranging from everyday electronics to factory and laboratory equipment.
- Annotations: Annotations cover language summaries, conversations, coarse and fine-grained actions, and mistake attributes.The dataset contains 414 coarse-grained actions and 1,887 fine-grained actions, with head and tail action splits.
- Annotations: Fine-grained actions exhibit a long-tail distribution, partly because open-world interactions produce varied and interchangeable verb-noun expressions.Annotators revisit vocabulary to merge expressions referring to the same actions.
- Annotations: Mistakes are labeled as correct, self-corrected, verbally corrected, or uncorrected, then consolidated into one benchmark mistake class.Annotators separately record three mistake types and map verbally corrected mistakes to instructor-correction dialogue sentences.
4. Observations and Tasks from HoloAssist
HoloAssist reveals how human instructors respond to mistakes and ground guidance in the physical environment. These observations motivate benchmarks for mistake detection, intervention prediction, hand forecasting, and action understanding.
- Observations: Critical mistakes trigger proactive instructor intervention within less than 5 seconds, whereas other mistakes may be self-corrected or corrected later.Intervention timing depends on mistake severity.
- Observations: Immediate interventions commonly concern linear tasks whose progression stalls when steps are not followed in order.Furniture-assembly mistakes are more often self-corrected or corrected later because errors can remain unclear during the task.
- Observations: Instructor guidance frequently uses spatial deixis to express locations, directions, and distances in the 3D environment.Such expressions appear especially often during interventions that correct mistakes.
- Benchmark tasks: The benchmarks include fine-grained mistake detection, intervention type prediction from 1-, 3-, or 5-second windows, and 3D hand pose forecasting.Mistake detection predicts correct versus mistake labels despite approximately 6% mistakes among fine-grained actions.
- Benchmark tasks: The observations motivate world-state estimation models that detect mistakes, predict interventions, and augment assistance with spatial guidance.The benchmark suite also includes action recognition tasks following prior egocentric-video conventions.
5. Experiments
The experiments benchmark action recognition, mistake detection, intervention prediction, and 3D hand pose forecasting using multimodal baselines. Results show that hands and eye gaze can improve task performance, while modality benefits vary by task and training setup.
- Action recognition: Pre-trained ViT models achieve around 35% top-1 accuracy on fine-grained and 50% on coarse-grained action recognition.
- Mistake detection: Adding hands to RGB improves mistake detection from 35.11 to 36.18 points, while hands alone achieve 40.19 points.
- Intervention type prediction: 48.31% precision and 37.59% recall are achieved by adding hands and eye gaze to RGB for intervention type prediction.This improves approximately 35 and 4 percentage points over RGB, respectively.
- 3D hand pose forecasting: 9.80, 10.68, and 11.25 centimeters are the mean per joint position errors at 0.5, 1, and 1.5 seconds for hand pose forecasting.The hands-only model outperforms the static-hand baseline, which uses the last input pose.
- Multimodal analysis: Hand poses and eye gaze help recognize actions, detect mistakes, and understand user intentions, but simply adding modalities does not necessarily improve performance.Additional modalities may require specialized encoders or architectures for effective joint processing.
6. Conclusion and Future Work
The paper presents HoloAssist as a multimodal egocentric dataset for studying interactive physical-world assistance. It augments the data with annotations and benchmarks for mistake detection, intervention prediction, and hand pose forecasting, while identifying hand pose and eye gaze as useful information sources.
- HoloAssist captures seven sensor modalities during HoloLens 2 task interactions between performers and remote verbal instructors.
- The dataset adds annotations for action segments, mistakes, and intervention types and supports benchmarks for mistake detection, intervention prediction, and 3D hand pose forecasting.
- The conclusion identifies hand pose and eye gaze as useful information sources for interactive AI agents.
- The work leaves room for future improvements, including object-pose annotation and investigation of object-centered extensions.
- The stated scope includes supporting future research on competent AI assistants for everyday real-world tasks.
Supplementary Material
The supplementary material provides additional dataset visualizations and qualitative results, alongside details about annotation, data capture, analysis, and implementation.
- The supplement includes qualitative results, sample visualizations, annotation-method details, data-capture procedures, data analysis, and implementation details.Additional qualitative results and visualizations are available through the project website.
S.1. Interactive Assistive Task Completion
HoloAssist studies interactive assistive task completion in which instructors intervene when performers make mistakes or become confused. Different performers require different intervention moments and supervision levels for the same task.
- Instructors intervene during task completion when performers make mistakes or become confused.
- The diversity and complexity of intervention moments and types highlight the challenge of building interactive AI assistants.
- For the same assemble-tray-table task, different performers receive interventions at different moments and instruction levels.
S.2.1 Instructor Training
Instructor training familiarizes instructors with the objects and example tasks while preserving flexibility during data collection.
- Instructors receive training to become familiar with the objects and tasks before data collection begins.
- Task instructions serve as references rather than strict procedures.
- Instructors may add or skip tasks, change their order, and adapt how they provide instructions.
S.2.2 IRB Approval for Data Collection
HoloAssist's data collection received prior IRB approval, with participant consent and privacy-conscious capture procedures.
- The HoloAssist data collection process was reviewed and approved by an IRB before the study began.
- Participants reviewed and signed consent forms and an information sheet covering data capture and future usage.
- Data capture mostly occurred in offices and labs and avoided recording human faces.
S.3. Data Capture Platform
HoloAssist combines synchronized sensor streams with structured annotations for actions, conversations, mistakes, interventions, and forecasting-oriented analysis. The dataset also reveals long-tailed action labels and provides interactive summaries and transcripts, while hand forecasting remains challenging.
- Data Capture: Spatial sensor data use a world coordinate system with axes X=Forward, Y=Left, and Z=Up.
- Data Capture: Camera intrinsics include a 3×3 matrix, distortion parameters, focal length, principal point, and image dimensions.
- Data Capture: Seven sensor modalities are captured live, while text is added after data collection.
- Annotations: Annotations include text summaries, coarse- and fine-grained actions, and conversations, with mistake and intervention attributes.
- Annotations: Fine-grained actions are indivisible movements typically lasting 1–2 seconds or less, whereas coarse-grained actions represent higher-level task steps.
- Analysis: Fine- and coarse-grained action labels follow long-tail distributions, while third-person summaries capture key events and transcripts provide interactive conversation records.