An Outlook into the Future of Egocentric Vision

1 Introduction

• The paper situates wearable computing as the anticipated next version of mobile technology, while noting that the ability to realize it lags despite AI and hardware advances.Mobile phones are already used for navigation, recording everyday footage, and connecting people to experiences.Source: §1 · pp. 1–2
• Current egocentric vision research focuses on a handful of applications where technology already makes a difference, such as industrial monitoring or infrequent tasks, rather than everyday needs.The paper argues this perspective has limited efforts to push toward integrating egocentric devices into everyday life.Source: §1 · p. 2
• The survey starts from envisaged future applications, then analyzes the tasks they require to identify technology gaps and promising research directions.The authors use a scenario-based approach, which they note can help prioritize R&D.Source: §1 · p. 2
• Unlike earlier surveys that mostly addressed specific topics, this paper provides a holistic, updated overview spanning twelve egocentric vision tasks.Coverage includes localisation, scene understanding, recognition, anticipation, gaze, social behaviour, pose, hand interaction, person identification, summarisation, dialogue, and privacy.Source: §1 · p. 2
• The survey moves from character-based future stories to task mapping, task-specific reviews, general datasets, and concluding research questions.Its task reviews cover seminal works, state-of-the-art methods, datasets, and limitations to future applications.Source: §1 · p. 2

2 Imagining the Future

• The authors build five location- or occupation-based future use cases, introducing relevant technology before character stories and linking story elements to research tasks.EgoAI is presented as a wearable for in-situ multimodal sensing from the wearer’s perspective and ego-based assistance.Source: §2 · p. 2

2.1 EGO-Home

• At home, EgoAI assists Sam with meal choices and cooking, immersive entertainment, remote social play, health-related advice, and control over sharing a daily clip.The story ranges from tracking ingredients and offering cooking guidance to helping Sam delete a social-media draft.Source: §2.1 · pp. 3–4

2.2 EGO-Worker

• At work, EgoAI guides Marco through factory navigation and procedures, detects a live electrical hazard, and carries feedback into later training and planning.It also checks protective equipment, communicates daily goals, and monitors whether procedures are completed safely.Source: §2.2 · pp. 4–5

2.3 EGO-Tourist

• For tourism, EgoAI personalizes Claire’s Turin itinerary, provides an interactive AR museum guide, and offers context-sensitive suggestions during her day.The story includes bringing artwork to life, answering questions about monuments, and recommending a nearby café while she waits for a bus.Source: §2.3 · pp. 5–6

2.4 EGO-Police

• For policing, EgoAI supports Judy with location and route guidance, suspect identification, multisensor backpack assessment, and privacy-aware reporting.The imagined device also connects her to experts and saves relevant events in a report while securing sensitive information under administrative rights.Source: §2.4 · pp. 6–7

2.5 EGO-Designer

• For film production, EgoAI lets Stanley redesign a set, search warehouse furniture, preview actor movement, and share a changing 3D scene with the crew.It also projects makeup guidance and lets the director preview planned scenes and lighting in real time while filming.Source: §2.5 · p. 7

From Narratives to Research Tasks

• EgoAI is envisioned as a person-centric device that integrates tasks from the narratives and travels with its wearer.The narratives-to-tasks mapping also examines whether current wearables can perform those tasks or require more advanced devices.Source: §3 · p. 8

Scene-level understanding

• Localisation and 3D scene understanding support navigation and augmented reality, but dynamic and outdoor scenes remain challenging for current devices.Devices can scan and model static environments; dynamic and outdoor settings remain active research areas.Source: §3 · p. 8

Action-level understanding

• Action recognition and anticipation are envisioned for workplace assistance, but smart glasses do not yet robustly recognise actions in real time.RGB and depth data are commonly processed offline because of hardware limitations.Source: §3 · p. 8

Understanding people and interaction

• Gaze tracking supports gaze-directed interaction, while social understanding and first-person body-pose estimation connect wearer activity with other people.The narratives also identify hand pose and hand-object interaction as important to EgoAI.Source: §3 · pp. 8–9

Privacy

• Always-on capture raises privacy concerns, and current wearable glasses leave privacy regulation to users through interactive switches.The paper notes differing data-protection laws across countries.Source: §3 · p. 9

Scope of the review

• The review focuses on tasks requiring visual understanding, while excluding AR-specific approaches, audio-only tasks, personalised recommendations, measuring devices, and assessment tasks.It also covers recurring user-engagement tasks such as summarisation and dialogue, and discusses integrating vision with other sensor cues.Source: §3 · p. 10

4 Research Tasks and Capabilities

• The review selects seminal and current methods, task-specific datasets, and reflections on the gap between current methods and anticipated applications.It does not attempt to cover every intermediate work or general-purpose datasets in this section.Source: §4 · p. 10

4.1 Localisation

• Localisation covers coarse 2D place recognition, 6-DoF camera-pose estimation, and SLAM, which builds a map while tracking the camera.The review broadens the scope beyond wearable-only methods because these tasks share similarities with handheld, remote, and vehicle viewpoints.Source: §4.1 · pp. 10–11

4.1.1 Visual Place Recognition

• Visual place recognition identifies a place from one or more images, with Recall@N measuring whether true places appear among the top N retrieved results.Recent approaches target scalable training and use learned descriptors, multi-level attention, or feature mixing for retrieval and re-ranking.Source: §4.1.1 · p. 11

4.1.2 Visual localisation

• Visual localisation estimates camera position and orientation in a known 3D scene through retrieval-and-matching pipelines or single-image pose regression.Methods include transformer-based pose regression and alternatives to costly 3D point clouds, such as sparse landmarks and compact dense scene models.Source: §4.1.2 · pp. 11–12

4.1.3 Simultaneous Localisation and Mapping (SLAM)

• SLAM jointly maps an unknown environment and estimates camera pose; monocular systems face scale drift, while egocentric head motion produces short, noisy tracks and reduced parallax.Proposed responses include using walking-induced body oscillation to recover scale and local loop closure aligned with head motion.Source: §4.1.3 · pp. 12–13
• Existing localisation datasets often use navigation-only recordings, and visual geolocalisation lacks a body-worn-camera dataset.EPIC-KITCHENS camera estimates offer activity footage but have no ground truth and are evaluated qualitatively; common benchmarks also have limited scene diversity and sensors.Source: §4.1.3 · pp. 13–14
• 45.6% / 61.3% recall at (1°, 10cm)/(5°, 1m) is reported for single-frame localisation on LaMAR.The review also notes 64.0% recall for visual place recognition on the Mapillary challenge and continuing robustness challenges in dynamic, changing environments.Source: §4.1.3 · p. 14

3D Scene Understanding

• 3D scene understanding aims to interpret the environment, identify relevant objects, and reason about their locations and possible interactions.The field spans task-specific problems, including task-relevant object discovery, action mapping, interaction-target prediction, visual queries, and collision localisation.Source: §4.2 · p. 14

State-of-the-art methods

• Recent approaches learn affordance landscapes, predict depth and surface normals, and use environment-aware video representations to predict local scene states.Other methods jointly recognise and localise actions or retrieve query objects in 3D by combining video, object proposals, and multiview geometry.Source: §4.2 · pp. 14–15

Datasets

• Available datasets include multimodal egocentric recordings with 3D poses, gaze, segmentation, depth, affordance labels, and social interaction footage.Examples include Aria Digital Twin, ODIN, EPIC Fields, EPIC-Aff, and Replay.Source: §4.2 · pp. 15–16

For the future

• Motion blur and unusual viewpoints challenge 3D reconstruction of dynamic content, leaving substantial work before egocentric systems can understand actions and activities in 3D.Egocentric and exocentric views may offer complementary information, but exocentric views are unlikely to be part of everyday EgoAI use.Source: §4.2 · p. 16

4.3.1 Action Recognition

• Action recognition classifies actions performed by the camera wearer, but dynamic, unpredictable camera motion and the wearer’s partial observability make it more complex than fixed-camera recognition.Audio, gaze, and optical-flow temporal dynamics can complement vision; research also targets low-energy models and higher-level action understanding, with top-1 and top-5 accuracy as common evaluation metrics.Source: §4.3.1 · p. 16
• 49.6% action recognition on EPIC-KITCHENS-100 validation was reported by Kazakos et al (2021), while Wu et al (2022a) reached 48.4% accuracy with 0.5× Patrick et al (2021)’s parameters.Girdhar et al (2022) reported 47.4% top-1 performance on validation; Yan et al (2022) achieved 47.2% action recognition, outperforming their own prior approach by 1% but remaining below Girdhar et al and Kazakos et al. All but Kazakos et al’s approach were developed for general action recognition rather than optimized specifically for egocentric vision.Source: §4.3.1 · p. 17
• 51.0% action recognition accuracy on EPIC-KITCHENS-100 validation set is the reported state of the art, achieved by Zhao et al (2023c) using LLM-generated video descriptions for contrastive video-text learning.Plizzari et al (2023) also use language narrations to guide domain-generalizable representations across Ego4D domains.Source: §4.3.1 · p. 18
• Shah et al (2023) improved the EgoProceL F1 Score by 14% and achieved state-of-the-art results by aligning modalities from multiple datasets with BMC2 loss for procedural step learning.This work uses optical flow, depth, or gaze to support AR applications; procedural recognition learns task steps across multiple videos rather than identifying isolated actions.Source: §4.3.1 · pp. 17–18

4.3.2 Recognising objects

• Object recognition is difficult in spontaneous, densely packed first-person scenes, and recent work addresses detection, continual learning, deformable-object tracking, query localization, and object-state-change benchmarks.Examples include Akiva et al (2023)’s temporal- and scale-space self-supervised detection, Huang et al (2023b)’s DETracker, and Xue et al (2023)’s EgoT2, which achieved state-of-the-art Object State Change Classification performance. Object-guided token sampling and object-aware attention have also been used to improve action recognition.Source: §4.3.2 · pp. 19–20
• Object-focused datasets support recognition, interaction, tracking, and referring-expression tasks, including fine-grained annotations and long-term tracks.Examples span TEgO’s 19 objects, HOI4D’s 800 instances across 16 categories, VISOR’s 272K manual semantic masks and 9.9M interpolated masks, EgoObjects’ 654K annotations across 368 categories, EgoTracks’ 22,028 tracks, PACO-Ego4D’s 140K part masks, and RefEgo’s more than 12k clips.Source: §4.3.2 · p. 20
• Action recognition remains limited by few ego-specific architectures, lower fine-grained performance than third-person recognition, slow 0.5–1% gains, long-tail data, and limited gaze and modality-specific annotations.The cited EPIC-KITCHENS-100 state of the art is 51.0%; the section notes uncertainty over whether progress is constrained by dataset size, label ambiguity, or architecture. Gaze-equipped large-scale datasets with fine-grained action labels are lacking, and EPIC-SOUNDS highlights the value of modality-specific action labels and temporal extents.Source: §4.3.2 · pp. 20–21
• Expensive labeled data and subjective closed action vocabularies limit models; open-label recognition remains unresolved, and LLM-based progress lacks evaluation metrics.The passage presents LLMs as a future direction while noting the lack of metrics to assess success and monitor progress.Source: §4.3.2 · p. 21
• Understanding how objects can be used or interacted with supports more intuitive systems for assistive robotics, augmented reality, and personal assistance.The passage identifies object affordances as going beyond basic recognition and as a key advancement for EgoAI.Source: §4.3.2 · p. 21

4.4.1 Anticipating Actions

• Action anticipation predicts the next semantic action from past video; methods include attention-based video transformers, latent-goal learning, and LLM-assisted procedure planning.Source: §4.4.1 · pp. 21–22

4.4.2 Anticipating Objects

• Object anticipation predicts the future interacted object, interaction verb, and time-to-contact; current approaches use multimodal transformers and language-derived action context.Source: §4.4.2 · p. 22

4.4.3 Anticipating Trajectories

• Trajectory prediction covers wearer locomotion, other people, and hands; recent methods use egocentric motion cues, social features, or semantic links to bird’s-eye coordinates.Source: §4.4.3 · pp. 22–23
• Ego4D supports anticipation of actions, objects, and trajectories, while earlier trajectory work used purpose-collected datasets because common action datasets lack substantial locomotion.Source: §4.4.3 · p. 23
• 18.1% mean top-5 per class recall on EPIC-KITCHENS-100 test; object anticipation reaches 24.7% top-5 mAP for next-active-object prediction and 3.4% when adding verb and time-to-contact on Ego4D test.Source: §4.4.3 · p. 23
• Most approaches assume a video segment sampled at a fixed time before an action, although the future action’s occurrence time is unknown at test time.Source: §4.4.3 · p. 23

• Gaze prediction estimates the wearer’s attention from an eye-mounted egocentric camera, and gaze measurements provide ground truth for evaluating attention maps.Source: §4.5 · p. 23

Seminal works

• Earlier methods combined saliency with egomotion, used hands and gaze-shift patterns, or incorporated object, hand-object, and head-motion cues.Source: §4.5 · pp. 23–24

State-of-the-art papers

• Lai et al. (2022) combine scene-object likelihood, object location, and head-motion cues with a Global-Local Correlation module, achieving state-of-the-art gaze prediction on EGTEA Gaze+ and Ego4D.Source: §4.5 · p. 24

For the future

• Public gaze datasets are limited; reported challenges include annotation needs, individual differences, blinks, occlusion, and illumination, with F1 scores of 44.8 on EGTEA Gaze+ and 43.1 on Ego4D.Source: §4.5 · p. 24

4.6.1 Modelling the relationships with the speakers

• Speaker-related work spans speech enhancement, auditory attention, identifying who speaks to the wearer, active-speaker detection, diarisation, and transcription.Approaches generally process audio and video, with segmentation maps or bounding boxes used to locate speakers.Source: §4.6.1 · p. 25

4.6.2 Detecting and modelling social interactions

• Social-interaction studies use disparate task definitions, and limited datasets leave only a handful of recent methods, including dyadic-relation modelling and persuasion-strategy prediction.The persuasion task uses utterance and video features to predict strategies in social deduction games.Source: §4.6.2 · pp. 25–26

4.6.3 Estimating attention towards the camera wearer

• Attention-to-wearer work asks whether visible people look at or talk to the wearer, with applications including contact alerts, diagnosis, and conversation logging.Evaluation uses manual ground truth or comparison with human performance, but has not been investigated systematically.Source: §4.6.3 · p. 26

4.6.4 Estimating joint attention

• Joint attention can be modelled through social saliency maps, jointly attended objects, or groups with coherent attention, with potential uses in monitoring interactions and video summarisation.It may also support disorder diagnosis and notify the wearer when shared focus changes.Source: §4.6.4 · p. 27
• Joint-attention estimation remains emerging; Huang et al. (2020a) temporally locate joint-attention periods before segmenting shared objects to handle clutter and noisy gaze.Their hierarchical graphical model yields more reliable segmentation than relying on regions near gaze points.Source: §4.6.4 · p. 27

4.6 Social Behaviour Understanding

• Available resources include FPSI, paired egocentric videos for dyadic interactions, joint-attention datasets, and multimodal conversational datasets such as EgoCom and EasyCom.EgoCom contains nearly 40 hours of recordings; EasyCom includes synchronised participant audio and realistic acoustic noise.Source: §4.6.4 · p. 28

4.6.4 Estimating joint attention

• 53.9% mAP and 54.3% accuracy on Ego4D’s “Talking To Me” test set contrast with eye-contact detection’s 0.936 precision and 0.943 recall, comparable to 10 trained human coders.The eye-contact results are across 18 validation subjects; performance varies significantly by task.Source: §4.6.4 · p. 28

4.7 Full-body Pose Estimation

• Wearable-camera pose methods represent the body with kinematic or volumetric models and are challenged by cameras’ limited field of view, which often misses the full body.Performance is assessed using Mean Per Joint Position Error (MPJPE).Source: §4.7 · p. 28
• Early work used 16 outward-facing cameras for motion capture, while later approaches explored downward-facing rigs, SLAM-based world coordinates, and outward-camera inference of invisible body pose.EgoCap used a head-mounted stereo rig; Xu et al. (2019) introduced real-time capture with a single monocular fisheye camera.Source: §4.7 · pp. 28–29
• Physics-based pose methods generate plausible poses or forecast future ones, while Luo et al. (2021) estimated physically plausible 3D human-object interactions using object poses.Their dataset includes 6 degrees of freedom object poses.Source: §4.7 · p. 29
• For downward-looking fisheye data, Wang et al. (2023b) combine inferred and inpainted depth with 2D pose features in a shared 3D voxel space to enforce scene constraints.The constraints are intended to avoid physically unrealistic poses such as floating or penetrating the environment.Source: §4.7 · p. 29
• For outward-looking data, Li et al. (2023) estimate head motion using SLAM and a transformer, then condition a diffusion model to infer full-body pose without paired video–3D-motion training data.Evaluation sequences contain navigation through virtual scenes, not other activities.Source: §4.7 · p. 29
• Pose datasets range from simulated to real-world wearer recordings, with outward-looking resources including EgoBody for multi-person scenes and EgoHumans for outdoor, multi-view analysis.Zheng et al. (2022) also introduced a large-scale motion-prediction dataset that includes gaze information.Source: §4.7 · p. 30
• 118.5 mm MPJPE for body-oriented methods compares with 121.1–152.1 mm for outward-looking methods, despite testing on different datasets; third-person results are around 20 mm.The third-person comparison is on the Humans3.6M benchmark.Source: §4.7 · p. 30
• 141.1 mm World PA First-MPJPE is reported for estimating other people’s poses on EgoBody, remaining notably distant from third-person performance.World PA First-MPJPE aligns the first predicted frame with the ground truth.Source: §4.7 · p. 30

4.8.1 Hand Pose Estimation

• Hand pose estimation regresses 3D keypoints from RGB, video, depth, or meshes, but egocentric evaluation remains limited despite methods addressing pose refinement and alternative inputs.Occlusion from object manipulation, limited field of view, and head motion make the task challenging; among the discussed recent methods, only Yang et al (2022) evaluated on egocentric hand pose.Source: §4.8.1 · p. 31

4.8.2 Hand Gestures

• Hand gestures are discrete, semantic counterparts to hand poses and support wearable-device interaction; approaches use temporal, spatiotemporal, and multimodal signals for recognition.Some methods combine hand and head-camera streams or share knowledge across modalities during training, while others jointly segment hands and recognize gestures.Source: §4.8.2 · pp. 32–33

4.8.3 2D Hand-Object Interaction

• 2D hand-object methods associate hands with objects and predict relations such as contact, active-object status, object state changes, and potential affordances.Some work uses semantic masks to represent hands and active objects, while query-point methods predict interaction-relevant depth, surface normals, physical properties, and affordances.Source: §4.8.3 · p. 33

4.8.4 3D Hand-Object Interaction

• 3D hand-object methods estimate hand and object geometry, poses, and actions, with recent work addressing reconstruction, consistent motion, and interaction fields.ArcticNet reconstructs hand and articulated-object motion, while InterField estimates each hand vertex’s distance to the closest object mesh.Source: §4.8.4 · pp. 33–34
• Synthetic-data efforts generate annotated hand-object interactions, diffusion-generated interactions, and diverse grasping poses subject to 3D geometric constraints, yet the synthetic-to-real gap remains unsolved.Leonardi et al (2023) generate depth maps, segmentation masks, boxes, attributes, and 3D distances; Ye et al (2023b) use diffusion models to reason about where and how to interact; FLEX (Tendulkar et al 2023) synthesizes full-body and hand grasping poses. Xu et al (2023b) use tactile sensing and a simulator to synthesize images for in-hand object reconstruction where ground truth deformation data is difficult to obtain.Source: §4.8.4 · p. 35
• 3D hand pose estimation reaches MPJPE of 23.46 mm on the AssemblyHands test set, but hands interacting with objects and complex scenarios remain difficult.The state-of-the-art result is reported by Ohkawa et al (2023).Source: §4.8.4 · pp. 35–36
• Gesture recognition reaches 96.9% accuracy on EgoGesture, but current systems and devices support relatively few gestures, leaving larger gesture sets open for exploration.The benchmark result is attributed to Chalasani et al (2018); devices can also support custom gestures and two-hand gestures coordinated with gaze.Source: §4.8.4 · p. 36
• Generic hand-object methods remain far from robust generalization across objects and environments, constrained by scarce interaction labels and difficult object transformations.Synthetic data can provide large-scale, multi-domain interactions across diverse objects (Tendulkar et al 2023; Leonardi et al 2022), but often lacks photorealism; object transformations have only recently begun to be studied (Darkhalil et al 2022; Fan et al 2023).Source: §4.8.4 · p. 36
• Hand-object segmentation reaches Hand Mask Average Precision of 95.6% and Active Object Average Precision of 25.7%, while object state change reaches accuracy of 67.6% and Average Precision of 15.5%.The results on EPIC-KITCHENS VISOR and Ego4D, respectively, expose difficulty relating actions to objects and their environment.Source: §4.8.4 · p. 36

Person identification

• The wearer-identification task can be formulated as closed-set classification or open-set matching, using hands and gait, with applications in theft prevention and personalisation.More broadly, person identification covers recognizing people in the camera’s view and identifying the wearer; head-height views often truncate or occlude people.Source: §4.9 · p. 36

Seminal works

• Early wearable systems identified faces for memory support or assistance, while later work also explored privacy filtering and identifying suspects or missing people.Farringdon and Oni (2000) developed a face-identification application; Krishna et al (2005) introduced iCare, and Wang et al (2013) targeted prosopagnosia; Thomaz et al (2013) explored privacy filtering and Sajjad et al (2020) law-enforcement identification.Source: §4.9 · pp. 36–37

State-of-the-art papers

• Recent approaches use egocentric appearance, motion, or cross-view matching to identify people or the camera wearer.Methods include style transfer for person re-identification, matching egocentric and top-view detections with geometric and temporal reasoning, and wearer signatures derived from gait or hands.Source: §4.9 · pp. 37–38

Datasets

• Available datasets support cross-view wearer matching, self-identification and re-identification; Ego2Top contains 50 top-view and 188 egocentric videos (approximately 225 thousand frames) with relatively few identities, and IITMD-WFP has 3 hours from 31 subjects.Source: §4.9 · p. 38

For the future

• Fixed-camera person re-identification approaches nearly 98% mAP, while egocentric models barely reach 65%; wearer identification remains at proof-of-concept, prompting a need for larger, more diverse benchmarks.The reported egocentric results are from small-scale datasets.Source: §4.9 · p. 38

Background

• Keyframe selection, video skimming, and fast-forwarding are three summarisation forms, commonly evaluated with F1-score.Fast-forwarding prioritises significant segments while slowing less important portions without necessarily trimming them.Source: §4.10 · p. 38

State-of-the-art papers

• General summarisation benchmarks contain few egocentric videos, and reported state-of-the-art methods are not designed for egocentric challenges such as head motion and long ordinary segments.SumMe contains 4 of 25 egocentric videos and TVSum 5 of 50; He et al (2023) had not been directly evaluated on egocentric benchmarks.Source: §4.10 · p. 39
• Nagar et al (2021) personalise summary length and content with reinforcement learning, while newer features remain unassessed for the approach.Its actions select or discard sub-shots, with basic and customisable rewards.Source: §4.10 · p. 39
• IntentVizor supports interactive, query-focused summary editing, and Elfeki et al (2022) select diverse events and camera viewpoints across multiple wearable streams.IntentVizor uses learned user intent and GSE-GCN to correlate video features with intent.Source: §4.10 · p. 39

Datasets

• Dataset availability is a bottleneck: many prior datasets are unpublished, while FPVSum is egocentric but only partially annotated and SumMe and TVSum contain roughly 15% egocentric video.The 80-hour Dataset of Multimodal Semantic Egocentric Videos supports evaluation of smoothness and semantic highlights.Source: §4.10 · pp. 39–40

For the future

• F1-scores are 63.4 on TVSum and 55.0 on SumMe for He et al (2023), while Wu et al (2022b) achieves 50.9 on the egocentric query-based dataset.The survey says the limited query-concept set falls short of reflecting envisioned real-life scenarios.Source: §4.10 · p. 40

4.11.1 Visual Question Answering (VQA)

• VQA settings include multiple-choice classification and open-ended responses assessed with recall@k, language metrics, or user studies.The paper describes open-ended VQA as more challenging and realistic than multiple-choice VQA.Source: §4.11.1 · p. 40
• VQA work spans episodic-memory queries, goal-oriented reasoning, and affordance-centric step-by-step assistance, while its datasets and subtasks are still rapidly evolving.The affordance-centric approach returns ordered text and bounding-box answers from video, script, question, and answer features.Source: §4.11.1 · p. 41
• Egocentric VQA remains at an early proof-of-concept stage, with indoor-only testbeds and limited investigation of speech and sound despite its potential for assistive applications.Outdoor use brings shifts in video features and question semantics.Source: §4.11.1 · p. 42
• On EgoSchema, models with billions of parameters achieve under 33% multiple-choice QA accuracy versus about 76% for humans; EgoTaskQA open-ended accuracy reaches 30% versus 82% for humans.The survey notes that large pretrained vision-language models may drop on the most challenging reasoning questions.Source: §4.11.1 · p. 42

4.11.2 Ego-Language Models

• Video-language pretraining aligns video and text representations; EgoNCE adapts InfoNCE with action-aware positive and scene-aware negative sampling for long egocentric videos containing multiple actions.Lin et al (2022) proposed the objective.Source: §4.11.2 · pp. 42–43
• Other ego-language approaches include episode-based word acquisition and LLM-based narration and rephrasing; EgoVLPv2 adds reusable cross-modal attention for retrieval and grounding.EgoCLIP contains 3.8M egocentric clip-text pairs, and EgoMCQ has 39K video-text alignment questions.Source: §4.11.2 · p. 43
• Standard CLIP-based video-language pretraining generalises poorly to egocentric video, motivating adaptation of existing models or tasks tailored to EgoAI dialogue.Head motion, occlusion, and limited field of view are identified as sources of complexity.Source: §4.11.2 · p. 43

4.12 Privacy

• Wearable cameras’ mobility enables always-on capture and processing of sensitive information about users and bystanders, creating privacy challenges distinct from fixed cameras.The survey notes that privacy has not been systematically investigated, likely because camera-equipped wearables are not yet mainstream.Source: §4.12 · p. 43

4.12.1 Users’ studies on individual privacy

• In a one-week study of 36 lifelogging-camera users, participants preferred in-situ removal of sensitive images, with sensitivity shaped by time, location, objects, and people.Bystander concern and respect for others’ privacy also appear in the cited user studies.Source: §4.12.1 · p. 44

4.12.2 Redacting sensitive information

• Redaction approaches include blocking images from sensitive locations or with detected screens, and routing images with detected bystanders to wearer review.Other transformations blur or alter images while aiming to preserve downstream tasks such as activity recognition.Source: §4.12.2 · p. 44

4.12.3 Privacy preserving by design

• Privacy-preserving-by-design methods mediate or reduce raw sensor data, protect localisation information, or shut off video when sensitive content is detected.The survey says studies are not systematic, often involve fewer than 50 participants, and mostly use custom rather than commercial devices; privacy-preserving solutions remain comparatively few.Source: §4.12.3 · pp. 44–45

4.13 Beyond individual tasks

• Combining recognition, dynamic memory, re-identification, and trajectory forecasting for assistance, or evaluating daily-task performance, remains largely unexplored in egocentric vision.The survey identifies joint understanding of surroundings, actions, and intentions as key to integrating EgoAI into daily life.Source: §4.13 · p. 45

5 General Datasets

• General datasets support varied egocentric vision research, and the section compares their collection characteristics, annotations, and task coverage.Tables 1–3 organize datasets by collection characteristics, current annotations, and the tasks they support.Source: §5 · pp. 45–46, Table 1 · p. 46, Table 2 · p. 46
• The section also describes task-focused datasets, from home and long-duration recordings to gaze-annotated cooking and multi-person social-interaction videos.ADL, UTE, and KrishnaCam offer distinct durations and signals; GTEA Gaze and EGTEA Gaze+ focus on recipe preparation with gaze, while the FPSI passage introduces social-interaction data.Source: §5 · pp. 46–47
• EPIC-KITCHENS-100 provides 100 hours of unscripted recordings from 37 participants in four countries, with 90K action segments and additional video, audio, and camera-pose annotations.Its extensions include VISOR, EPIC-SOUNDS, and EPIC Fields; the dataset is widely used for action recognition, privacy, anticipation, and other tasks.Source: §5 · pp. 47–48
• MECCANO, Assembly101, and HOI4D provide procedural or object-interaction data, including synchronized gaze, depth, RGB, action segments, hand poses, and 3D scene annotations.Assembly101 includes over 100K coarse and 1M fine-grained action segments, while HOI4D contains 2.4M RGB-D frames and 4,000 sequences.Source: §5 · p. 48
• Ego4D spans 3,670 hours of daily-life video from 931 camera wearers across 74 locations and nine countries, with benchmarks covering past, present, and future activities.Its scale and diversity support many existing and newly introduced tasks, and its potential remains to be fully explored.Source: §5 · p. 48
• Existing datasets leave gaps in the forecast scenarios: home datasets omit downtime, grooming, and personal health; worker datasets lack holistic daily activities and safety feedback; touring remains small-scale; and no public dataset corresponds to EGO-Police.The survey also calls for more egocentric-understanding datasets from the entertainment industry.Source: §5 · pp. 48–49

6 Conclusion

• EgoAI is envisaged as a wearable device integrated into everyday life across indoor and outdoor settings, work, home, and holidays.The paper uses character-based scenarios to present this future-to-present perspective.Source: §6 · p. 49
• A competent egocentric assistant is expected to require multimodal capabilities and the ability to solve multiple fundamental vision tasks.The conclusion argues that a separate deep model for every task would be inefficient and that tasks are co-dependent.Source: §6 · p. 49
• The survey identifies generative tasks as a future research direction, including guidance through illustrated work steps and proposing scene variations.It describes current generative approaches as limited to contexts such as future head-motion prediction, gaze anticipation, and hand-object interaction modelling.Source: §6 · p. 49
• The survey reviews 12 egocentric vision tasks, covering seminal works, current methods, datasets, and directions for future innovation.It notes that mobile wearable cameras and user-specific scene understanding create challenges distinct to egocentric vision.Source: §6 · p. 49
• The conclusion encourages open-set task settings to manage novelty and reduce reliance on predefined label sets.It presents this as a way to enhance model trustworthiness.Source: §6 · p. 49
• The survey stopped short of examining real-time sensing and learning, although it says future models need real-time or very low-latency performance.It also points to online connectivity and privacy- and protection-aware models as requirements for future devices.Source: §6 · pp. 49–50