Ego4D: Around the World in 3,000 Hours of Egocentric Video

• Ego4D targets limitations of existing datasets: first-person perception requires continuous, uncurated video understanding, persistent 3D context, and interpretation of human-object interactions and social behavior.Unlike brief third-person Internet imagery, robotics and augmented reality receive long first-person video streams from agents interacting with their environments.Source: §1 · pp. 1–2
• 3,670 hours of egocentric video from 931 participants span 74 locations in 9 countries, capturing daily life largely unscripted and in the wild.Participants vary in background, occupation, gender, and age; recordings cover home, workplace, leisure, social settings, and commuting.Source: §1 · p. 2
• 1 to 10 hours per camera-wearing session provide long-form video showing extended interactions with environments, objects, and people.Portions also include audio, 3D meshes, gaze, stereo, and synchronized multi-camera views of the same event.Source: §1 · p. 2
• Five benchmark tasks address past experiences, present interactions, and future activity, supported by millions of annotations.Over 250,000 hours of annotator effort produced temporal, spatial, and semantic labels, dense activity narrations, natural-language queries, and speech transcriptions.Source: §1 · pp. 2–3
• Ego4D emerged from a two-year collaboration between Facebook and 13 universities, publicly sharing data intended to support research beyond its proposed benchmarks.The authors envision applications in robotics, augmented reality, 3D sensing, multimodal learning, speech, and language, and announced a June 2022 benchmark challenge.Source: §1 · p. 3

Large-scale third-person datasets

• Third-person pretraining [70] suffers from a sizeable domain mismatch [201] with passively captured egocentric video.Unlike photographer-curated third-person Web data, wearable-camera footage features unusual viewpoints, motion blur, and no temporal curation.Source: §2 · p. 3

Egocentric video datasets

• Ego4D follows out-of-lab collection pioneered in, while including much broader wearer demographics than prior datasets largely relying on graduate students.Unlike kitchen-focused collections [44], Ego4D covers indoor and outdoor environments. Other predecessors use (semi-)scripted activities, including Charades-Ego [201] and EGTEA.Source: §2 · p. 3
• 3,670 video hours versus 100 [43], and 931 wearers versus 71 [201], distinguish Ego4D’s scale across hundreds of environments, 74 locations, and nine countries.Existing collections span one or dozens of environments and typically only one or a few cities.Source: §2 · p. 3

Egocentric video understanding

• Millions of annotations support multiple complex tasks, while Ego4D’s benchmarks widen the tasks available for egocentric-video research.The dataset can facilitate research on human-object interactions, anticipation [4] [75], and social interactions, among other areas.Source: §2 · p. 3

3.1. Collection strategy and camera wearers

• 3,670 hours of video from 931 camera wearers across 74 cities were collected by 14 teams in nine countries and five continents.Participants wore cameras for 1–10 hours per session to capture unscripted activity. Recruitment used word of mouth, advertisements, community postings, and occupations offering varied visual contexts.Source: §3.1 · pp. 3–4
• 45% of participants were female, and 96 were over 50, alongside varied occupations and age groups.Demographic information was available for 64% of participants because of collection protocols or optional responses. Two participants identified as non-binary, and two preferred not to state a gender.Source: §3.1 · p. 4, §3.2 · p. 4

3.2. Scenarios composing the dataset

• 8 minutes is the typical raw clip duration, capturing naturally unfolding, participant-selected activities across diverse daily-life scenarios.A U.S. Bureau of Labor Statistics survey informed coverage of home, leisure, transportation, errands, and work, with many actions within each scenario. For audiovisual and social benchmarks, consenting participants at five sites undertook social activities while sharing conversation audio and unblurred faces.Source: §3.2 · p. 4

3.3. Cameras and modalities

• Seven head-mounted camera types vary in modalities, field of view, mounting, and battery life to reduce overfitting to a single capture device.A downward-pointing head-mounted GoPro captures detailed hand–object interactions, whereas a heads-up Vuzix approximates the wearer’s eye-level viewpoint but misses interactions close to the body.Source: §3.3 · p. 4
• Additional modalities include 3D scans, audio, gaze, stereo, synchronized wearable videos, and dense, timestamped activity narrations.Environment meshes connect dynamic activities to persistent 3D context. Synchronized views support social-interaction analysis, while audio supports analysis of conversations, acoustic scenes, and events.Source: §3.3 · pp. 4–5, Table 1 · p. 5

3.4. Privacy and ethics

• Privacy safeguards include informed consent, withdrawal and video-redaction rights, protection of private spaces, and de-identification of personally identifiable information.Partners follow institutional research policies, including ethics review where relevant. Capture typically occurs in controlled environments with everyone's consent or in public spaces with faces and other identifying information blurred.Source: §3.4 · p. 5

3.5. Possible sources of bias

• Coverage biases favor urban or college-town locations, pandemic-era home activities, and active periods; narrations may reflect annotators’ local vocabulary.Geographic coverage remains incomplete, major public social events are underrepresented, and battery limits prevent daylong filming. Crowdsourced annotators work at two sites in Africa.Source: §3.5 · p. 5

3.6. Dataset accessibility

• Accessibility measures include precomputed action features, benchmark-specific train/test subsets, and downloads filtered by benchmark or modality.Features use SlowFast 8x8 with a ResNet 101 backbone pretrained for Kinetics 400. Standardized mini-sets are planned as annotations expand.Source: §3.6 · p. 5

• 3.85M narration sentences describe camera-wearer activities, averaging 13.2 sentences per minute of video.Narration precedes all other annotation.Source: §4 · p. 5
• Each video receives two independent narrations using a summarize-then-rewatch procedure inspired by the pause-and-talk narrator [44].Annotators summarize a five-minute clip, then rewatch and repeatedly pause to describe each camera-wearer action. Each free-form sentence is recorded with its timestamp.Source: §4 · p. 5
• 1,772 unique verbs and 4,336 unique nouns describe the activities and objects represented in the narrations.Source: §4 · p. 5
• The narrations enable data-driven action and object taxonomy construction, assignment of videos to relevant benchmarks, and identification of temporal windows for annotation seeding.Taxonomy construction uses text mining, while benchmark assignment relies on sorting videos by their content.Source: §4 · pp. 5–6
• The authors characterize Ego4D as the largest aligned language–video repository to their knowledge, potentially supporting research on video with weakly aligned natural language.They contrast it with HowTo100M, whose noisy spoken narrations only sometimes describe the activities taking place.Source: §4 · p. 6

5. Ego4D Benchmark Suite

• 48-1,000 annotated hours per benchmark accompany 3,670 narrated hours in the first release, supporting five benchmarks spanning past, present, and future.The suite addresses long-form, passively captured video, attention cues, person-object interactions, and multisensory data; benchmark sampling balances task relevance with geographic diversity. Baseline models and quantitative results are presented in the appendices.Source: §5 · p. 6

5.1. Episodic Memory

• Episodic Memory localizes answers in past video through natural-language queries, visual object queries, and high-level activity queries.Natural-language queries return answer windows; visual queries return an object's last sighting temporally and spatially, using a 2D box and optionally a 3D displacement vector. Moments queries return every temporal window containing the requested activity.Source: §5.1 · p. 6
• ∼74K queries spanning 800 hours support Episodic Memory, with 13 language-question templates and a 110-activity moments taxonomy mined from narration summaries.Annotators create free-form language or visual queries paired with response tracks and label every moments interval. NLQ uses top-k recall at a tIoU threshold; MQ uses mAP at multiple tIoU thresholds and top-kx recall; VQ measures temporal/spatio-temporal localization and search timeliness.Source: §5.1 · pp. 6–7

5.2. Hands and Objects

• Hands and Objects annotations link pre-change, point-of-no-return, and post-change frames to hand, tool, and object boxes, state-change types, verbs, and nouns.Data selection targets activities likely to involve hand-object interactions. Evaluation uses absolute temporal error in seconds for localization, classification accuracy for state-change classification, and average precision (AP) for changed-object detection.Source: §5.2 · p. 7
• Hands and Objects introduces the first dedicated video benchmark for object state changes, testing onset localization, changed-object detection, and state-change classification.The point-of-no-return marks when change begins; detection uses pre-change, point-of-no-return, and post-change frames, while classification determines whether a change occurred. The target is a physical state change that generalizes across different tools, grasps, and object-tool-action combinations.Source: §5.2 · pp. 7–8

5.3. Audio-Visual Diarization

• Audio-Visual Diarization tracks visible participants, detects active speakers including the unseen wearer, segments speaker-specific speech, and transcribes English speech.Evaluation uses MOT metrics, speaker error rate, diarization error rate, and word error rate for the respective tasks. Participant-perspective recordings capture noisy backgrounds, overlapping or interrupted speech, moving wearers, and speakers facing away.Source: §5.3 · p. 8

5.4. Social Interactions

• Social Interactions uses egocentric conversational cues to identify whether visible partners are looking at or talking to the camera wearer.Looking at me (LAM) and Talking to me (TTM) build on AVD face tracks and active-speaker annotations, adding per-face, per-frame binary labels. Both tasks use mAP and Top-1 accuracy, measuring precision at every frame.Source: §5.4 · pp. 8–9

5.5. Forecasting

• Forecasting unifies prediction of wearer trajectories, hand positions, upcoming object interactions with verbs and time-to-contact, and future action sequences.Narrations seed refined verb-noun taxonomies; annotations mark contact/pre-condition frames and earlier object/hand boxes, while structure from motion supplies trajectory ground truth. Evaluation uses L2 distance for locomotion/hands, Top-5 mean Average Precision for short-term interactions (discounting Top-4 false negatives), and edit distance for action sequences.Source: §5.5 · p. 9

6. Conclusion

• Orders of magnitude larger in scale and diversity than existing work, Ego4D targets multimodal perception of egocentric video.Source: §6 · p. 9
• Ego4D aims to enable AI to learn from daily-life visual and auditory experiences around the world.Source: §6 · p. 9
• The benchmark suite supports video-understanding innovations relevant to augmented reality, robotics, and other domains.Source: §6 · p. 9

Contribution statement

• Kristen Grauman initiated and led the project, Andrew Westbury led program management and operations, and Jitendra Malik provided scientific advising.Source: Contribution statement · p. 9
• Benchmark development involved researchers from all participating institutions, including cross-institution baseline evaluations.Source: Contribution statement · p. 9
• Starred authors (∗) drove implementation, collection, or annotation development, while dagger-marked authors (†) were faculty PIs and working group leads.Source: Contribution statement · p. 9
• University partners managed all video collection and participant recruitment outside Facebook Reality Labs’ collection.Source: Contribution statement · p. 10
• Facebook Reality Labs collected video using Vuzix Blade® Smart Glasses inside Facebook buildings, with paid participants who signed consent to share their data.Collection took place in a closed environment.Source: Contribution statement · p. 10
• Facebook AI led the annotation effort.Source: Contribution statement · p. 10

Appendix

• Appendix A details collection by site and acknowledges primary contributors; Appendices F through J detail individual authors’ benchmark contributions.Source: Contribution statement · pp. 9–10

A. Data Collection

• 660.5 hours from 138 participants at IIIT Hyderabad cover 36 scenarios across five Indian states, with participants aged 18–84.Local coordinators collected natural activities through remotely guided, COVID-aware procedures, followed by centralized verification and applicable de-identification. Every video was manually inspected for sensitive content before sharing.Source: A. Data Collection · p. 11
• 90 hours of cooking and 51 hours of handcraft were recorded by 81 Tokyo participants, balanced by gender and age.Cooking was unscripted at home using GoPro cameras, whereas laboratory handcraft used stereo cameras. The University of Tokyo ethical review board approved the protocol.Source: A. Data Collection · p. 11
• 262 hours from 82 Bristol participants span commuting, leisure, work, sports, household activities, DIY, and crafts.Pandemic-era collection used mailed cameras, Zoom training, consent forms, and encouraged paired household recordings of participants’ natural activities. Researchers de-identified 36% of videos using Secure Redact and manual review; publication followed GDPR compliance.Source: A. Data Collection · pp. 11–12
• Minnesota collected approximately 313 hours of data from 45 participants encouraged to engage in unstructured natural social interactions; Georgia Tech collected approximately 43 hours of egocentric video from 19 participants.Minnesota required participants in each social group to belong to the same household to minimize COVID-19 risk. Georgia Tech also captured binaural audio and third-person video.Source: A. Data Collection · pp. 12–13
• 400 hours from 206 FRL participants cover six staged laboratory scenes representing apartments, a clothing store, and a grocery store.Three of four apartments received textured 3D reconstructions using [209], with mesh polygons annotated into 35 object categories plus background. Emails and bills were mock data, and video calls occurred only between participants.Source: A. Data Collection · p. 14
• Pittsburgh contributed over 500 hours of video overall, with a large portion gathered from skilled workers, alongside recordings from Rwanda, Colombia, and Catania, Italy.Rwanda captured over 150 hours, with a portion available in the current release; Colombia recorded 302.5 hours from 77 participants across 20 scenarios, focusing on simultaneous viewpoints of scenes and social interactions. Catania recorded more than 359 hours from 57 participants across 25 scenarios and collected 3D scans of seven locations.Source: A. Data Collection · pp. 14–15
• 453 hours from 66 KAUST participants cover 80 scenarios within the university compound, including snorkeling, golfing, cycling, and driving.Participants came from 20 countries, ranged from 22 to 53 years old, and were approximately half female. After camera training, each recorded 1–10 hours across at least two activities within two days, followed by quality and IRB-requirement checks.Source: A. Data Collection · p. 15

B. De-identification Process

• Two video categories distinguish indoor recordings with explicit identity consent from the majority requiring de-identification because identity consent was not given.Only the first category supports the Audio-Visual Diarization and Social Interaction benchmarks and includes all 400 FRL hours. Only university-collected footage falls into the second category, including public-space recordings.Source: B. De-identification Process · p. 16

B.1 De-identification overview

• Manual screening separated videos without PII, requiring no redaction, from videos containing PII that university partners de-identified for their own data.PII could include participants, accidentally captured addresses or reflections, and bystanders or cars in public spaces.Source: §B.1 · p. 16
• brighter.ai and Primloc’s Secure Redact automatically detected faces and number plates, with all automated blurring outputs reviewed for false positives and false negatives.Review checked both unnecessary blurring of non-private items and inaccurate or insufficient blurring of identifiers.Source: §B.1 · p. 16
• Other PII, including written names/addresses, phone screens/passwords, and tattoos, required manual identification and per-frame blurring.This work used commercial tools alongside open-source software including Computer Vision Annotation Tool (CVAT), Anonymal, and SiamMask.Source: §B.1 · p. 16
• 10x the video length could be needed to carefully redact outdoor recordings, with relative time costs varying significantly across scenarios.Source: §B.1 · p. 16

Sensitive object detection

• The four-stage process begins with reviewers flagging videos containing sensitive objects, followed by automatic detection using brighter.ai.Sensitive objects include human faces, license plates, and credit cards.Source: §B.2 · p. 16

False positive removal

• Reviewers manually inspect automatically detected bounding boxes and reject those that contain no sensitive information.Source: §B.2 · p. 16

False negative correction

• Reviewers inspect every video for missed sensitive objects, manually annotate bounding boxes, and verify cross-frame proposals generated by an online object tracking algorithm [222].Source: §B.2 · p. 17

Image blurring

• After detections are corrected, a robust blurring process de-identifies the image regions defined by the bounding boxes.Source: §B.2 · p. 17

Time costs

• 780 hours of manual labor were required to de-identify 500 hours of video, with time costs varying substantially by scenario.Three review passes required 250, 115, and 250 hours; false-positive removal required 115 hours, and false-negative correction required 50 hours.Source: §B.2 · p. 17

C. Demographics

• Self-declared ethnicity and/or birthplace are reported separately by state or country because grouping granularity differs; Minnesota participants lack these data.Participants are residents of the locations under which their demographics are reported.Source: C. Demographics · p. 18
• 63% of UK participants reported ethnicity, versus 100% in India and 90% in Colombia; 95% of Indiana participants reported birthplace.UK demographic reporting was optional, with 52/82 participants responding.Source: C. Demographics · p. 18

D. Narrations

• Dense, temporally aligned narrations describe camera-wearer activities and object interactions, provide visually grounded language data, and support benchmark preparation.Narrations support action and object taxonomies, identification of benchmark-relevant videos, and selection of video regions requiring annotation.Source: D. Narrations · p. 20

D.1 Narration instructions and content

• 5-minute maximum clips receive two independent narration sets to improve coverage and account for narration errors.Both sets are retained without merging because they do not serve as benchmark ground truth.Source: §D.1 · p. 20
• 1-3-sentence clip summaries tagged #summary precede timestamped descriptions, created by rewatching, pausing at events, and describing ongoing actions or interactions.Annotators first watch the entire clip, then summarize its overall activity and setting.Source: §D.1 · p. 20
• Each dense narration describes an atomic action or object interaction, including interactions with others; #C identifies the camera wearer and other letters identify others.Annotators are prompted to describe everything happening as though speaking by phone to a friend who cannot see the video.Source: §D.1 · p. 20

D.2 Narration analysis

• 7.4 words on average describe individual timepoints, while summary narrations average 16.8 words and describe activities at a higher level.Individual timepoint narrations highlight a single action or object interaction, covering object interactions, tool use, camera-wearer motions, and other people’s activities.Source: §D.2 · p. 20, Figure 15 · p. 23
• Narration word frequencies reveal scenario-specific objects alongside shared objects, highlighting the diversity of captured video content.Cooking narrations feature bowls, spoons, and plates; board-game narrations feature cards, dice, and pawns, while hands, paper, and phones appear across scenarios.Source: §D.2 · p. 20

D.3 Action and object taxonomy

• Following ideas from [44], taxonomy construction uses Spacy parsers to extract verbs and direct-object nouns, then disambiguates and manually clusters them.Heuristics split verb senses, decompose compound nouns into roots and modifiers, and map collective nouns to their main entities. Manual clustering merges redundant terms, such as “cut”, “chop”, and “slice”.Source: §D.3 · pp. 20–21
• 115 verbs and 478 nouns form the final taxonomy, constructed from raw narrations containing 1,772 unique verbs and 4,336 unique nouns.The taxonomy is used in benchmark tasks.Source: §D.3 · pp. 20–21

D.4 Narrations for annotation prioritization

• Narrations and summaries automatically prioritize task-relevant videos for manual benchmark labeling rather than selecting instances randomly.All videos are narrated, but benchmark labels cover subsets: conversation-rich videos suit Audio-Visual Diarization, while hand-object interactions suit Hands and Objects.Source: §D.4 · p. 21

D.5 Contributions statement

• Tushar Nagarajan developed the taxonomy and analyzed narrations; Kristen Grauman developed instructions and helped coordinate annotation; Michael Wray co-developed the taxonomy.Nagarajan also helped develop narration instructions; Grauman helped coordinate pilots and contributed to taxonomy formation.Source: §D.5 · p. 22

D.2 Narration analysis

• 3.85M narration sentences cover 3,670 hours of video, averaging 13.2 sentences per minute.Annotation frequency depends on activity: examples include 0.17 sentences/minute for television watching and 63.6 sentences/minute for crop harvesting.Source: §D.2 · p. 20, Figure 15 · p. 23

E. Benchmark Data Splits

• Related benchmark tasks share consistent splits, preventing cross-task training–validation overlap within families, while substantially different task families can have different splits.AVD and Social have 764 relevant video hours, including 47.7 annotated hours; other tasks have less restrictive content requirements. Test annotations are accessible only through a submission server, and narrations overlapping any test set are withheld.Source: E. Benchmark Data Splits · p. 24

F. Episodic Memory Benchmark

• Episodic Memory comprises visual, natural-language, and moments queries, each requiring responses localized within egocentric video.Natural-language queries require visual and linguistic understanding to identify a contiguous response sufficient to answer without external knowledge. Moments queries retrieve all occurrences of a requested predefined action category.Source: §F.1 · p. 25, F. Episodic Memory Benchmark · p. 25

F.1 Formal task definitions

• Visual queries use an identifiable object crop to retrieve its most recent occurrence before a query frame as a contiguous bounding-box track.This supports open-world object localization rather than categorical queries. When an associated 3D scan exists, the response additionally specifies displacement from the query location to where the object was last seen.Source: §F.1 · p. 25

F.2 Selecting clips for annotation

• Clip selection combines geographic and scenario diversity with task-specific narration scores favoring navigation, nonrepetitive actions, or overlap with the moments taxonomy.VQ selection favors spatially and temporally separated object occurrences, filters scenarios, ranks navigation-verb entropy, and prioritizes videos with 3D scans regardless of entropy. NLQ adds a unique-to-total verb ratio favoring nonrepetition; MQ ranks taxonomy-overlapping verbs and nouns using an entropy-based score and restricts clips to relevant categories.Source: §F.2 · p. 26

F.3 Annotation

• 3 visual queries per 5-, 8-, or 16-minute clip include an object crop, textual name, query frame, and response track.Clear representative crops and accurate, contiguous bounding-box tracks are required, with no object recurrence between the response track and query frame. Automatic and manual quality checks trigger reannotation of low-quality VQ and NLQ clips.Source: §F.3 · pp. 26–28
• 1.5 for dnorm and 15 for Vnorm are rejection thresholds for disagreement between two independently annotated 3D boxes; exceeding either rejects the query.The boxes establish ground-truth displacement from the query frame to the object's last-seen location. Because scans and videos were recorded at different times, annotators may need to place a box where the video indicates an object existed despite its absence from the scan.Source: §F.3 · p. 27
• 8-minute and 20-minute clips support NLQ annotation using 13 paraphrased query templates and visually answerable temporal windows.Queries are past-tense questions posed at clip end; answers cannot require separate windows more than 3 seconds apart or exceed 50% of clip length. Responses should span diverse clip locations and entities, while questions requiring external knowledge or guesses about intentions are excluded.Source: §F.1 · p. 25, §F.3 · pp. 27–28
• 3 independent annotators label activity windows in 8-minute moments clips, and their annotations are unioned for completeness under an annotator-reliability assumption.The taxonomy clusters five-minute summary narrations using pretrained BERT [51] features concatenated with main-verb and main-noun embeddings, followed by manual labeling. It covers Cooking, Cleaning, Shopping, Handyman, and Farmer/Gardener, with each instance recorded as start time, end time, and label.Source: §F.1 · p. 25, §F.3 · pp. 28–29

F.4 Data Analysis

• 22,602 visual queries annotate 433 video hours spanning 54 scenarios and 10 universities, with video-disjoint train/validation/test splits.The 3D localization subset contains 1,043 queries across 13 video hours associated with four University of Catania scans.Source: §F.4 · p. 29
• 19.2K natural-language queries annotate 227 video hours, with short response windows making retrieval a challenging search within much longer clips.The collection covers 13 query templates, 34 scenarios, and 10 universities, using disjoint 60%/20%/20% train/validation/test splits. Queries contain 8.3 ± 2.1 words, and response windows are typically 9.3 ± 21.5 seconds long.Source: §F.4 · p. 29
• 22.2k action instances span 326.4 hours and 2,488 clips across 110 moment-query categories, whose frequencies are long-tailed.Categories require at least 50 instances and representation in every disjoint 6:2:2 train/validation/test split. Instances average 45.2 seconds; 22.4% last under three seconds, most last under one minute, and most clips contain 1–20 instances.Source: §F.4 · pp. 29–30

F.5 Evaluation measures

• Visual-query evaluation measures 2D temporal and spatio-temporal localization, success, recovery, and search efficiency, alongside 3D RMSE, angular error, and annotation-based success.tAP and stAP use thresholds {0.25, 0.50, 0.75, 0.95}; Succ requires spatio-temporal IoU ≥0.05, and rec% requires per-frame box IoU ≥0.5. Search efficiency encourages contextual search, assigning 0.0 when every search-window frame is accessed.Source: §F.5 · pp. 30–31
• NLQ follows with recall@k, while moment queries combine temporal mAP and instance-count-scaled recall@kx to evaluate localization.NLQ uses k = {1, 5} and IoU thresholds {0.3, 0.5}; moment-query mAP uses tIoU thresholds {0.1, 0.2, 0.3, 0.4, 0.5} and their average. Moment-query recall@kx uses k = {1, 2, 3}, m = {0.3, 0.5, 0.7}, and x instances per category per video, without penalizing predictions for absent categories.Source: §F.5 · pp. 31–33

F.4 Data Analysis

• Visual-query separation distances and response-track lengths show potential biases, while bounding-box positions are largely spatially unbiased.Separation distances span 1–200 frames with a mode of approximately 30; track lengths span 1–40 frames with a mode of approximately eight. Boxes are near-uniformly distributed across most of the image, but few occupy its top 10%.Source: §F.4 · p. 29, Figure 22 · p. 32

F.6 Baselines

• Task-specific baseline models incorporate state-of-the-art components where relevant and serve as starting points for future research.Source: §F.6 · p. 33

Visual queries 2D localization baseline

• The VQ2D baseline detects crop-matched proposals, selects the most recent similarity peak, and tracks bidirectionally using KYS [22] to recover response tracks.The pretrained backbone, proposal network, and tracker remain frozen while the similarity head is trained with hard-negative mining. Augmentation replaces query crops with response-track boxes and applies rotations and brightness changes.Source: Figure 28 · p. 34, Visual queries 2D localization baseline · p. 35
• 6.8% success, 5% tAP25, 4% stAP25, and 5% recovery gains follow replacing the Simple SiamHead projection with four residual blocks.With Simple projection, replacing particle-filter tracking with KYS [22] changes validation success from 32.4% to 33.0%, while improving tAP, stAP25, and recovery by 2%, 2%, and 14.3% respectively. Similar trends occur on the test set.Source: Visual queries 2D localization baseline · p. 36, Table 9 · p. 37
• 48.0% search efficiency is achieved by searching the nearest 50% of frames, with only a 6–16% relative performance drop.Uniform subsampling degrades performance sharply, whereas searching recent frames is more stable; reducing recent-frame coverage to 10% causes a 38–60% performance reduction. Exhaustive detection has 0% search efficiency.Source: Visual queries 2D localization baseline · p. 36, Figure 30 · p. 37

Visual queries 3D localization baseline

• 15% of video frames receive camera pose estimates after temporal refinement, compared with roughly 2% before refinement.Iterative SuperGlue matching, triangulation, and PnP extend pose coverage. Qualitative examples show successful estimation despite scene changes, while abrupt motion and close-up views leave frames unlocalized.Source: Visual queries 3D localization baseline · p. 38
• The VQ3D pipeline combines VQ2D response tracks, camera poses, and DPT depth estimates to locate the target object in 3D.Depth comes from the most recent response-track frame with an estimated pose, using the median around the detected centroid; DPT is pretrained on NYU v2. A pinhole camera model reconstructs displacement from centroid depth and estimated camera intrinsics.Source: Visual queries 3D localization baseline · p. 38
• The best VQ2D baseline combined with pretrained DPT achieves the best VQ3D L2 and success performance, although sparse camera poses limit success.Evaluation covers 164 validation queries, with QwP measuring availability of both response-track and query-frame poses. With ground-truth response tracks, scan-rendered depth outperforms DPT, indicating additional depth-estimation limitations.Source: Table 10 · p. 39, Visual queries 3D localization baseline · p. 39

Natural language query baselines

• VSLNet uses a shared Transformer [215], query-conditioned video attention, and query-guided highlighting to localize language-described temporal spans.The implementation uses SlowFast [70] video features and frozen BERT [52] text features, with an internal multimodal dimension of 128. Training lasts 200 epochs, selecting the checkpoint with the best validation performance.Source: Natural language query baselines · p. 40
• 5.80% top-1 and 13.90% top-5 recall at IoU=0.3 are achieved by the 2D-TAN natural-language query baseline.Overlapping 40-second windows with 20-second stride address the O(N^2) temporal-map space burden. Training uses windows containing or adjacent to ground-truth moments, whereas testing uses all windows.Source: Natural language query baselines · pp. 39–40, Table 11 · p. 40

Moments queries baseline

• DETAD [9] analysis identifies localization and classification errors; short moments perform poorly, while clips with 2–3 same-category instances are easiest to detect.The baseline uses VSGN without VSS, combining a graph pyramid with two-stage anchor scoring and boundary refinement. It filters the resulting temporal action detections to the queried categories.Source: Moments queries baseline · p. 41

Discussion

• 42.9% success rate contrasts with 0.13 tAP and 0.06 stAP for the visual-query baseline, whose best performance occurs at 0% search efficiency.Naïve attempts to improve search efficiency drastically reduce performance.Source: Discussion · p. 42
• About 10s response windows within 8-minute clips make natural-language episodic-memory retrieval challenging for existing state-of-the-art video-localization models.Flexible natural-language queries enable users to search for diverse pieces of episodic memory.Source: Discussion · p. 42
• 5.96% average mAP exposes weak all-category recognition in moment queries, despite reasonable category-specific recall @kx, tIoU=m.Wrong-label errors are significant, while long-tailed categories and widely varying durations complicate recognition. Short moments constitute a large proportion of the dataset but perform worse than long moments.Source: Figure 35 · p. 41, Discussion · p. 42, Figure 36 · p. 42

G.1 Motivation

• Object state changes reveal activity, intention, and actionable affordances while supporting reasoning and applications such as robotic manipulation.Their representation combines temporal stages, spatial boxes for hands/tools/objects, and semantic action, object, and state-change labels. Temporal stages comprise PRE, PNR—the beginning of a change that cannot easily be reversed—and POST showing completion.Source: §G.1 · pp. 44–45

G.2 Related Work

• Prior work studies attribute-object composition in images, exemplified by MIT States, and object-state discovery or temporal transformations in video.Video approaches include weakly supervised object-state detection by Fathi et al. and state-transformational procedure planning by Chang et al. [30].Source: §G.2 · p. 45

G.3 Benchmark Definitions

• Hands and Objects evaluates PNR temporal localization, state-change object detection, and object state-change classification.Temporal localization measures absolute error in seconds only on state-change videos; detection measures AP for PNR-frame object boxes using PRE, PNR, and POST frames as input. Classification measures accuracy over N state-change types plus one category without state change.Source: §G.3 · pp. 45–46

G.4 Data Selection

• 120 hours of selected data span 53 scenarios, 7 universities, and 406 participants.Scenarios receive hand-object-interaction likelihood ratings from 0 to 3, guiding sampling. Within scenarios, sampling proceeds randomly through participant, video, and five-minute clip, taking shorter videos whole and balancing universities for geographic diversity.Source: §G.4 · p. 46

G.5 Data Annotation

• Annotation proceeds through critical-frame selection, pre-period labeling, and post-period labeling for narrated hand-object interactions.Narration-centered 8-second snippets receive taxonomy verb labels and PNR-first frame selection; PRE and POST may vary provided relevant objects are fully visible. Annotators label hand/tool/object boxes and tool/object categories backward from PNR to PRE, correct detector-initialized hand boxes, and use the labeled PNR to guide POST annotation.Source: §G.5 · p. 46

G.6 Data Analysis

• PNR frames cluster near narration, with nearby pre/post frames highlighting rapid state changes and boundary modes reflecting long repetitive actions.The PNR distribution centers around the 4-second narration point within 8-second snippets. Annotations emphasize low-level atomic actions, with common manipulation verbs and object nouns accompanied by long tails.Source: §G.6 · pp. 46–47, Figure 40 · p. 47
• ∼825K bounding boxes annotate hands, objects, and tools across varied sizes and image locations.These comprise ∼245K left-hand, ∼260K right-hand, ∼280K object, and ∼40K tool boxes. Of 478 object categories, 442 extend beyond the 80 COCO categories.Source: §G.6 · pp. 46–47

G.7 Baselines: Object State Change Classification and PNR Temporal Localization

• 0.425-second validation and 0.489-second test PNR temporal localization errors make SlowFast + Perceiver the best localization baseline; Bidirectional LSTM leads state-change classification.Other learnable localization baselines achieve around 0.85 seconds or less, versus around 1.1 seconds for center-frame prediction. All learnable classification baselines exceed 60% accuracy, versus close to 50% for always-positive prediction.Source: §G.7 · p. 48, Table 15 · p. 48, Table 16 · p. 49

G.8 Baselines: State Change Object Detection

• 8-14% AP across single-frame state-change object detection baselines demonstrates difficulty when only the PNR frame is available.Baselines directly detect objects using Faster-RCNN, CenterNet [241], and DETR [27], or condition predictions on detected hands using 100DOH. Large object-size variation and missing appearance changes across frames motivate future multi-frame models incorporating tracking or association.Source: Table 18 · p. 49, §G.8 · p. 49

G.9 Discussion

• The benchmark motivates jointly understanding when, where, and what objects change, rather than treating temporal localization, spatial localization, and semantics independently.For example, recognizing splitting could inform PNR localization and predict multiple post-change bounding boxes. Methods exploiting these task dependencies remain to be developed.Source: §G.9 · pp. 49–50

H.1 Motivation

• Audio-visual diarization requires locating speakers, identifying their speaking intervals, and recovering speech content from egocentric recordings with explicit identity consent.Speakers may be visible intermittently or never, while the unseen camera wearer is often amplified by microphone proximity. Camera and head motion introduce visual blur and distortion.Source: §H.1 · p. 51, H. Audio-Visual Diarization Benchmark · p. 51

H.2 Related Audio Visual Learning Work

• Audio-visual learning spans localization, cross-modal features, spatialization, and source separation, while addressing overlapping conversations also requires understanding the camera wearer’s auditory attention.Prior work uses audio to disambiguate actions [226] and visual information to assist speech recognition [5]. Visual signals also benefit diarization and source separation [6] [57] [79].Source: §H.2 · pp. 51–52

H.3 Related Datasets and Benchmarks

• Existing diarization datasets include non-egocentric conversational video and small, staged or constrained egocentric recordings.VoxConverse [39] contains over 50 hours of non-egocentric multi-speaker clips, while AVDIAR contains 27 minutes across 23 staged dummy-head sequences. EASYCOM [53] provides 5 hours of conversations with multichannel audio in a constrained room setting.Source: §H.3 · p. 52
• Ego4D combines first-person, in-the-wild, multi-speaker conversations, addressing limitations of existing datasets involving solitary activity, partly scripted conversations, restricted environments, or non-egocentric views.EPIC-Kitchens does not support multi-person conversation tasks; EASYCOM has stationary participants and partly scripted conversations, while AVDIAR uses a dummy head in one environment. Unlike VoxCeleb, VoxConverse, and AVA, Ego4D provides first-person conversations in casual daily-life settings.Source: §H.3 · p. 52

H.4 Tasks: Definition and Annotations

• Localization and tracking predicts bounding boxes for probable speakers despite apparent spatial movement caused by the camera wearer’s head motion.Human annotators validate and correct machine-generated boxes frame by frame, excluding distant speakers without conversational contact. Evaluation uses MOTA from CLEARMOT metrics and IDF1, IDP, IDR from Identity metrics to assess tracking and identity preservation.Source: §H.4 · p. 53
• Active speaker detection identifies speaking faces within the field of view, while separate voice activity detection covers the unseen camera wearer.Clip-specific anonymous speaker labels connect face tracks belonging to the same person. Evaluation uses frame-wise object-detection mAP with a face-box IoU threshold of 0.5, and object-classification mAP for the wearer’s voice activity.Source: §H.4 · p. 53
• Diarization determines who spoke when, using active-speaker outputs and evaluating temporal speaker attribution with diarization error rate (DER).Audio-visual annotations mark each speaker’s speech intervals, including overlapping speech and the camera wearer but excluding irrelevant background speech. DER accounts for missed speech, false alarms, and incorrect speaker attribution, normalized by total speech.Source: §H.4 · pp. 53–54
• Transcription recovers each speaker’s words despite overlapping speech, environmental noise, and head-motion-induced volume changes, with evaluation using Word Error Rate (WER).Multiple annotation passes combine human and automatic transcriptions, correct speaker assignments, and account for low-volume and overlapping speech. Outputs contain timestamps, transcripts, and speaker IDs; their segments differ from diarization segments, and ASR is evaluated separately on each segment.Source: §H.4 · p. 54

H.5 Data Statistics

• 572 five-minute clips constitute the initial annotated release, sampled from approximately 764 conversational hours within Ego4D’s 3,670 hours.Training receives 389 clips, validation 50, and testing the remainder. Long-tailed mean and maximum speaker counts accompany marginal split shifts and possible accent, vocabulary, and interaction biases; the authors expect shifts to diminish with more annotations.Source: §H.5 · p. 54

H.6 Baseline Modeling Framework

• The audio-only baseline combines VBx, Kaldi voice-activity detection, speaker embeddings, hierarchical clustering, and Bayesian hidden Markov refinement.ResNet101 produces 256-dimensional x-vectors [206], while hidden Markov states represent speakers. Kaldi VAD was selected for availability despite its telephone-domain mismatch, and diarization hyperparameters were tuned on Ego4D validation DER.Source: §H.6 · pp. 56–57
• Short-term tracking matches head detections using normalized bounding-box distances and appearance features, prioritizing longer trajectories to reduce identity switches.Matching excludes incompatible position, size, or appearance pairs; a two-stage scheme matches longer trajectories before shorter ones.Source: §H.6 · p. 57
• Long-term tracking greedily merges appearance-similar tracklets subject to temporal and distance constraints, avoiding exponential-complexity integer programming without guaranteeing global optimality.The algorithm repeatedly merges the closest compatible trajectories and caches pairwise tracklet distances for efficiency. Time-conflicting tracklets can produce suboptimal groupings.Source: §H.6 · p. 58
• Active-speaker baselines compare mouth-region classification with audiovisual TalkNet [211]; voice-activity filtering substantially improves the video-only approach.TalkNet [211] combines temporal audio/video encoders with cross-attention and self-attention. Mouth classification ignores faces turned away, and stacking multiple mouth images did not significantly improve results.Source: §H.6 · pp. 58–59
• Out-of-view speech is matched against visible speakers’ audio-signature libraries, but noisy active-speaker detections can contaminate those libraries and degrade classification.Matching uses a minimum-distance threshold, and audio embeddings remain confined to the same five-minute clip with anonymous person tags.Source: §H.6 · p. 60
• Camera-wearer speech detection uses either high-energy audio selection followed by voice matching or a supervised spectrogram classifier.The energy heuristic fails when the wearer never speaks or speaks very quietly, and works better with near-range microphones. The classifier uses a modified ResNet trained with cross-entropy on Ego4D audiovisual data.Source: §H.6 · p. 60
• Diarization evaluation uses aggregated audio-only DER but averaged audio-visual DER, with voice-activity detection affecting audio-only performance.Baselines include training on Ego4D alone and on existing diarization datasets; the authors identify adaptation to Ego4D-like data as an opportunity for improvement.Source: §H.6 · p. 60, Table 28 · p. 61, Table 26 · p. 61
• Transcription uses the pretrained Gigaspeech model from ESPNet, decoding oracle speech segments separately and pooling errors for validation/test WER.Oracle segmentation accommodates the pretrained model’s inability to process five-minute videos. Evaluation with alternative segmentation is challenging, especially for overlapping speech, leaving joint segmentation and transcription unresolved.Source: §H.6 · pp. 56, 60, 61
• ASR word error rates are high on both validation and test sets, reflecting challenges from overlapping speech, noise, speaker-volume differences, and occasional foreign words.Scoring removed punctuation, allowed contraction matching using the Kaldi repository’s English mapping, and used NIST sclite.Source: Table 29 · p. 61, §H.6 · p. 61

H.7 Discussion

• DER > 80% and WER > 60% expose the difficulty of joint audio-video diarization and transcription in egocentric, in-the-wild recordings.Overlapping speech complicates annotation and evaluation. Audiovisual source separation may help, but differences and changes in overlapping speakers’ amplitudes remain challenging.Source: §H.7 · p. 61
• The baseline lacks efficient information sharing across its four tasks, motivating joint cross-modal learning that accounts for overlapping speakers, interruptions, and environmental noise.The relationship between robust localization and tracking and multi-speaker diarization remains unstudied in this framework and poorly understood in the literature.Source: §H.7 · pp. 61–62
• Accents, culturally influenced vocabulary, and head-motion facial blur complicate speech capture and annotation, prompting human-annotation quality checks.The authors anticipate that unsupervised and self-supervised learning could help address these subjective attributes.Source: §H.7 · p. 62
• Proposed extensions include additional tasks and annotations, plus spatial-audio research using 3D environments coupled with SoundSpaces [32].Potential tasks include audiovisual sound-source localization and direction-of-arrival estimation. A small fraction of the dataset already contains binaural audio recorded with in-ear microphones.Source: §H.7 · p. 62

I. Social Interaction Benchmark

• Social interaction recordings retain participants’ faces and voices with informed consent, while screening removes directly identifying information such as last names and social-media accounts.The benchmark includes multi-person capture with participants consenting to unblurred faces and recorded conversations.Source: §I · p. 63

I.1 Formal Task Definition

• LAM predicts whether a target looks at the camera wearer using video and bounding boxes; TTM additionally uses audio to predict speech directed at the wearer.Both produce frame-level binary labels using temporally synchronized inputs. A speaker addressing the wearer while momentarily looking away can have LAM = 0 and TTM = 1.Source: §I.1 · p. 63

I.2 Annotation Statistics

• 1 or 2 seconds is the most frequent LAM or TTM annotation duration; these events are sparse, with LAM less frequent than TTM.Annotations reuse AV diarization clips, and many clips contain LAM or TTM events in 10% or fewer frames. Training uses 389 clips (32.4 hours), validation 50 clips (4.2 hours), and testing 133 clips (11.1 hours).Source: §I.2 · p. 63

I.3 Social Baseline Models and Results

• 78.07 mAP for LAM and 55.06 mAP for TTM indicate substantial room for improvement in detecting attention and speech directed toward the camera-wearer.LAM achieves 78.07% test mAP with Gaze360 [111] initialization versus 66.07% with random initialization. TTM improves test mAP by 9.77% over random guessing.Source: Table 30 · p. 64, §I.4 · p. 64, §I.3 · p. 64

I.4 Discussion

• TTM requires conversational semantics and audio-video fusion, while LAM must handle motion blur, fast head movements, and conversational gaze patterns.Proposed improvements include hierarchical analysis of dialogue, utterance semantics, and audio cues for TTM, plus more explicit modeling of head movement and gaze behavior for LAM.Source: §I.4 · p. 64

I.3 Social Baseline Models and Results

• LAM combines ResNet-18 face features with a bidirectional LSTM, while TTM concatenates visual and audio embeddings to classify the target of an utterance.LAM processes seven consecutive face crops and uses weighted cross-entropy for class imbalance; TTM encodes MFCC features with an audio ResNet-18 [38]. TTM limits segments to 1.5s, skips training segments shorter than 0.15s, and pads missing faces with blank images.Source: §I.3 · pp. 63–64, Figure 49 · p. 65

I.5 Social Dataset Collection

• 764 hours of social video and audio were collected across Atlanta, Bloomington, Redmond, Twin Cities, and Singapore, emphasizing naturalistic interactions and diverse participants and environments.Friends and families participated in structured activities and unstructured daily-life recording, with site-dependent wearable cameras, eye trackers, binaural audio, and smart watches. Most collection occurred during COVID-19 under protocols designed to avoid additional participant risk.Source: §I.5 · p. 65

I.6 Derived Tasks for Future Social Benchmarks

• Social Gaze Prediction generalizes LAM from looking at the camera-wearer to identifying each visible person’s gaze target.Labels can be derived by matching participant identities across synchronized recordings, although synchronization is approximate because cameras are not genlocked; alternative or unknown targets receive a separate label. Eye-tracker recordings also support egocentric attention prediction, extending prior video-based gaze prediction work.Source: §I.6 · p. 65
• SCG-2D restricts gaze targets to visible participants, whereas SCG-3D includes everyone present, requiring inference about off-screen targets.SCG-2D is a social version of video-based gaze following [37]. SCG-3D could combine a spatial layout map with Gaze360, potentially using binaural audio to help locate unseen targets.Source: §I.6 · p. 66
• Utterance Target Prediction extends TTM to all recipients, while transcript-based variants add time-stamped words to support inference about conversational targets.Existing annotations distinguish multiple addressees, information ignored by simpler TTM; UTP also represents no recipient or recipients without cameras. Transcript-assisted variants such as TTM-T could exploit dialogue cues for both utterance targets and social gaze.Source: §I.6 · p. 66

I.7 Contributions statement

• James M. Rehg, Hanbyul Joo, and Mike Zheng Shou co-led the social benchmark, supported by contributors responsible for collection, annotation, modeling, and privacy infrastructure.David Crandall, Hyun Soo Park, and Fiona Ryan led collection at Bloomington, Twin Cities, and Atlanta, respectively. Weslie Khoo developed Bloomington’s multi-camera synchronization and de-identification pipelines.Source: §I.7 · p. 66

Future Locomotion Movements Prediction

• Locomotion forecasting predicts multiple plausible future trajectories on a 2D ground plane from past egocentric images.The representation assumes walking over a major plane. Ground truth comes from structure-from-motion camera trajectories projected onto a RANSAC-estimated ground plane.Source: Future Locomotion Movements Prediction · pp. 67, 69, 70

Future Hand Prediction

• Future hand prediction estimates both hands’ positions at contact, pre-condition, and three earlier keyframes from video ending 1.5 seconds before pre-condition.Earlier keyframes precede pre-condition by 0.5, 1, and 1.5 seconds; contact is the first touch, while pre-condition precedes object state change.Source: Future Hand Prediction · p. 67

Short-Term Object Interaction Anticipation

• Short-term anticipation predicts next-active-object boxes and noun classes, future actions, and time to contact from video observed up to a stopping time.Models may return arbitrary numbers of confidence-scored predictions to represent multiple objects and plausible futures. Objects are annotated at contact and in preceding frames sampled at fixed intervals.Source: Short-Term Object Interaction… · pp. 67, 68, 71

Long-Term Action Anticipation

• Long-term anticipation predicts multiple plausible, ordered sequences of future verb–noun actions rather than interaction times or object boxes.Stopping times mark the end of annotated interactions; targets are the first Z subsequently annotated actions.Source: Long-Term Action Anticipation · pp. 68, 69, 71

Future Locomotion Movements Prediction

• Over 10 seconds long, locomotion clips require substantial translation, surrounding-scene visibility, and accurate 3D reconstruction.Glass-mounted cameras approximately align with the wearer’s view; frames are sampled every second. Participant-based training/validation/testing splits are approximately 70%/15%/15% across indoor, outdoor, and mixed scenes.Source: Future Locomotion Movements Prediction · pp. 69–70

Remaining Tasks

• Forecasting clip selection prioritizes applicable activities while distributing annotation hours across universities to preserve geographic diversity.Eligible participants contribute at least 15 minutes and scenarios at least 20 minutes; videos are divided into five-minute clips. Sampling proceeds by participant, video, and clip, limiting repetitive scenarios to two selected clips per video.Source: Remaining Tasks · p. 69, Annotation analysis · p. 71

Future Hands Movements Prediction

• Hand-position ground truth uses left/right-labeled bounding-box centers across the five interaction keyframes targeted by future hand prediction.Annotations cover contact and the preceding keyframes, preserving which hand each box represents.Source: Future Hand Prediction · p. 67, Future Hands Movements Prediction · p. 70

Annotation analysis

• 120 hours of annotated video span 53 scenarios, 7 universities, and 406 participants, supporting activity and geographic diversity.Geographic diversity also occurs within scenarios, capturing how similar activities differ across locations.Source: Annotation analysis · p. 71
• 0.89 NPMI for “fill tire →close valve” exemplifies temporal action structure, alongside repetitive sequences and other multi-object transitions.Normalized Pointwise Mutual Information (NPMI), following prior work, measures action-transition structure that can inform long-term anticipation. Training/validation/test splits contain 40%/30%/30%, assigned by five-minute clips, with test annotations withheld for server-based evaluation.Source: Annotation analysis · pp. 71–72

Future Locomotion and Hands Movements Prediction

• Locomotion evaluation uses K best mean trajectory error (K-MTE) and probability of correct trajectory (PCT); hand evaluation uses mean and contact keyframe displacement errors.Locomotion evaluation accounts for missing ground-truth positions, while PCT counts retrieved trajectories within an error tolerance. Hand metrics measure visible keyframe positions (M.Disp.) and contact-frame positions (C.Disp.).Source: Future Locomotion and Hands Movements… · p. 72

Short-Term Object Interaction Anticipation

• Short-term evaluation uses Top-K mAP with IoU > 0.5, discounting false positives to accommodate multiple plausible next objects.Matching greedily prioritizes confidence and overlap, allowing each ground-truth object only one match. Different matching criteria assess interaction granularity, with K = 5 and time-to-contact tolerance Tδ = 0.25.Source: Short-Term Object Interaction… · p. 73

Long-Term Action Anticipation

• ED@20 with K = 5 evaluates future sequences using best-of-many Damerau-Levenshtein distance, separately scoring verbs, nouns, and joint actions.Insertions, deletions, substitutions, and transpositions accommodate ordering errors while retaining sequence-sensitive evaluation. Z = 20 is chosen because baselines begin predicting randomly at longer horizons.Source: Long-Term Action Anticipation · pp. 73–74

Future Locomotion Movements Prediction

• The locomotion baseline uses the method of Park et al., combining KNN trajectory retrieval, AlexNet image features, and egocentric polar-coordinate trajectories.Trajectories encode ground-plane distance and direction relative to the wearer’s feet.Source: Future Locomotion Movements Prediction · p. 74

Future Hands Movements Prediction

• Future hand prediction regresses hand positions at five future keyframes from I3D video features, masking unobserved hands during weighted smoothed L1 training.The regression head comprises two linear operations and uses 2s of observation; experiments set β = 5.Source: Future Hands Movements Prediction · p. 74

Short-Term Object Interaction Anticipation

• Short-term anticipation combines Faster R-CNN next-object detection with SlowFast [71] predictions of verbs and time to contact.Experiments retain contact-frame annotations, merge semantically similar labels, and remove null, inconsistent, non-object, and infrequent noun classes. The resulting dataset contains 64,798 examples spanning 87 nouns and 74 verbs.Source: Short-Term Object Interaction… · pp. 74–75

Long-Term Action Anticipation

• Long-term baselines combine SlowFast [71] or MViT [63] encoders, concatenation or self-attention aggregation, and No Change or MultiHead decoders.No Change repeats the current action, whereas MultiHead independently predicts each future step. Sampling predicted distributions produces multiple plausible sequences, and training sums losses across future-step predictions.Source: Long-Term Action Anticipation · pp. 75–76

Future Locomotion Movements Prediction

• 8.81m for 1 −MTE and 0.39 for PCTϵ=3m characterize the KNN baseline over trajectories lasting 7–15 seconds.Visual cues indicate future movement, but global image features lack sufficient detail for complex walking behavior such as avoiding pedestrians.Source: Future Locomotion Movements Prediction · pp. 77, 79

Future Hands Movements Prediction

• 52.98/53.68 M.Disp. and 56.37/56.17 C.Disp. are the test-set errors for left/right hand positions, with contact-frame prediction harder than other keyframes.Contact frames require a longer anticipation horizon. Drastic head movements and stochastic futures limit predictions; explicitly modeling head motion and next-active objects is suggested as a potential improvement.Source: Table 35 · p. 77, Future Hands Movements Prediction · pp. 77, 79

Short-Term Object Interaction Anticipation

• 2.45% test and 2.07% validation overall Top-5 mAP were achieved by FRCNN+SF, outperforming FRCNN+Rnd. when predicting verbs and time to contact.Uncertain futures and difficulty detecting next-active objects limit performance, especially in complex scenarios.Source: Table 36 · p. 77, Short-Term Object Interaction… · pp. 77, 79

Long-Term Action Anticipation

• Transformer aggregation of multiple input clips yields the best long-term anticipation performance, while explicit multihead training improves verb, noun, and action predictions.The No Change baseline performs poorly. Replacing SlowFast with MViT improves verb forecasting but worsens noun forecasting despite similar Kinetics action-classification performance.Source: Long-Term Action Anticipation · p. 78, Table 37 · p. 79

Data Annotation

• A large taxonomy, ambiguous bounding boxes for “stuff,” and unclear tool–object roles complicate forecasting annotations.OTHER labels were manually mapped where possible; future annotation will request the closest taxonomy item alongside free-form labels. Narrators determine which interaction is labeled, guiding tool and object assignments.Source: Data Annotation · p. 79

Long-Term Action Anticipation

• Ego4D action-recognition pretraining substantially improves downstream forecasting, and increasing past context improves transformer-based models at the cost of memory.Pretraining helps address differences in viewpoint and activity diversity between Kinetics and Ego4D. Eight input clips is the maximum that fits in GPU memory.Source: Long-Term Action Anticipation · pp. 79–80, Table 38 · p. 80, Table 39 · p. 80

Long-Term Action Anticipation; J.8 Contributions statement

• Z = 20 is the evaluation horizon because longer forecasts approach random prediction; anticipating immediately next actions is easier, with performance eventually plateauing.Forecasting Benchmark leadership is credited to Giovanni Maria Farinella and co-leader Rohit Girdhar for task definition, collection, and paper writing.Source: Long-Term Action Anticipation · p. 80, §J.8 · p. 81

Long-Term Action Anticipation

• K = 5 candidate futures are evaluated best-of-five, with candidates generated by sampling classifier heads independently.The authors propose beam search, explicit modeling of multimodal futures, and sequence-aware decoders as directions for improvement.Source: Long-Term Action Anticipation · p. 80

K. Societal Impact

• Ego4D supports video-understanding research through a diverse, large-scale resource collected under rigorous privacy and ethics standards and benchmarks intended to promote reproducible advances.Potential applications include assistive technology, education, fitness, entertainment and gaming, eldercare, robotics, and augmented reality.Source: K. Societal Impact · p. 82
• Guarding against misuse requires wearable-camera notice and consent protocols, controls over data use, storage, and sharing, and model controls to remove or obscure sensitive content.These safeguards address public-space recording and downstream tasks such as speech transcription.Source: K. Societal Impact · p. 82
• Audio-visual and social benchmarks use fully consenting participants, while dataset access requires a license specifying allowable uses to discourage harmful applications.Consent covers unblurred faces and conversational audio; the real-world, diverse data respects privacy protocols across countries.Source: K. Societal Impact · p. 82
• Ego4D could inspire less-careful collection efforts; the authors address this risk through comprehensive procedural documentation and planned dissemination of best-practice recommendations.The concern is that future collectors may not give privacy and ethical issues the same attention.Source: K. Societal Impact · p. 82
• Remaining dataset imbalances motivate expanding collaborations with researchers and participants in underrepresented areas, while full global coverage of daily activities remains elusive.Rwanda contributes relatively little data, and coverage of 74 cities does not encompass all demographics.Source: K. Societal Impact · p. 82