NextMe-800: Anticipating Personal Behavior from Months of Egocentric Video

1 INTRODUCTION

• Reliable naturalistic previews require sustained, detailed observations of routines and an evaluation of predictions against actual behavior, from immediate actions to broader activities.The motivating use is helping people formulate more realistic plans based on plausible trajectories from personal history.Source: §1 · p. 2
• Dense egocentric observation captures brief actions, interruptions, and transitions, while months of recording reveal recurrence across contexts and changes in routine.Existing resources address different needs, including broad diversity, multi-person observation, long-term memory evaluation, and digital-only personalized prediction.Source: §1 · p. 2
• The paper argues that forecasting needs dense sampling, extended temporal coverage, documented recording gaps, and behavior represented from immediate actions to broader activities.Longitudinal depth within one person complements diversity across individuals by enabling repeated observations across situations and time.Source: §1 · p. 2
• NextMe-800 and NextAct combine months-long personal recordings with fixed evaluation points for predicting the next K recorded events at a chosen abstraction level.NextAct combines points from NextMe-800 and EgoLife; unobserved gaps are not treated as known behavior.Source: §1 · p. 2
• The authors validate automatic scorers against human rankings before using semantic similarity as the substitution cost in edit distance for open-vocabulary event sequences.This addresses the tension between exact string matching that rejects paraphrases and broad semantic similarity that can reward the wrong action.Source: §1 · p. 2
• The contributions include a months-long egocentric resource, multi-resolution open-vocabulary anticipation, an embedding-based sequence metric, and benchmarking across predictors and prediction horizons.The benchmark analysis also examines how predictability varies with abstraction level.Source: §1 · pp. 2–3

Long-horizon action anticipation

• Earlier action anticipation commonly predicts verb–noun sequences from a fixed taxonomy, while personalized open-vocabulary prediction from one month of phone-screen actions covers only digital interactions [Grauman et al., 2022] [Shaikh et al., 2026].NextMe-800 offers a longer record of everyday physical and digital activities and five abstraction levels for studying whether weeks or months of history help beyond immediate context.Source: §2 · p. 3

Longitudinal egocentric datasets

• Longitudinal resources trade off duration, sampling density, and participant breadth; NextMe-800 combines 1 Hz images, gaze, and audio with abstraction levels spanning atomic actions to major activities.The paper contrasts sparse long-span recordings, dense multi-person recordings lasting up to a week, and month-long personal-memory benchmarks with sparser sampling.Source: §2 · p. 3

Ultralong video understanding and behavior simulation

• Behavior simulations can be plausible without demonstrating accurate prediction of an individual’s future actions, whereas this dataset enables direct comparison with observed future behavior.Evaluation uses an embedding-based metric normalized against random baselines.Source: §2 · p. 3

Collection and design choices

• The collection records 795.5 hours over 126 calendar days, averaging 6.3 hours daily, with 1 Hz RGB at 2880 × 2880 resolution plus continuous gaze and audio.The recordings total 8.5 TB.Source: §3 · p. 3
• Recording spans from shortly after waking until bedtime but includes intermittent pauses, averaging 4.9 sessions per day; 18 calendar days have no recording.Multi-step predictions bridge both within-day session gaps and missing days.Source: §3 · p. 3

Why this temporal scale?

• Figure 2 summarizes daily recording hours and annotation scale, including event counts, mean duration, and compression ratios across abstraction levels.Days without recordings are retained in the daily coverage display.Source: Figure 2 · p. 4

What activities does the collection cover?

• The collection covers recurring routines and long-tail events across physical environments, social contexts, cognitive tasks, and personal interests.Examples range from research, meals, and guitar practice to flights, squash, and observing marine animals.Source: §3 · p. 4

Why this temporal scale?

• NextMe-800 achieves 39.5% awake-time coverage across 126 calendar days, combining months-long depth with 1 Hz video, gaze, and audio for one individual.Awake-time coverage assumes 16 waking hours per day; EgoLife has comparable awake-hour coverage but spans seven days, while other resources are sparse or domain-limited.Source: §3 · p. 4

Privacy-aware processing

• The de-identification pipeline masks detected faces and identified sensitive text, and transforms voice characteristics; subtle contextual cues may still persist in a months-long record.Behavior-descriptive screen text is retained while identified sensitive spans are masked.Source: §3 · pp. 4–5

Dense captioning

• Each 30-second captioning window combines 1 Hz frames overlaid with gaze, OCR screen text, aligned speech, and an additional gaze-fixation crop for small on-screen text.Gemini-family models produce first-person action descriptions and environment descriptions that record scene state changes.Source: §4 · p. 5
• An audit found that 88.25% of source segments received no error flag; remaining errors mostly involved minor perceptual details rather than the core action.Four annotators audited 613 caption segments, with injected erroneous captions used as quality controls.Source: §4 · p. 5

Hierarchical caption tree

• Lower abstraction levels preserve more information but are noisier and less predictable, while higher levels reduce autoregressive steps at the cost of detail.The hierarchy allows granularity to be selected to balance noise, information content, and compounding error.Source: §4 · p. 5
• The five-level tree recursively groups events by shared semantic goal shifts without fixed duration bounds, producing single-objective parent captions and a multi-resolution view of behavior.Adjacent-level compression ratios range from 1.6× to 11×; the hierarchy supports switching between atomic detail and routine structure by prediction horizon.Source: §4 · pp. 5–6

Action-level similarity

• Table 2 reports action-level agreement and score entropy on 350 annotated points, with LLM judges in its middle block.The table caption specifies 95% confidence intervals from 2,000 bootstrap resamples in Appendix B.Source: Table 2 · p. 6

Atomic action similarity

• Four annotators rank candidate actions, and pairwise accuracy measures whether a metric respects their strict preferences; normalized score entropy measures score discrimination.Low entropy indicates scores compressed into a narrow band, limiting their usefulness as graded substitution costs.Source: §5 · p. 6
• The validation set pairs 350 ground-truth actions with four candidates, combining real model predictions and controlled rewrites to cover a broad similarity spectrum.Real predictions cluster at low similarity, so rewrites fill the upper range.Source: §5 · p. 6
• H10 = 0.9724 for Qwen3-Embedding-8B, whose scores span the range; Qwen3 reranker has pairwise accuracy 0.9417 but H10 = 0.6429, motivating the embedding metric.Embedding scoring also needs O(K) forward passes per sequence pair, versus O(K^2) LLM-judge calls.Source: §5 · p. 6

Soft edit distance

• Soft edit distance aligns variable-length action sequences using insertion and deletion costs of 1 and graded substitution costs based on embedding cosine similarity.The minimum-cost alignment is normalized; at K=1, the score reduces to clipped cosine similarity and is normalized against a random-prediction baseline.Source: §5 · p. 7

The NextAct benchmark

• The benchmark uses 1,000 NextMe-800 points and 500 EgoLife points, with recorded target windows and three candidate trajectories scored by their best match.Target windows account for recording gaps and events ranging from seconds to hours, supporting fair alignment across evaluation points.Source: §6 · p. 7

Predictor performance

• At K=1, frontier predictors score 0.32–0.34 versus repeat-last 0.19; at K=10 they fall to 0.18–0.19, with Gemini and GPT leading the respective horizons.Qwen3.5-4B falls below repeat-last at K=10; Table 3 reports normalized embedding scores and reranker comparisons.Source: §6 · pp. 7–8, Table 3 · p. 8

Context scaling

• Fine-grained predictions saturate quickly and decline with longer history, while coarser predictions benefit from longer context but eventually saturate.A 50-event context is typically near the best-performing range and is used by default in subsequent in-context experiments.Source: §6 · p. 7, Figure 5 · p. 8

Retrieval and task switches

• Retrieval-augmented Codex + DeepSeek does not lead at either horizon, suggesting that selecting relevant history is difficult; caption-based prediction also omits visual cues behind task switches.The paper suggests visual grounding may help anticipate transitions triggered by what a person sees rather than by narrative continuation.Source: §6 · pp. 8–9

Predictability by abstraction level

• Across abstraction levels, scores peak at intermediate granularity; the K=10 decline at L4–L5 is sharper because ten L5 events span 5.8 hours.At K=1, scores peak around L3–L4; abstraction filters incidental detail, while very coarse events connect less-continuous activities.Source: Figure 6 · p. 9, §6 · p. 9

Discussion

• NextMe-800 records one person’s daily life over four months, capturing everyday physical and digital activities as they unfold.The dataset is intended to support study of recurring routines, contextual variation, and change over time.Source: §7 · p. 9
• The paper combines a five-level caption hierarchy and open-vocabulary soft edit distance with a 1,500-point benchmark of repeat-last, frontier LLM, and retrieval-augmented predictors.Source: §7 · p. 9
• Models achieve modest scores, and performance consistently declines as the prediction horizon lengthens; strongly personalized anticipation remains challenging even for frontier models.The authors position the dataset and evaluation framework as a foundation for future study, with better-informed personal decisions as a longer-term goal.Source: §7 · p. 9

AI use statement

• Generative AI assisted with drafting and editing the manuscript and LaTeX authoring.Source: AI USE STATEMENT · p. 10
• AI assistance also included finding and verifying related work and writing analysis and plotting scripts.Source: AI USE STATEMENT · p. 10
• Model-generated captions and predictions are part of the methodology, and the authors reviewed all AI-assisted content.Source: AI USE STATEMENT · p. 10
• The authors state that they remain responsible for all claims and the final submission.Source: AI USE STATEMENT · p. 10

• The volunteer consented to continuous recording of daily life for research purposes.Source: ETHICS STATEMENT · p. 10
• Faces, voices, personal speech, and sensitive on-screen text are de-identified before analysis or model processing.Source: ETHICS STATEMENT · p. 10
• Recorded individuals may request withdrawal, after which corresponding segments are removed from the dataset and derived materials in subsequent releases.Source: ETHICS STATEMENT · p. 10
• The collection was exempt from full institutional ethics review because it involved one consenting adult and third-party information was de-identified before analysis.Source: ETHICS STATEMENT · p. 10

• The recordings, captions, evaluation index, scoring code, and caption and hierarchy prompts are released with the dataset.Source: REPRODUCIBILITY STATEMENT · p. 10
• Numerical tables are reconstructed from saved experimental artifacts.Source: REPRODUCIBILITY STATEMENT · p. 10
• Model predictions depend on commercial API endpoints whose behavior may change over time.Source: REPRODUCIBILITY STATEMENT · p. 10

• After excluding one annotator with low control-detection performance, 88.25% of source segments received no error flag and 3.36% were flagged by multiple annotators.The audit sampled 613 caption segments from 180 fifteen-second clips spanning 44 days and 78 recordings.Source: A CAPTION AUDIT AND HIERARCHY… · p. 14, Table 4 · p. 14
• About 20% of presented segments were injected erroneous captions as quality controls; typical errors involved minor perceptual details rather than core behavior.Source: A CAPTION AUDIT AND HIERARCHY… · p. 14
• The audit reports five error categories, and the categories can overlap.Source: A CAPTION AUDIT AND HIERARCHY… · p. 14, Table 5 · p. 14
• The hierarchy keeps L1 immutable and represents each generated parent with child references and a goal-oriented description; validation checks complete unique coverage.Parent timestamps span the minimum child start to maximum child end, so overlapping children can yield overlapping parent envelopes.Source: A CAPTION AUDIT AND HIERARCHY… · p. 14
• In a 3 h 37 min example, the hierarchy compresses 2,287 L1 events to 178 L2 events and 2 L5 events; higher levels are more abstract but lose incidental information.Lower levels retain more detail but also more noise.Source: Table 6 · p. 14

B METRIC PROTOCOL

• The reranker concentrates 69.6% of scores at the endpoints and has H10 = 0.6429, while embeddings have H10 = 0.9724, 3.6% endpoint mass, and fewer near-ties.Despite higher accuracy, the reranker compresses within-group distinctions; embeddings provide more graded separation. Human–human pairwise agreement is 0.8619 (95% CI: 0.8507–0.8732); normalized scores use random-prediction baselines, with 0 at chance and 1 at a perfect match, applied pointwise before averaging.Source: B METRIC PROTOCOL · p. 15, Table 7 · p. 15

C CONTEXT SCALING FOR DEEPSEEK

• The DeepSeek-v4-flash context-scaling test reports single-run means without repeated-generation error bars and marks trend-estimated scores at 50 history events.The figure uses the same axes as Figure 5.Source: Figure 7 · p. 16

D PREDICTION EXAMPLES

• High-scoring examples continue an ongoing activity or familiar routine, while low-scoring examples miss transitions and extend the task in progress.The examples use Gemini 3.1 Pro at K=1 and GPT-5.6-sol at K=10, selecting the highest-scoring of three candidates; the K=10 L3 example scores 0.49 and the L2 example scores 0.00.Source: D PREDICTION EXAMPLES · p. 16, Table 9 · p. 16, Table 10 · p. 17

E ADDITIONAL SMALLER-MODEL SETTINGS

• Qwen3.5-4B and Gemma4-12B-it use hosted APIs and the same frozen evaluation points, contexts, prompts, embedding model, baselines, and best-of-three scoring as other predictors, but structured-output differences qualify cross-model comparisons.Both return three candidates for all 1,500 points at both horizons using JSON-schema-constrained output, temperature 0.7, and an 8,192-token output limit; Qwen runs without thinking mode, and reranker scores were not computed.Source: E ADDITIONAL SMALLER-MODEL SETTINGS · p. 17

F.1 FORECASTING TEMPLATE

• The forecasting template requests exactly three chronological candidate trajectories of K timestamped actions, aligned to target windows and context granularity, while skipping unrecorded gaps.Candidates have independent plausibility values in descending order that need not sum to one, and predictions include environmental observations when appropriate.Source: §F.1 · p. 18

F.2 EMBEDDING INSTRUCTION

• The embedding instruction prioritizes core verb-object alignment and downweights modifiers, tense, style, and incidental context.Source: §F.2 · p. 18

F.3 GOAL-BASED HIERARCHY PROMPT

• The hierarchy prompt generates one level at a time, grouping L1 briefs through L5 major activities into evidence-supported goals while preserving child coverage and meaningful boundaries.It prohibits inventing unsupported outcomes, treats behavior-changing triggers as standalone only when they redirect a major goal, and requires each input unit to have exactly one parent at the next level.Source: §F.3 · pp. 18–19

F.4 DENSE CAPTIONING PROMPT

• The dense-captioning prompt requires first-person, appropriately granular action captions and an exhaustive description of visible surroundings, screens, text, media, and ambient audio.It uses gaze markings to indicate looking direction and asks for specific action and interface details rather than broad summaries.Source: §F.4 · p. 20
• The first segment establishes the full world state, while later segments report only meaningful changes rather than viewpoint shifts or unchanged details.Meaningful changes include screen content, object or people activity, and lighting, location, or ambient-audio changes.Source: §F.4 · p. 21
• Each segment uses a condensed action with one main verb while preserving informative specifics, and speech must remain verbatim in action and speech fields.Segmentation separates speech and environmental observations from actions, while merging continuous or repetitive actions.Source: §F.4 · pp. 22–23

F.5 LLM-JUDGE RUBRIC

• Three prompt variants were used across campaigns, sharing evaluation dimensions of action-level similarity based on verb, object, and temporal alignment, while differing in example language and scoring anchors.Source: §F.5 · p. 23