EgoLCD: Egocentric Video Generation with Long Context Diffusion

1. Introduction

• Autoregressive models use KV caching for efficiency but remain vulnerable to content drift, gradually losing object identity, appearance, and scene semantics.The problem is especially consequential for egocentric video with high-variance camera motion, complex hand-object dynamics, and strict procedural logic.Source: §1 · p. 2
• EgoLCD treats long-form egocentric video generation as robust, efficient memory management and combines low-level memory operators with higher-level reasoning.The framework is designed to generate long, logically coherent, computationally tractable sequences from first-person interactions.Source: §1 · p. 2
• EgoLCD leads on all key EgoVid benchmark metrics and shows exceptional temporal consistency under a new protocol for assessing long-term stability.The protocol addresses the limitation of average quality scores, which can fail to capture content drift in long sequences.Source: §1 · p. 2
• The framework pairs a Long-Term Sparse KV Cache with limited-window attention enhanced by LoRA, and uses Memory Regulation Loss to align historical memory with new representations.Structured Narrative Prompting supplies temporally ordered detailed captions to guide complex action sequences.Source: §1 · p. 2
• EgoLCD is trained in two stages using general video corpora and egocentric datasets, and the authors report its potential as a high-performance egocentric video world model.The authors identify the need for an evaluation protocol that better measures long-term temporal stability.Source: §1 · p. 2

2.1. Video Generation as World Simulators

• World simulators are framed as models that simulate future outcomes in response to actions, while egocentric datasets capture fine-grained human-object interactions relevant to complex tasks.The text distinguishes these datasets from driving simulators’ macro-level interactions and notes their relevance to manipulation policies.Source: §2.1 · pp. 2–3

2.2. Long Video Generation

• Long video generation faces quadratic self-attention costs, motivating autoregressive KV caching and optimizations such as cache sharing, denoising compression, quantization, and layer-wise allocation.Other approaches target content drift through motion-guided losses, optical-flow propagation, temporal correlation structuring, or blending low- and high-frequency features.Source: §2.2 · p. 3
• Recent work models long-range dependencies with Mixture of Contexts and Long Context Tuning; EgoLCD contributes diffusion for long-context egocentric video generation.The authors also identify structured narrative prompting and memory regulation loss as approaches to generation quality and continuity.Source: §2.2 · p. 3

3.1. Overview

• EgoLCD generates long egocentric videos block by block, conditioning each new segment on preceding context within a semi-autoregressive long–short memory framework.The framework addresses costly context propagation and the challenge of balancing long-term coherence with short-term adaptation.Source: §3.1 · p. 3

3.2. Long-Short Memory

• The Long-Term Sparse KV Cache retains important historical key–value pairs, while limited-window attention and LoRA adapt to transient scene changes.The sparse cache serves distant dependencies; LoRA enhances short-term attention as an implicit memory unit.Source: §3.2 · pp. 4–5
• Probe-based importance scores select sparse tokens, and semantic retrieval fuses historical key–value pairs, yielding sub-linear memory complexity with training–inference consistency.The framework uses a consistent sparse-caching strategy in training and inference.Source: §3.2 · pp. 4–5
• Memory Regulation Loss builds a semantic anchor from retrieved, temporally resampled history and regularizes predicted velocity toward a memory-conditioned target.Using sparse memory retrieval during both training and inference is reported to improve memory stability and reliability.Source: §3.2 · pp. 5–6

3.3. Structured Narrative Prompting

• Structured narrative prompting uses GPT-4o captions for coherent 5-second clips and retrieves semantically similar prior prompts to guide later segments.The prompts provide segment-level guidance and support visual consistency and narrative continuity across the video.Source: §3.3 · p. 6

4.1. Datasets and Evaliation Metrics

• The evaluation combines VBench quality measures with NRDP metrics, which compare later chunks with the first chunk and weight earlier drift more heavily.Videos are split into 10 uniform chunks; NRDP is applied to Aesthetic, Motion, and Clarity, with Clarity derived from Imaging Quality.Source: §4.1 · p. 7

4.2. Implementation Details

• Training uses a two-stage sequence: general videos establish appearance and motion priors, then egocentric data specialize the model with memory components enabled.The general-video collection contains 1,000 videos, and Ego4D supplies 3,670 hours of egocentric activity video.Source: §4.1 · p. 6, §4.2 · p. 7

4.3. Evaluation and Analysis on General Long Video Generation

• EgoLCD is compared with MAGI, Self-Forcing, Framepack, and SkyReels-v2 on perceptual quality and NRDP measures.Optimization uses a hybrid rectified-flow, MAE reconstruction, and memory-regulation loss; training takes approximately 50 hours on 8×H100 GPUs.Source: §4.3 · p. 7, §4.2 · p. 7

4.4. Performance Analysis in Egocentric Video Generation

• With EgoVid training, EgoLCD outperforms SVD, DynamiCrafter, and OpenSora across all six egocentric evaluation metrics.The reported metrics include CD-FVD, semantic and action consistency, clarity, motion smoothness, and motion strength.Source: Table 2 · p. 8, §4.4 · p. 8

4.3. Evaluation and Analysis on General Long Video Generation

• EgoLCD achieves the best performance across all five NRDP dimensions, and ranks first in Background Consistency, Subject Consistency, Imaging Quality, and Motion Smoothness, though it trails MAGI in Aesthetic Quality.Removing Memory Regulation Loss degrades consistency performance.Source: §4.3 · p. 8

• Sparse memory still requires substantial GPU resources, while fixed 5-second segments limit the temporal horizon.Source: §5 · p. 8
• Dependence on precisely aligned text makes the method susceptible to noise-induced semantic drift, and automated metrics may not reflect user preferences.Source: §5 · p. 8
• Planned work targets generation-length constraints, noise-resistant narrative comprehension, and evaluation systems better aligned with human perception.Source: §5 · p. 8

6. Conclusion

• EgoLCD reframes long-video generation as memory management through a unified long–short memory design for extended visual and semantic consistency.The design pairs global structure with adaptation to changing first-person scenes.Source: §6 · p. 8
• The Long-Term Sparse KV Cache maintains global structure, while LoRA-enhanced short-term attention captures fine-grained motion and appearance changes.Source: §6 · p. 8
• Memory Regulation Loss aligns memory usage with historical context, and Structured Narrative Prompting guides temporal progression across multi-stage actions.Source: §6 · p. 8
• On EgoVid-5M, evaluations show gains in perceptual quality, semantic stability, and motion coherence while mitigating generative forgetting and temporal drift.Source: §6 · p. 8

Supplementary Material

• The supplementary material provides detailed algorithmic procedures for the proposed EgoLCD framework.Source: A. Algorithm Details · p. 11
• The procedures elaborate the EgoLCD framework described in the paper.Source: A. Algorithm Details · p. 11
• The supplementary procedures correspond to the methodology in Section 3.Source: A. Algorithm Details · p. 11

A.1. Inference Pipeline

• At inference, EgoLCD segments the script into coherent 5-second clips with detailed GPT-4o captions, retrieves historical context by cosine similarity, and conditions generation on the previous clip when t > 1.After decoding a clip, it sparsely compresses Kt and Vt, then appends (Et, Ksparse, Vsparse) to H.Source: §A.1 · p. 11

A.2. Dual-Memory DiT Block

• The DiT block combines LoRA-enhanced local attention for short-term adaptation with retrieved sparse key-value pairs for long-term context.The retrieved pairs are concatenated with local features before attention.Source: §A.2 · p. 11

A.3. Sparse KV Compression

• Sparse KV compression scores tokens using probe-token attention weights, normalizes scores by position, and prunes by threshold to retain critical key-value pairs.Source: §A.3 · p. 12

B. Ablation Study

• All ablation variants, including the baseline, use block-wise generation and Structured Narrative Prompting, controlling for those shared components.Source: B. Ablation Study · pp. 12–13, Table 3 · p. 13
• 0.0844 NRDP-Subject is reported for full EgoLCD, compared with 1.8292 for the no-cache baseline, which also has 0.9323 NRDP-Background.All variants use block-wise generation and Structured Narrative Prompting; the caption identifies Table 3 as the component ablation.Source: B. Ablation Study · p. 13, Table 3 · p. 13
• 1.1124 vs. 0.2945 NRDP-Background is reported for EgoLCD without Memory Regulation Loss and full EgoLCD, respectively.The no-loss variant includes the Sparse KV Cache; the study attributes gains to architecture components rather than the shared inference framework or prompting.Source: B. Ablation Study · p. 13, Table 3 · p. 13

C. Implementation Details

• Training uses 8 NVIDIA H100 GPUs with FSDP and bf16 precision, AdamW at a 1×10−5 learning rate, and a 200-step warmup.The setup also uses 1×10−4 weight decay and EMA with a 0.99 decay rate.Source: C. Implementation Details · p. 13
• Inference generates 81-frame segments at 480 × 848 resolution, conditioning each new segment on the preceding segment’s last 9 frames.The sequence parallel degree is 4, with a maximum sequence length of 75,600 tokens.Source: C. Implementation Details · p. 13
• Inference uses 20 Rectified Flow sampling steps with a guidance scale and sample shift of 5.0, while reusing cached attention key-value states.The stated purpose of caching is to maintain long-term consistency and computational efficiency without recomputation.Source: C. Implementation Details · p. 13

D. Structured Narrative Prompting (SNP) Details

• Training and inference use distinct prompting structures: dense frame alignment for training and temporal narrative guidance for inference.The stated rationale is to address the different requirements of the two phases.Source: D. Structured Narrative Prompting… · p. 14

D.1. Training Phase: Dense Frame-Level Anchoring

• Training prompts use dense, exhaustive descriptions aligned with specific frame ranges to teach fine-grained visual-text correspondences.The described correspondences include visual features such as texture and object placement.Source: §D.1 · p. 14
• EgoVid-5M training examples describe scene details such as blue tiles, a stainless steel sink, and utensils to anchor scene identity across clips.Table 4 is described as containing prompts strictly aligned with frame indices and dense scene details.Source: §D.1 · p. 14, Table 4 · p. 14
• The training examples are real EgoVid data, with dense scene details intended to mitigate forgetting.The table caption describes prompts aligned with frame indices.Source: Table 4 · p. 14

D.2. Inference Phase: Temporal Narrative Guidance

• Inference prompts are organized by time segments and guide temporal progression, including lighting changes, while locking attributes such as clothing consistency..Source: §D.2 · p. 14
• The presenter example guides the environment from dusk to deepest night while keeping blonde hair and a grey blazer consistent across all 12 segments.Table 5 describes a continuous 60-second generation task guided by 12 sequential prompts.Source: §D.2 · p. 14, Table 5 · p. 15