PaperlayerView exact revision on arXiv

Source-linked AI summary

EgoEdit: Dataset, Real-Time Streaming Model, and Benchmark for Egocentric Video Editing

Runjia Li, Moayed Haji-Ali, Ashkan Mirzaei, Chaoyang Wang, Arpit Sahni, Ivan Skorokhodov, Aliaksandr Siarohin, Tomas Jakab, Junlin Han, Sergey Tulyakov, Philip Torr, Willi Menapace

arXiv:2512.06065v1cs.CVcs.AI

TL;DR

Egocentric video editing for AR faces a domain gap from camera motion and hand-object interactions, as well as latency constraints. The paper introduces a curated dataset, a real-time editor, and an evaluation benchmark for this setting. It reports gains on egocentric editing while remaining competitive on general editing tasks.

  • Problem

    Existing editors and training corpora largely target third-person footage, while egocentric editing involves moving first-person views, hand-object interactions, and a need for low-latency responses.

    Source: §1 · p. 2

  • Method

    The paper curates EgoEditData, trains the instruction-following EgoEdit editor with streaming distillation, and introduces EgoEditBench for egocentric editing evaluation.

    Source: §1 · pp. 2–3, §4.1 · p. 5

  • Results

    EgoEdit improves egocentric editing over recent baselines while performing closely to concurrent state-of-the-art methods on exocentric tasks.

    Source: §1 · p. 3

  • Takeaways & Limitations

    EgoEdit runs at 38.1fps with 855ms first-frame latency on a single H100 GPU, and EgoEditData and EgoEditBench are intended for public release.

    Source: §1 · p. 3

    EgoEdit-RT has lower proficiency on out-of-distribution instructions, less robust editing of temporarily occluded objects, and lower temporal consistency than EgoEdit; EgoEdit's 855ms first-frame latency is suboptimal.

    Source: §6 · p. 8

Abstract

from arXiv · show

We study instruction-guided editing of egocentric videos for interactive AR applications. While recent AI video editors perform well on third-person footage, egocentric views present unique challenges - including rapid egomotion and frequent hand-object interactions - that create a significant domain gap. Moreover, existing offline editing pipelines suffer from high latency, limiting real-time interaction. To address these issues, we present a complete ecosystem for egocentric video editing. First, we construct EgoEditData, a carefully designed and manually curated dataset specifically designed for egocentric editing scenarios, featuring rich hand-object interactions, while explicitly preserving hands. Second, we develop EgoEdit, an instruction-following egocentric video editor that supports real-time streaming inference on a single GPU. Finally, we introduce EgoEditBench, an evaluation suite targeting instruction faithfulness, hand and interaction preservation, and temporal stability under egomotion. Across both egocentric and general editing tasks, EgoEdit produces temporally stable, instruction-faithful results with interactive latency. It achieves clear gains on egocentric editing benchmarks-where existing methods struggle-while maintaining performance comparable to the strongest baselines on general editing tasks. EgoEditData and EgoEditBench will be made public for the research community. See our website at https://snap-research.github.io/EgoEdit

1. Introduction

The paper targets instruction-guided editing as a potential engine for AR, addressing the challenges of egocentric motion, hand-object interaction, and latency. It presents a dataset, editor, and benchmark for this setting, with reported gains on egocentric editing and competitive performance on general tasks.

1. Introduction

  • The paper asks whether instruction-guided editing can let AR users add, remove, or modify scene elements with simple language.This is framed against traditional AR pipelines that require expert effort to handcraft applications.Source: §1 · pp. 1–2
  • Egocentric views bring first-person camera motion, frequent hand occlusions and manipulation, and complex interactions that challenge editors trained for third-person footage.The paper also identifies high latency in many diffusion editing pipelines as incompatible with interactive AR.Source: §1 · p. 2
  • The paper addresses egocentric editing end-to-end with EgoEditData, a real-time editor, and EgoEditBench for assessment.The benchmark targets instruction faithfulness, preservation of hands and manipulated objects, and temporal consistency under egocentric scenarios.Source: §1 · pp. 2–3
  • EgoEdit improves egocentric editing over recent baselines while performing closely to concurrent state-of-the-art methods on exocentric editing.The comparison reports consistent qualitative and quantitative improvements in the egocentric case.Source: Figure 3 · p. 2, §1 · p. 3

2. Related Work

Prior editing work has advanced through paired data, curated corpora, and unified video editors, while video generation methods seek streaming inference to reduce latency. EgoEdit draws on these lines of work to address egocentric editing.

2. Related Work

  • Video editing spans inversion and attention-control methods, paired-data training, and unified in-context editors; approaches remain brittle for large structural edits and long-range consistency.Source: §2 · p. 3
  • Video editing datasets build on synthetic edit pairs, propagated edited frames, and specialized pipelines, with EditVerse [26] curating 232K video-edit samples.Across image and video editing, the literature emphasizes paired-data scale and curation, alongside smaller curated sets where domain alignment and quality can improve results.Source: §2 · p. 3
  • Streaming video generation uses chunk-wise autoregression and causal distillation to reduce latency, with methods including CausVid [74], Self-Forcing [23], and APT2 [36].Self-Forcing rolls out the student during training to address exposure bias.Source: §2 · p. 3
  • EgoEdit follows the Self-Forcing strategy for interactive egocentric video editing, against a backdrop where training-free real-time editors still have a quality gap with trained methods [26] [72].This motivates a curated dataset centered on egocentric motion and complex hand-object interactions that existing video editing datasets lack.Source: §2 · pp. 3–4

3. EgoEditData

EgoEditData is a manually curated egocentric video editing dataset focused on rich hand-object interactions. Its curation pipeline filters real videos, generates and reviews object edits, and yields 99.7k editing pairs.

3. EgoEditData

  • EgoEditData is manually curated for egocentric editing, emphasizing high-quality examples with rich hand-object interactions.Egocentric editing poses challenges from complex interactions, frequent occlusions, and large egomotion.Source: §3 · p. 4

3.1. Data Curation Pipeline

  • The curation pipeline selects real videos with actively manipulated objects, segments hands and objects, generates object replacements or removals, and manually filters edits.It prioritizes quality and domain alignment through strict filtering, and pairs source and target videos with precise natural-language instructions.Source: §3.1 · p. 4

3.2. Statistics

  • 0.4% of the original Ego4D and EgoExo4D videos are retained, yielding 99.7k editing pairs from 10.9k original and 38.8k synthetic videos.The resulting videos total 70 hours, with an average of 3.6 synthetic videos per original video.Source: §3.2 · p. 4

4. EgoEdit

EgoEdit adapts video generation for real-time egocentric editing, combining a pretrained editor with streaming distillation and a benchmark designed for diverse egocentric tasks. Its Self Forcing model reaches 38.1 fps with 855 ms first-frame latency on one H100 GPU.

4. EgoEdit

  • EgoEdit adapts a pretrained video generator for instruction-guided egocentric editing, then distills the editor into a real-time autoregressive generator.The distillation uses bidirectional DMD followed by autoregressive Self Forcing.Source: §4 · p. 5

4.1. Preliminaries: Flow Matching

  • Rectified Flow flow matching trains a deterministic path from noise to data by predicting its velocity from a noised input and time.At inference, an Euler solver integrates the learned ODE from noise to data.Source: §4.1 · p. 5

4.2. Architecture

  • EgoEdit conditions editing on the source video by channel-wise concatenating it with the noisy target before patchification, avoiding sequence-length growth.The model processes the noisy target and source video alongside the text instruction.Source: §4.2 · p. 5

4.3. Distillation Procedure

  • 38.1 fps with 855 ms first-frame latency on a single H100 GPU: the Self Forcing distilled model runs in real time.It generates one chunk at a time, with each chunk comprising three latent frames.Source: §4.3 · pp. 5–6

4.4. EgoEditBench

  • EgoEditBench contains 1,700 source-video/instruction pairs across 15 egocentric editing tasks, with results averaged per task for equal weighting.It samples 100 diverse source videos and includes tasks such as object edits, stylization, reasoning, and video conversions.Source: §4.4 · p. 6

5. Experiments

The experiments compare EgoEdit with video-editing baselines on egocentric and general benchmarks, then examine distillation, dataset contribution, and in-the-wild behavior. Results show stronger egocentric performance, interactive streaming, and remaining limitations in complex real-world edits.

5.1. Experimental Setup

  • Evaluation uses EgoEditBench for egocentric editing and EditVerseBench for non-egocentric editing, excluding reference-conditioned EditVerseBench tasks unsupported by the model.The baselines span attention manipulation, first-frame propagation, instruction-guided video translation, and real-time editing; they use recommended inference settings.Source: §5.1 · p. 7, §5.2 · p. 7

5.2. Comparison to Baselines

  • On challenging egocentric editing, EgoEdit loses only 0.24 VLM points versus 0.83 for Lucy Edit and 0.47 for InsV2V when switching from general editing tasks; EgoEdit-RT achieves markedly stronger performance than real-time streaming editors across both benchmarks.EgoEdit-RT delivers comparable results to the bidirectional full model on all quantitative metrics.Source: §5.2 · p. 7

5.3. Ablations

  • Only the Self Forcing variant achieves sub-second latency compatible with interactive use while retaining VLM scores comparable to the bidirectional teacher.The ablation compares the non-distilled, 4-step DMD-distilled, and final Self Forcing models; Table 2 reports VLM score, latency, and throughput on one H100.Source: §5.3 · pp. 7–8, Table 2 · p. 8
  • EgoEditBench performance steadily improves as more EgoEditData training videos are included.Table 3 evaluates progressively smaller EgoEditData subsets at the 10k-iteration checkpoint.Source: §5.3 · p. 8, Table 3 · p. 8

5.4. In-the-Wild Evaluation

  • In-the-wild examples include preserved hands, environmental and lighting effects, fast-object replacement, and inserted animals that interact with the scene.The authors also report limited structural changes to the surroundings: swords do not cut furniture, and animals do not move real objects.Source: §5.4 · p. 8

6. Discussion

The discussion reports strengths and boundaries of EgoEdit-RT and describes the dataset construction pipeline. The appendix pipeline selects egocentric videos, creates hand and object masks, and uses those masks for editing.

6. Discussion

  • EgoEdit-RT has lower proficiency on out-of-distribution instructions, less robust editing of temporarily occluded objects, and lower temporal consistency than EgoEdit.EgoEdit’s first-frame latency is 855 ms; it operates at 512×384px and 16 fps, described as slightly below common 480p resolution.Source: §6 · p. 8
  • The authors present EgoEditData, EgoEdit, and EgoEditBench as a framework for learning-based AR applications and evaluation under realistic hand–object interactions.The dataset contains 99.7k editing pairs across 49.7k unique videos; the paper describes EgoEdit as real-time and reports strong in-the-wild generalization.Source: §6 · p. 8

A.1. Pipeline Details

  • Video selection filters monocular footage from specified GoPro cameras using jitter and aesthetic scores, then segments hands and objects to produce editing masks.WiLoR detections seed SAM2 hand masks; object masks are generated with Grounded-SAM and refined with SAM2 before VACE-14B edits the masked regions.Source: §A.1 · p. 13

A.2. Dataset statistics

EgoEditData consists mainly of object substitution tasks, with varied target objects and detailed prompts. The section also describes the pretrained text-to-video model used in the work.

A.2. Dataset statistics

  • EgoEditData contains about 99,700 pairs, including 54,164 Change Object pairs, 39,465 Change Object with Special Effects pairs, 3,651 Add Object pairs, and 2,379 Remove Object pairs.The pairs combine a source video, target video, and instruction; 93.6% of videos come from Ego4D and 6.4% from EgoExo4D.Source: §A.2 · p. 13
  • Target objects number 13,632, compared with 3,199 source objects, and prompts average 378 characters; Figure 7 shows object-description word clouds and prompt-length distribution.The passage describes scenario categories as balanced; Figure 8 presents the ten most frequent scenarios.Source: §A.2 · p. 13, Figure 7 · p. 14, Figure 8 · p. 14

B.1. Text-to-Video Model

  • The pretrained video model uses a 10.7B transformer with latent video compression, spatial patchification, text conditioning, and 40-step Euler inference.It is trained at 512 px and uses T5 and CLIP text encoders.Source: §B.1 · p. 14

B.2. Training Details

The training pipeline combines EgoEditData with other video and image editing corpora, then distills the finetuned editor for faster streaming. DMD reduces sampling steps, while Self Forcing enables chunk-wise generation and lowers first-frame latency.

B.2. Training Details

  • Training combines EgoEditData with other editing corpora using sampling weights of 28% for EgoEditData, 52% for other video editing datasets, and 20% for image editing datasets.The additional corpora include publicly available editing datasets and internally generated image and video editing pairs.Source: §B.2 · p. 14

B.3. Distillation Procedure Details

  • DMD reduces generation from 80 model evaluations to four and raises throughput 20×, but full-video sampling still leaves first-frame latency at 6.93 seconds.The original model’s throughput is 11.9 fps, falling to 9.68 fps with autoencoder encoding and decoding; the DMD model reaches 43.5 fps after the autoencoder.Source: §B.3 · pp. 14–15
  • Self Forcing reduces first-frame latency to 855 ms by generating and decoding the first chunk before the full video is complete.The model denoises three latent frames per chunk and uses seven chunks for generation.Source: §B.3 · p. 15

C. Additional Benchmark Details

EgoEditBench covers 15 editing tasks built from 100 egocentric source videos, including object edits, scene changes, reasoning, and video transformations. Its VLM preferences align closely with human preferences against two baselines.

C.1. Evaluation Tasks

  • EgoEditBench samples 100 egocentric source videos across 15 editing tasks, with source videos uniformly sampled across object and scenario clusters.The benchmark also includes reasoning, depth-, sketch-, and pose-based video transformations, as well as combined multi-task instructions.Source: §C.1 · pp. 15–16
  • The benchmark tests object removal, insertion, and replacement, alongside background and camera changes, effects, and stylization.The object-removal and insertion sets each use 50 held-out EgoEditData source videos; Change Object includes 400 evaluation videos.Source: §C.1 · p. 15

C.2. EgoEditBench-Human Alignment

  • VLM and human preferences agree in 86.2% and 84.9% of cases on average, respectively, against LucyEdit and InsV2V.The authors therefore use the VLM score as the benchmark’s main metric.Source: §C.2 · p. 16

D. Evaluation Details

The evaluation uses baseline-specific default inference settings and a shared set of editing, quality, alignment, and temporal-consistency metrics. The VLM editing score is the primary quality signal, supported by alignment with human preferences.

D.1. Baseline Details

  • Baseline comparisons use each method’s default inference settings, including frame count, FPS, resolution, guidance scale, and sampling steps.EditVerse is omitted from EgoEditBench because the authors could not generate the required results with the closed-source model.Source: §D.1 · p. 16

D.2. Metrics Details

  • VLM and human preferences agree in 86.2% and 84.9% of cases on average, respectively, against LucyEdit and InsV2V.The VLM score is consequently used as the primary quality signal.Source: Table 4 · p. 17, §D.2 · p. 17
  • The evaluation measures editing quality with a VLM judge, video quality with PickScore, video-level text alignment, and temporal consistency with CLIP.The authors omit frame-level text alignment and DINO consistency because each is strongly correlated with its corresponding reported metric.Source: §D.2 · p. 17

E.1. Additional In-the-Wild Results

EgoEdit-RT produces varied in-the-wild edits, while qualitative comparisons show baseline failures on egocentric videos. Distilled variants retain similar qualitative performance but have boundary shifts and reduced capacity on complex out-of-distribution instructions; increasing EgoEditData improves editing quality, including on an exocentric benchmark.

E.1. Additional In-the-Wild Results

  • EgoEdit-RT performs varied edits in real time, including object transformations, fluid effects, location changes, animal insertion, and stylization.The examples cover both egocentric and exocentric videos on a single H100 GPU.Source: §E.1 · p. 17

E.2. Additional Comparison to Baselines

  • EgoEdit and EgoEdit-RT handle diverse edits, while concurrent offline and real-time baselines often fail to change egocentric videos or alter more than the instruction requests.Table 5 reports per-category EgoEditBench VLM scores for EgoEdit and baselines.Source: §E.2 · p. 17, Table 5 · p. 18

E.3. Additional Distillation Ablation Results

  • Distilled variants show similar qualitative performance, but EgoEdit-RT sometimes has temporal shifts at chunk boundaries and distilled variants handle complex out-of-distribution instructions less well.Table 6 reports EgoEditBench scores for the distilled variants.Source: §E.3 · p. 17, Chg. BG · p. 18, Table 6 · p. 18

E.4. Additional Dataset Ablation Results

  • Increasing the retained share of EgoEditData improves edit quality and source-video alignment, and raises performance on the mostly exocentric EditVerseBench.The authors speculate that strict quality filtering and descriptive captions account for benefits beyond egocentric editing.Source: §E.4 · p. 18

F. Failed Experiments

Experiments with lightly filtered pairs and template-based instructions performed weakly; the authors report improved editing after extensive curation and more descriptive instructions.

F. Failed Experiments

  • Lightly filtered editing pairs led to weak performance, including failures to replace or add objects, motivating extensive automated and manual data curation.The authors report increased editing performance after curation, at the expense of dataset size.Source: F. Failed Experiments · p. 18
  • Template instructions naming only source and target objects produced low editing performance, while GPT-5 Mini-generated instructions improved instruction following.The authors identify limited variety, short object names, and mismatches between requested and generated objects as issues with the templates.Source: F. Failed Experiments · p. 18
  • On EditVerseBench, retaining more EgoEditData samples increases non-egocentric editing performance.Table 7 compares models trained with different retained amounts of EgoEditData.Source: Table 7 · p. 19

Related papers

Loading 2512.06065v1…