PaperlayerView exact revision on arXiv

Source-linked AI summary

StreamErase: Towards Ultra-Fast Streaming Video Erasing

Zhuobai Dong, Xiaobing Tu, Xiantao Zhang, Jinkui Ren, Yinggui Wang, Yue Ma, Linfeng Zhang

arXiv:2610.20137v1cs.CV

TL;DR

Video erasing remains slow, non-causal, and short-horizon, limiting real-time interaction. StreamErase transfers erasing capability from a bidirectional model to a streaming autoregressive eraser. It reports competitive visual and temporal quality while running more than 46× faster than recent methods.

  • Problem

    Existing video erasing is too slow, non-causal, and short-duration, preventing interactive partial results and limiting extended creative use.

    Source: §1 · p. 2

  • Method

    StreamErase progressively transfers erasing capability from an offline bidirectional diffusion model to an efficient autoregressive generator for real-time interactive erasing.

    Source: §1 · p. 2

  • Results

    StreamErase achieves competitive visual and temporal quality across diverse scenarios and is more than 46× faster than recent methods, reaching 26.06 FPS with sub-second latency on a single H200 GPU.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Users can continuously interact with and guide erasing as results unfold in real time.

    Source: §1 · p. 2

    StreamErase requires users to provide explicit masks, adding interaction cost and limiting flexibility.

    Source: F LIMITATION AND FUTURE WORK · p. 17

Abstract

from arXiv · show

Current video object removal methods suffer from prohibitive latency (minutes per video) and non-causal processing that prevents real-time interaction. We present StreamErase, enabling sub-second latency with up to 26 FPS streaming generation on a single GPU. Our approach first augments a text-to-video model with mask input to generate high-quality videos that erase target objects and synthesize coherent backgrounds, but does not perform inference on the fly. As such, we progressively transfer the erasing capability from a powerful bidirectional foundation model to an efficient unidirectional streaming eraser, enabling long-horizon erasing without sacrificing visual fidelity. Several key challenges arise when generating videos over long, potentially infinite horizons: (1) bridging the domain gap from training on finite length and extrapolating to long horizons, (2) sustaining high quality by preventing error accumulation, and (3) maintaining fast inference, without increasing computational cost as the context grows. To address these challenges, we carefully designed sliding-window causal attention with attention sinks. By incorporating self-rollout with attention sinks and KV cache rolling during training, we simulate inference-time extrapolations with a fixed context window, enabling constant-speed generation of arbitrarily long videos. Finally, we establish a dedicated benchmark, LongErase, for streaming long-video object removal. Our model achieves strong performance compared to existing video object erasing models and generalizes well to real-world video scenarios, uniquely enabling infinite-length streaming. With StreamErase, users can input a mask and see the erasing results unfold in real-time, delivering a truly interactive experience.

1 INTRODUCTION

StreamErase targets slow, non-causal, short-duration video erasing with a streaming autoregressive model distilled from a bidirectional eraser. It reports sub-second latency and real-time generation while maintaining quality on longer clips.

  • Existing methods are too slow, non-causal, and limited to short videos, preventing partial results during processing and constraining extended creative use.Source: §1 · p. 2
  • A bidirectional teacher is distilled through causal autoregression, teacher forcing, and ODE distillation to four denoising steps, then DMD yields a one-step streaming student.Source: §1 · p. 2
  • Attention sinks and rolling KV caches are incorporated during training to simulate inference extrapolation, supporting stable indefinite-length generation at constant latency with fixed context windows.Source: §1 · p. 2
  • 26.06 FPS with sub-second latency on a single H200 GPU; StreamErase is more than 46× faster than recent methods and maintains stable visual and temporal quality from 5-second to 30-second clips.Source: §1 · p. 2

2 RELATED WORK

Related work spans video inpainting, streaming autoregressive generation, and diffusion distillation. The review highlights that existing erasers remain offline, long-horizon streaming can drift, and direct transfer of distillation schemes to general erasing can degrade details.

2.1 VIDEO INPAINTING

  • Inpainting advances from 3D CNNs and flow or homography to attention and diffusion; object erasing must model object effects, yet current diffusion systems remain offline.Source: §2.1 · pp. 2–3

2.2 STREAMING AUTOREGRESSIVE VIDEO MODELS.

  • Streaming autoregressive models use block-wise sequential denoising, causal attention, KV caches, and distillation, but can drift beyond training horizons or require complex long-video finetuning.Source: §2.2 · p. 3

2.3 DIFFUSION DISTILLATION

  • One-step distillation supports efficient video generation, restoration, super-resolution, and portrait animation, but direct transfer to general video erasing degrades high-fidelity details.Source: §2.3 · p. 3

3 METHOD

StreamErase adapts an inpainting backbone for object-aware removal, then distills bidirectional erasing into a causal streaming generator. Attention sinks and rolling caches support long-video extrapolation.

3.1 STREAMERASE

  • The inpainting model receives noisy latents alongside the source video and masks, with the source-video and mask latents conditioning user-specified regions.The source video, background video, and mask are encoded into latent representations before training.Source: §3.1 · p. 3
  • Region Guidance maps CLIP foreground features into the prompt-token space, and cross-attention uses that representation to localize object-related effects.The foreground embedding replaces the prompt's “object” placeholder; attention maps from DiT blocks are aggregated.Source: §3.1 · p. 4, Figure 2 · p. 4
  • Effect Erasing Loss supervises a predicted effect map against a difference mask derived from the original and background videos.The predicted map is upsampled to full resolution and trained with MSE supervision against the downsampled difference mask.Source: §3.1 · p. 4

3.2 THREE-STAGE DISTILLATION PIPELINE

  • A three-stage distillation pipeline transfers erasing capability from a bidirectional teacher through a four-step causal student to a one-step DMD generator.The teacher uses full spatiotemporal attention; the causal student is initialized by regressing on ODE solution pairs sampled from the teacher.Source: §3.2 · p. 5
  • One-step generation conditions on the input video, mask, and noise rather than previous generated latents; KV-cache representations maintain temporal continuity across streaming chunks.This design avoids feeding prediction errors back into future steps.Source: §3.2 · p. 6

3.3 LONG VIDEO STREAMING GENERATION

  • Attention sinking preserves tokens from the initial frame alongside a rolling local window, while training simulates this fixed-context inference setup for long-video extrapolation.Inference rolls the recent-window KV cache forward to keep its size constant.Source: §3.3 · p. 6

4 EXPERIMENTS

Experiments evaluate StreamErase on synthetic, real-world, and long-video benchmarks, reporting strong quality and real-time throughput. Ablations examine the distillation pipeline and design choices for stable long-video streaming.

4 EXPERIMENTS

  • Evaluation spans 60 synthetic ROSE-Benchmark videos, 43 real-world VOR-Eval pairs, and LongErase; standard fidelity metrics and VBench assess quality.Training uses 16K video-video pairs, with 81-frame samples at 832 × 480 resolution.Source: §4 · pp. 6–7

4.1 COMPARISON WITH STATE-OF-THE-ART METHODS.

  • StreamErase performs strongly on ROSE-Benchmark and VOR-Eval at 26.06 FPS and remains comparable in visual quality from 5-second to 30-second videos.Table 2 evaluates LongErase at both durations.Source: Table 2 · p. 7, §4.1 · p. 7
  • In real-world comparisons, StreamErase removes targets and side effects with temporal consistency, while other methods leave residuals or flicker and blur in dynamic scenes.The comparisons include complex interactions such as shadows, occlusions, and illumination changes.Source: Figure 4 · p. 7, §4.1 · p. 8

4.2 ABLATION STUDIES

  • The pipeline moves from a bidirectional foundation using 100 NFEs to causal 3-frame chunks using 8 NFEs, then one-step DMD with no observable error accumulation.Stage 3 retains the high-fidelity erasing behavior and streaming capability established by the earlier stages.Source: §4.2 · p. 8
  • In the ROSE ablation, adding EE loss raises baseline PSNR from 30.104 to 32.571, while adding MMRG improves SSIM from 0.818 to 0.872 by capturing spatiotemporal correlations between objects and their side effects.Source: §4.2 · p. 8, Table 4 · p. 8

4.3 LONG-VIDEO STREAMING ANALYSIS

  • Chunk size trades quality, latency, and throughput: size 1 degrades quality, sizes above 3 hinder real-time interaction, so the default is 3.Small chunks also yield lower throughput.Source: §4.3 · pp. 8–9
  • On LongErase clips averaging 486 frames, a single-chunk sink and window perform best; larger windows degrade performance, while removing sinks harms long-term stability.Additional sinks offer marginal gains while increasing latency.Source: §4.3 · p. 9, Table 5 · p. 9, Figure 5 · p. 9
  • The interactive demo supports real-time user control during streaming, and the method handles flexible video lengths while preserving high visual fidelity.Figure 6 illustrates long-video erasing and a streaming demo.Source: §4.3 · p. 9, Figure 6 · p. 9

5 CONCLUSION

StreamErase transfers bidirectional diffusion priors into an efficient unidirectional autoregressive framework. It achieves strong performance across real-world erasing scenarios while running significantly faster than prior methods.

  • StreamErase adapts bidirectional diffusion priors to an efficient, unidirectional autoregressive paradigm.Source: §5 · p. 9
  • A progressive three-stage distillation pipeline combines attention sinks and rolling KV caches to address attention distribution shift during causal execution.Source: §5 · p. 9
  • The framework achieves strong performance across diverse real-world video erasing scenarios while running significantly faster than prior methods.Source: §5 · p. 9

AI USE STATEMENT

The authors used generative AI for translation assistance and readability editing, and reviewed AI-assisted work for accuracy and clarity.

  • Generative AI tools assisted with translation during drafting.Source: AI USE STATEMENT · p. 10
  • The authors report that generative AI was not used for dataset creation, methodology, experiments, implementation, or interpretation of results.Source: AI USE STATEMENT · p. 10
  • AI-assisted text was reviewed and revised by the authors, who take responsibility for the final paper content.Source: AI USE STATEMENT · p. 10

A MORE TRAINING DETAILS

The three-stage distillation pipeline is trained on eight NVIDIA A100 GPUs with AdamW. The stages use separate training schedules, including a short final DMD stage.

B ADDITIONAL ABLATION EXPERIMENTS

The ablations examine chunk size and sampling steps as speed–quality trade-offs, while the training-module details describe how object-image features are projected into the model’s latent space.

B ADDITIONAL ABLATION EXPERIMENTS

  • Smaller chunks require more autoregressive rollouts and less bidirectional attention, worsening performance; chunk sizes above 3 induce high latency for interactive use.With sink and window sizes of 1, chunk size 7 has lower FPS than chunk size 3 because its attention sequence is longer.Source: B ADDITIONAL ABLATION EXPERIMENTS · p. 14, Figure 7 · p. 14
  • Four-step training is optimal for this setup; the primary configuration uses chunk size 3 with 1 sampling step for a balance of interactivity and video quality.More than 4 sampling steps yields only marginal gains, while reducing to 2 causes a noticeable quality drop; Figure 7 compares latency, throughput, and LPIPS across configurations.Source: B ADDITIONAL ABLATION EXPERIMENTS · p. 14, Figure 7 · p. 14

C DETAILS OF THE TRAINING MODULES

  • The MLP maps object-image features into the model’s latent space through two sequential blocks, with a residual connection in the second block and final LayerNorm.Each block uses LayerNorm, a linear projection, GELU, and a second linear projection.Source: C DETAILS OF THE TRAINING MODULES · p. 15

D DETAILED EVALUATION ON ROSE-BENCH

The ROSE evaluation reports that Stage 3 DMD achieves the strongest average metrics, following Stage 1 initialization and Stage 2 streaming adaptation. Additional qualitative comparisons address real-world samples and training stages.

D DETAILED EVALUATION ON ROSE-BENCH

  • Stage 3 DMD achieves the highest average metrics across the ROSE benchmark, improving overall visual quality over the preceding stages.Table 6 presents quantitative comparisons across subcategories and metrics PSNR, SSIM, and LPIPS.Source: Table 6 · p. 15, D DETAILED EVALUATION ON ROSE-BENCH · p. 15
  • Stage 1 establishes a structural foundation with strong reconstruction quality across general lighting, particularly in the Common and Shadow subsets.The qualitative comparison across training stages is shown in Figure 9.Source: D DETAILED EVALUATION ON ROSE-BENCH · p. 15, Figure 9 · p. 16
  • Stage 2 enables continuous processing and preserves structural stability across subcategories, with a minor reconstruction-fidelity trade-off.Source: D DETAILED EVALUATION ON ROSE-BENCH · p. 15
  • Real-world qualitative comparisons highlight StreamErase’s handling of shadows, occlusions, and illumination changes in complex object-environment interactions.Source: Figure 8 · p. 16

E MORE QUALITATIVE COMPARISONS AND ADDITIONAL VISUAL RESULTS

The qualitative results compare erasing across training stages and on long videos; they describe Stage 3 DMD as mitigating the temporal drift observed in Stage 2.

E MORE QUALITATIVE COMPARISONS AND ADDITIONAL VISUAL RESULTS

F LIMITATION AND FUTURE WORK

The paper identifies three limitations: fixed-context generation may adapt less well to major scene changes, training data constrain performance, and explicit masks add interaction cost.

F LIMITATION AND FUTURE WORK

  • Fixed-context generation with attention sinks may preserve earlier context at the expense of adapting fully to substantial scene changes over long horizons.The stated boundary is newly emerging scenes; future work could explore more adaptive context management and attention.Source: F LIMITATION AND FUTURE WORK · p. 17
  • Performance remains constrained by the scale, duration, and real-world coverage of available video-erasing training data.The paper identifies diverse object appearances, camera motions, occlusions, and scene changes as challenging coverage needs.Source: F LIMITATION AND FUTURE WORK · p. 17
  • Explicit masks provide spatial control but require manual input, adding interaction cost and limiting flexibility.The paper suggests mask-free specification through natural-language instructions or simple interactions such as clicking.Source: F LIMITATION AND FUTURE WORK · p. 17

Related papers

Loading 2610.20137v1…