Source-linked AI summary
EditaLive! Unified Character Video Editing for Live Streaming
Zhiyuan Li, Chi-Man Pun, Peng-Tao Jiang, Bo Li, Xiaodong Cun
TL;DR
EditaLive addresses the difficulty of real-time human-centric video editing, where existing methods can disrupt facial expressions and depend on offline processing. It decouples appearance from motion, adapts generation to causal streaming, and uses aligned distillation for two-step inference; experiments report strong editing quality with real-time throughput and low latency.
Problem
Existing video-editing methods can introduce facial-expression inconsistencies and rely on offline bidirectional or multi-step inference unsuitable for live interaction.
Method
EditaLive combines reference-frame editing and motion-aligned reconstruction with chunk-wise causal generation and aligned self-rollout distillation using sparse attention.
Results
14.47 FPS throughput and 0.829-second average inter-chunk latency are reported on one NVIDIA H100, with a 0.720 edit success rate versus at most 0.428 for dedicated streaming baselines.
Takeaways & Limitations
EditaLive provides a unified approach to instruction-guided character appearance editing with motion control, facial-expression preservation, and low-latency streaming inference.
Takeaways & Limitations
The current framework relies on skeleton sequences for motion control, leaving further improvement directions.
Abstract
from arXiv · showhide
Conventional video editing primarily focuses on scene-level content, whereas live streaming places greater emphasis on the human subject. However, directly applying existing video-editing methods to human-centric live streaming remains challenging, as they may introduce facial-expression inconsistencies and typically depend on multiple offline inference steps, making them unsuitable for real-time interaction. We propose EditaLive, a novel framework for real-time streaming character video editing. In detail, we start from a pretrained image animation model (Wan-Animate), which naturally decouples appearance from motion, and repurpose it as the base model for instruction-based human-centric video editing by reference frame editing and video reconstruction via the collected CharEdit-50K dataset. Besides, we adapt the model from offline bidirectional to causal streaming generation, and design an aligned self-rollout distillation strategy that compresses the model into a two-step sampler, where fixed RoPE and align forcing reduce training--inference discrepancies, and first-frame preserved sparse attention filters redundant historical information to mitigate appearance drift. Extensive experiments demonstrate that EditaLive delivers state-of-the-art editing performance with faithful preservation of facial expressions and low-latency real-time streaming inference.
1 Introduction
EditaLive targets real-time character video editing by separating appearance from motion and replacing offline, multi-step processing with causal streaming generation. Its reconstruction-based training, CharEdit-50K dataset, and aligned distillation aim to preserve facial expressions while supporting low-latency editing.
- Live-streaming character editing lets streamers and viewers customize human-centric content with simple text prompts.
- Existing video-editing methods focus on scenes or objects, while frame-wise editing can cause temporal flickering and native video methods require offline processing.
- EditaLive treats character appearance and motion as separable, using edited reference frames, reverse instructions, and motion-conditioned reconstruction.
- The framework adapts offline editing to chunk-wise causal streaming generation and distills it into a two-step sampler.
- EditaLive supports diverse appearance editing and motion-driven animation while preserving facial expressions through explicit motion control.
- Fixed RoPE, align forcing, and first-frame-preserved sparse attention reduce training–inference discrepancies and mitigate long-term appearance drift.
2 Related Work
Related work spans instruction-based image and video editing, motion-driven character animation, and efficient streaming video generation. EditaLive extends these lines by combining character appearance editing with causal, two-step streaming generation.
- Instruction-based image editing uses natural-language prompts, while frame-wise video extensions improve efficiency but lack stable temporal consistency.
- Motion-driven character animation conditions video generation on reference images and motion representations such as skeletons, 3D keypoints, motion frames, and facial representations.
- Video-generation efficiency methods include quantization, sparse attention, sampling-step distillation, and autoregressive paradigms for real-time streaming.
- EditaLive extends autoregressive streaming generation to character editing with aligned self-rollout distillation for efficient two-step inference and long-horizon appearance preservation.
3 Preliminaries
The preliminaries describe latent video diffusion and distribution matching distillation. Video latents follow a flow-matching path from Gaussian noise to data, while DMD trains a few-step student toward a target distribution using score differences.
- Video diffusion encodes an input video into a compact spatio-temporal latent representation before generation in latent space.
- Flow matching constructs noisy latent Z_t by interpolating between Gaussian noise Z_0 and the clean video latent Z_1.
- At inference, numerical integration moves the latent from t = 0 toward the data distribution at t = 1, followed by VAE decoding into video.
- Distribution Matching Distillation trains a student generator with substantially fewer denoising steps by matching its perturbed distribution to real-data or teacher distributions.
- DMD optimization uses score-function differences, keeping the real-score model fixed while updating the fake-score model and student generator alternately.
4 EditaLive : Human-Centric Video Editing Model
EditaLive reformulates character video editing around separated appearance and motion, then adapts generation for causal streaming and compresses it into a stable two-step sampler. Its training data, causal attention, and sparse historical-context design target faithful motion preservation and efficient long-term editing.
- Task formulation: EditaLive defines character video editing as modifying a primary human or human-like subject’s appearance in real time from an instruction.
- Appearance–motion decoupling: The model separates editable appearance from preserved body motion and facial dynamics using a reference frame, skeleton sequence, and implicit facial representations.The reference frame represents appearance, while the extracted motion signals condition reconstruction of the source video.
- Training data: CharEdit-50K restricts synthetic edits to reference images while retaining authentic source videos as reconstruction targets, reducing facial-expression mismatch in training pairs.Samples include edited images, reverse instructions, and corresponding source videos after quality filtering.
- Causal streaming adaptation: Chunk-wise causal adaptation lets target chunks attend to clean preceding context while preserving bidirectional interactions within each chunk for streaming inference.All target chunks can be trained in parallel, while future context and other noisy target chunks are excluded from attention.
- Aligned distillation: Aligned self-rollout distillation compresses generation into a two-step sampler, aligning rollout and inference states while Fixed RoPE and First-frame Preserved Sparse Attention mitigate long-term appearance drift.Sparse attention retains selected historical blocks and preserves reference-related context to reduce redundant history.
5 Experiments
EditaLive is evaluated against bidirectional, streaming, and cascaded baselines for quality, consistency, success, and efficiency, with ablations testing its streaming and distillation components.
- Qualitative & Quantitative Comparison: EditaLive ranks first or second across all eight quality metrics while preserving facial expressions, body poses, and scene structure.Existing streaming methods sometimes fail to apply requested edits or introduce facial-expression inconsistencies.
- Efficiency: 14.47 FPS throughput and 0.829-second inter-chunk latency are achieved on one NVIDIA H100 GPU.EditaLive also reaches a 0.720 edit success rate, compared with at most 0.428 for dedicated streaming baselines.
- Efficiency: Compared with LucyEdit, EditaLive provides 4.7× higher throughput and 31.7× lower latency.LucyEdit is identified as the fastest bidirectional baseline.
- Causal Streaming Adaptation: Target-only optimization keeps the reference latent as clean conditioning and avoids conflicting supervision during causal adaptation.Optimizing reference and target positions together weakens reference conditioning and causes unintended character changes.
- Aligned Self-Rollout Distillation: Fixed RoPE and first-frame-preserved sparse attention improve long-term identity stability, with the complete model achieving ID-SIM 0.492.Without Fixed RoPE, ID-SIM is 0.381; without FPSA, it is 0.455, while adding sparse attention without first-frame preservation reaches 0.466.
6 Conclusion
EditaLive brings instruction-guided character appearance editing to live streams by separating appearance from motion and converting offline editing into causal two-step generation.
- Conclusion: EditaLive learns appearance transformations from edited reference images and motion-aligned real-video targets, supported by CharEdit-50K.The framework decouples character appearance from body motion and facial dynamics.
- Conclusion: The bidirectional editing model is converted into a chunk-wise causal generator and distilled into a two-step sampler.Align Forcing, Fixed RoPE, and FPSA address streaming alignment and long-sequence appearance stability.
- Conclusion: Experiments demonstrate advantages in editing quality, motion consistency, long-term stability, and inference efficiency.
A Additional Information about Dataset
The paper constructs CharEdit-50K from diverse videos, reference-frame edits, generated instructions, and filtered synthetic images, then evaluates editing with CharEdit-Bench subsets.
- Video Collection and Reference Selection: CharEdit-50K samples videos from SpeakerVid-5M, the Seamless Interaction Dataset, and additional Internet sources.A single high-quality frame is selected from each video using facial and hand visibility and sharpness criteria.
- Instruction Generation: GPT-5.5 generates forward and reverse instructions across four editing categories conditioned on each reference image.The instructions include add–remove and remove–add transformations.
- Image Synthesis and Filtering: Qwen-Image-Edit and Nano Banana 2 synthesize candidate images, which GPT-5.5 filters for edit accuracy, target clarity, and character consistency.Bidirectional edits are additionally checked for an unambiguous inverse instruction.
- Dataset Examples: The dataset examples include tie-color changes, Snoopy and low-poly style transformations, and eyeglass-frame recoloring.
- Benchmark Construction: CharEdit-Bench-S contains 150 five-second videos with 150 evaluation cases, while CharEdit-Bench-L contains 30 videos longer than one minute.Both subsets use 480 × 832 resolution; the long subset runs at 16 FPS and pairs each video with two prompts.
B.1 Training Details
Training uses a three-stage pipeline that first learns appearance–motion decoupling, then adapts the model for causal streaming and subsequent distillation.
- Training Pipeline: Stage 1 performs reconstruction-based appearance–motion decoupled training using independently provided motion conditions.ViTPose extracts skeleton sequences and localizes facial regions for corresponding facial-video construction.
- Evaluation Setup: Baselines are evaluated with official implementations and released checkpoints, with resolution adjustments applied to SANA-Streaming.Ditto uses its released local and global-style LoRA checkpoints.
B.3 Evaluation Details
EditaLive evaluates identity, facial-expression, body-pose, text-alignment, edit-quality, and background consistency using paired metrics and VLM judgments. Its streaming runtime is measured on 81-frame, 384 × 672 clips with supported acceleration enabled.
- Evaluation Metrics: ID-SIM, AED, and APD measure preservation of identity, facial expressions, and body poses between temporally corresponding source and edited frames.Higher ID-SIM and lower AED and APD indicate better preservation.
- VLM Evaluation: GPT-5.5 rates Text Alignment, Edit Quality, and Background Consistency from 0 to 3 and returns a binary overall-success judgment.Three temporally aligned frame pairs are sampled from each source and edited video.
- Training Procedure: The aligned self-rollout training procedure uses fixed RoPE indices and a fixed-length active cache while completing full rollouts across video chunks.The algorithm specifies reference, sink, and local cache lengths of 1, 3, and 3.
- Inference Efficiency: The evaluation runtime covers the generation pipeline from model-ready conditions through decoded RGB frames on an NVIDIA H100 GPU.EditaLive uses torch.compile, and acceleration techniques include FlashAttention, torch.compile, and warm-up.
- Benchmark Reporting: CharEdit-Bench-L compares methods quantitatively, with APD multiplied by 10 and unavailable LiveEdit global-style metrics omitted.The omission reflects that those scores are not comparable when the requested global edit is not successfully performed.
C More Experimental Results
EditaLive maintains editing quality, character consistency, and source motion over long sequences, while also supporting cross-character editing and diverse qualitative edits with reduced appearance drift.
- Long-Video Evaluation: EditaLive achieves the strongest overall performance on CharEdit-Bench-L long-video evaluation.The reported results indicate that editing quality and character consistency remain maintained over long sequences.
- Long-Video Evaluation: Qualitative long-video comparisons show consistent edited appearance while faithfully following source motion.These results are presented as evidence of long-term stability.
- Cross-Character Editing: EditaLive supports cross-character editing by combining one reference character’s appearance with motion signals from a different driving video.This capability follows from the appearance–motion decoupled formulation.
- Additional Qualitative Results: Across diverse characters and instructions, EditaLive shows less appearance drift while maintaining the requested edit, character appearance, and source motion.Additional comparisons cover both diverse edits and extended video sequences.
D Limitations & Future Work
EditaLive’s remaining limitations concern fine-grained hand-motion control and incomplete appearance cues from a single reference image. The paper also discusses privacy, consent, copyright, and misuse risks.
- Motion Control: Skeleton-based body-motion control does not explicitly represent fine-grained finger articulation, so subtle or complex hand gestures may be reproduced less accurately.The paper suggests dense hand keypoints or hand-specific representations as possible improvements.
- Qualitative Limitations: The Snoopy-style example is presented as a limitation case for EditaLive.The supplied figure text identifies the editing prompt but does not state the specific failure beyond labeling it a limitation.
- Appearance Representation: A single reference image provides limited appearance cues for regions occluded in that image.When those regions later become visible, their inferred appearance may differ from the source video; multiple references are proposed.
- Ethics and Responsible Use: Realistic character editing raises concerns about privacy, consent, copyright, impersonation, deceptive manipulation, and non-consensual use.The paper recommends consent and responsible deployment practices such as access control, disclosure, and watermarking.