PaperlayerView exact revision on arXiv

Source-linked AI summary

LIVEditor-14B: Lightning Unified Video Editing via In-Context Sparse Attention

Shitong Shao, Zikai Zhou, Haopeng Li, Yingwei Song, Wenliang Zhong, Lichen Bai, Zeke Xie

arXiv:2605.04569v2cs.CV

TL;DR

In-context video editing faces costly quadratic attention, especially with long sequences and many context tokens. LIVEditor-14B introduces ISA to prune redundant context and route queries adaptively, achieving lossless acceleration while outperforming state-of-the-art methods in quality and latency.

  • Problem

    In-context video editing lacks efficient sparse attention suited to its long sequences and distinct source-context token structure, making attention costly in memory and latency.

  • Method

    ISA prunes redundant context tokens and dynamically routes queries by sharpness to standard attention or block-wise 0-th order Taylor sparse attention.

  • Results

    LIVEditor-14B achieves leading performance across EditVerseBench metrics and significantly outperforms competing sparse attention methods while improving inference latency.

  • Takeaways & Limitations

    ISA provides experimentally lossless acceleration for unified video editing while retaining strong generation quality across evaluated benchmarks.

  • Takeaways & Limitations

    Higher-order Taylor expansions were discarded because they were poorly suited to hardware acceleration and incurred prohibitive computational overhead.

Abstract

from arXiv · show

Video editing has evolved toward In-Context Learning (ICL) paradigms, yet the resulting quadratic attention costs create a critical computational bottleneck. In this work, we propose In-context Sparse Attention (ISA), the first near-lossless empirical sparse framework tailored for ICL video editing. Our design is grounded in two key insights: first, context tokens exhibit significantly lower saliency than source tokens; second, we theoretically prove and empirically validate that Query sharpness correlates with approximation error. Motivated by these findings, ISA implements an efficient pre-selection strategy to prune redundant context, followed by a dynamic query grouping mechanism that routes high-error queries to full attention and low-error ones to a computationally efficient 0-th order Taylor sparse attention. Furthermore, we build \textbf{\texttt{LIVEditor-14B}} , a novel lightning video editing model via ISA and a proposed video-editing data pipeline that curated a 1.7M high-quality dataset. Extensive experiments demonstrate that LIVEditor-14B achieves a $\sim$60% reduction in attention-module latency while surpassing state-of-the-art methods across EditVerseBench, IVE-Bench, and VIE-Bench, delivering near-lossless acceleration without compromising visual fidelity.

1. Introduction

LIVEditor-14B addresses the quadratic attention bottleneck in in-context video editing with In-context Sparse Attention (ISA), combining theoretical analysis with a practical sparse-attention framework. The approach prunes low-saliency context tokens and routes queries by sharpness to reduce computation while preserving representational fidelity.

  • Challenge: Video attention becomes a primary bottleneck as sequence lengths grow from 5K to 50K, with computational cost increasing quadratically.
  • Contribution: ISA addresses the lack of efficient sparse attention for in-context learning through a systematic investigation linking theoretical insights to practical application.
  • Key Finding: Context tokens typically contribute negligibly to total attention scores, indicating limited saliency and enabling pruning when critical tokens are retained.
  • Method: ISA optimizes efficiency across Query and Key/Value dimensions by retaining salient context Key/Value pairs and partitioning Queries using a sharpness metric.

2. Preliminary

This section introduces standard attention and motivates hardware-efficient block sparsification for video models. It then describes pooling attention as a lightweight mechanism for selecting blocks with reduced computational complexity.

  • Standard Attention: Standard attention computes scaled Query–Key scores, applies softmax to obtain attention weights, and multiplies them by Values to produce the output.The inputs are Query, Key, and Value tensors with batch, head, sequence-length, and head-dimension axes.
  • Block Sparse Attention: Block sparse attention selects subsets of Key and Value indices, replacing unstructured element-wise masks with hardware-efficient blocks formed from contiguous spatiotemporal video tiles.The flattened tile sequence is partitioned into non-overlapping Query and Key/Value blocks.
  • ICL Attention Patterns: In ICL, attention matrices exhibit four distinct distributional regions, motivating attention mechanisms tailored specifically to in-context scenarios.The section also notes that source Queries typically attend more strongly to source Keys than to context Keys, especially in deeper layers.
  • Pooling Attention: O(NSD) to O(NQNKD) reduction is achieved by pooling sequence representations before selecting top-k entries in a coarse attention map to derive the block mask.Pooling produces compressed Query, Key, and Value representations at the Query-block and Key/Value-block levels.

3. In-Context Sparse Attention

ISA accelerates in-context video-editing attention by pre-selecting salient context blocks, approximating low-error interactions with block-wise 0-th order Taylor attention, and routing queries by estimated approximation error. The resulting LIVEditor-14B combines ISA with a two-stage 1.7M-sample training pipeline for efficient, robust video editing.

  • Pre-Selection: Pre-selection retains the αs⌈Lctx/b⌉ most salient context blocks, reducing complexity from O(NSD) to O(N(Lsrc + αsLctx)D).The method exploits the lower saliency of context tokens relative to source tokens and reconstructs sparse tensors using gather and concatenation.
  • Taylor Sparse Attention: ISA computes salient blocks exactly and approximates remaining block interactions with 0-th order Taylor expansions, reducing complexity from O(LQLKD) to O(LQD).The sparse mechanism uses OnlineSoftmax for selected blocks, while unselected blocks use compressed Key and Value tensors.
  • Grouped Computation: Mi is adopted as the query-grouping metric because it is efficiently derived from pooling scores and strongly positively correlates with Taylor approximation error.The theoretically motivated ||Q(K − Kc)⊤||2 metric is computationally prohibitive and is shown to be an ineffective proxy.
  • Grouped Computation: Queries are divided by sparse-attention error, routing high-error queries to more complex computation and low-error queries to efficient sparse attention.The grouping follows the error bound established for block-wise 0-th order Taylor sparse attention.
  • LIVEditor-14B: 1.7M mixed-quality samples are used for pre-training, followed by 0.089M curated high-quality samples for refining visual aesthetics in LIVEditor-14B.The model integrates ISA to target lightning-fast unified video editing while improving editing robustness and fidelity.

4. Experiment

Experiments evaluate LIVEditor-14B across four video-editing benchmarks, compare ISA with full attention and alternative sparse-attention methods, and test the two-stage training strategy and ISA hyperparameters. LIVEditor-14B (ISA) achieves the best overall performance, while Stage II improves all reported metrics.

  • Benchmarks: The evaluation suite uses EditVerseBench, VIE-Bench, IVE-Bench, and FiVEBench to assess model superiority, sparse-attention performance, and ablation results.EditVerseBench evaluates LIVEditor-14B against existing methods and compares sparse-attention mechanisms, while EditVerseBench and FiVE-Bench support ablations.
  • Evaluation on Other Benchmarks: LIVEditor-14B (ISA) achieves the best overall performance on VIE-Bench and IVE-Bench against an expanded set of state-of-the-art video-editing methods.The comparisons include both previously mentioned baselines and additional methods such as Ditto, VACE, ICVE, Omni-Video, AnyV2V, StableV2V, and Pika.
  • Sparse Attention Comparison: ISA is benchmarked on EditVerseBench against Radial Attention, Sparge Attention, STA, SWA, and VSA in a training-free comparison.ISA is applied directly to the pre-trained LIVEditor-14B (full-attn) model.
  • Stage I vs. Stage II: Stage II fine-tuning improves performance across all metrics on the evaluated benchmarks.The model scores rise from 6.46 to 7.89 in Quality, 19.50 to 20.09 in Text Align, 25.27 to 27.19 in Temporal Consistency, and 22.63 to 24.55 in Editing Quality.
  • Hyperparameters in ISA: ISA sparsity is controlled by the Flat Ratio αf, No Sparsity Ratio αns, and Select Ratio αs, with defaults αf = 0.5, αns = 0.0625, and αs = 0.125.The hyperparameter evaluation varies one parameter at a time while applying ISA directly to the pre-trained full-attention model.

5. Conclusion

The paper presents LIVEditor-14B as a unified, lightning video-editing framework that addresses ICL’s computational bottleneck with In-context Sparse Attention (ISA). ISA links query sharpness to Taylor approximation errors and combines dynamic query routing with redundant-context pruning for experimentally lossless acceleration.

  • LIVEditor-14B is presented as a lightning, unified framework for video editing.
  • ISA addresses ICL’s computational bottleneck through a theoretical link between query sharpness and Taylor approximation errors.
  • ISA dynamically routes queries and prunes redundant context tokens to achieve experimentally lossless acceleration.
  • The work also establishes a comprehensive data processing pipeline for the framework.

A. Evaluation Benchmark … C.1. Data Processing Pipeline

The paper evaluates LIVEditor-14B across four video-editing benchmarks, specifies ISA’s inference configuration and speedup, and constructs training data through a four-phase automated video-to-video synthesis pipeline.

  • A. Evaluation Benchmark: The evaluation spans EditVerseBench, IVE-Bench, VIE-Bench, and FiVE-Bench to assess effectiveness, robustness, and generalization under varied complexity.The benchmarks cover broad video-editing evaluation, source diversity, temporal scalability, reference-guided tasks, and fine-grained object-level editing.
  • B. Hyperparameter Setting: Training initializes from the high-noise Wan2.2-T2V variant and uses AdamW with β1 = 0.9, β2 = 0.999, and weight decay 10^-2.The first training stage uses approximately 1.7 million samples, learning rate 10^-5, 32 80GB GPUs across 4 nodes, sequence parallelism size 2, and global batch size 16.
  • B. Hyperparameter Setting: 1.47× speedup over FlashAttention-3 is achieved by ISA during full 32-step inference with CFG using Block-Wise 0-th Taylor Sparse Attention.The configuration uses a flat ratio of 0.5, a no-sparsity ratio of 0.0625, and a select ratio of 0.125.
  • C. Data Pipeline: The constructed dataset distributes editing scenarios across seven primary tasks, with global editing largest and fine-grained semantic edits heavily represented.The distribution is intended to support stylistic diversity and instruction-following precision on localized regions.
  • C.1. Data Processing Pipeline: The video-to-video synthesis pipeline generates instruction-aligned editing pairs through four primary phases.The phases are Instruction Preparation, Target Frame Generation, Target Video Generation, and Post-Processing & Latent Encoding.
  • C.1. Data Processing Pipeline: Instruction Preparation extracts a representative source frame and uses Gemini-2.5-Pro/GPT-4o VLMs to sample edit types and generate precise target-image instructions.The edit task pool includes object addition, object removal, object swap, object stylization, style transfer, and motion edit.
  • C.1. Data Processing Pipeline: Target Frame Generation uses Gemini 2.5 Image Preview to transform the source frame, then VLMs generate a target-video prompt from the source frame, source-video prompt, and target frame.The pipeline applies a VLM-based quality-control process to maintain data quality.
  • C.1. Data Processing Pipeline: Target Video Generation synthesizes videos with an internal 14-billion-parameter text-to-image-to-video diffusion model and filters quality using inconsistency scores from YOLO, GroundingDINO, and SAM.The source and target videos are additionally assessed by VLMs, while DOVER, VideoReward, and VLMs select samples for visual aesthetics and prompt-following ability; selected pairs are resized, resampled, instruction-processed, and encoded through Wan 2.1 VAE.

C.2. Data Distribution · C.3. Data Scheduling Process

The dataset spans over 1.7M high-quality video-to-video editing pairs across seven modalities, with emphasis on Style Transfer and fine-grained Human Edit data. Training uses a two-stage schedule combining synthetic and Ditto-filtered samples, while later results report ISA outperforming full attention on almost all FiVE-Bench color-alteration metrics.

  • C.2. Data Distribution: The dataset contains over 1.7M high-quality video-to-video editing pairs organized into 7 primary editing modalities.The modalities are Style Transfer, Object Swap, Object Addition, Object Stylization, Object Removal, Human Edit, and Other.
  • C.2. Data Distribution: Style Transfer is the largest category at 33.87% of the dataset.The distribution balances global transformations with localized semantic manipulations.
  • C.2. Data Distribution: Human Edit comprises 12.80% of the training dataset and includes Avatar Transformation and Face Detail Replacement.This fine-grained category provides data for evaluating complex human priors.
  • C.3. Data Scheduling Process: The first training stage aggregates approximately 1 million synthetic samples with 0.7 million Ditto-filtered samples.The resulting pre-training collection intentionally mixes high- and low-quality data.
  • C.3. Data Scheduling Process: After pre-training, the model performed strongly on most tasks but remained limited in object removal.This limitation motivated the continuation of the training pipeline beyond the initial stage.
  • C.3. Data Scheduling Process: LIVEditor (ISA) outperforms LIVEditor (full-attn) on almost all reported FiVE-Bench color-alteration metrics.This comparison is reported in both Table 9 and Table 10.

D. Additional Analysis · E. Additional Experimental Result

Additional analysis shows that Taylor approximation error is bounded by query-key terms and block-mean variance, with sharpness serving as the practical proxy because it dominates the bound and is computationally tractable.

  • D. Additional Analysis: Theorem 3.1 upper-bounds Taylor error E_i using ||Q(K − K_c)^⊤||_2 and M_i.The bound depends on query-key interactions and a block-mean variance term.
  • D. Additional Analysis: Empirical analysis indicates that the error bound is dominated by M_i rather than mean intra-block variance.This finding is reported from Figs. 6 and D.1.
  • D. Additional Analysis: Because intra-block variance is computationally prohibitive to calculate, the method adopts M_i, termed sharpness, as its effective proxy metric.The proxy choice is supported by the empirical dominance of M_i.

E.1. Evaluation on FiVE-Bench · E.2. Evaluation on IVE-Bench · E.3. Evaluation on VIE-Bench

Across FiVE-Bench, IVE-Bench, and VIE-Bench, LIVEditor-14B (ISA) consistently outperforms or matches full-attention and state-of-the-art baselines across structural, perceptual, instruction, fidelity, and task-level metrics. These results support near-lossless sparse-attention video editing across diverse editing tasks.

  • E.1. Evaluation on FiVE-Bench: On FiVE-Bench rigid object replacement, ISA lowers Structure Distance to 71.37 versus 81.31 for full-attention, while achieving PSNR 15.40 and Motion Fidelity 58.4.ISA is also reported to achieve Accuracy 50.71 and FiVE-Union 37.88, versus 43.80 and 25.86 for full-attention.
  • E.1. Evaluation on FiVE-Bench: For non-rigid object replacement, ISA achieves Structure Distance 81.78 and LPIPS 319.22, compared with 90.55 and 359.51 for full-attention.Its FiVE metrics include YN 57.00, FiVE-Intersection 40.72, and Accuracy 44.49.
  • E.1. Evaluation on FiVE-Bench: On FiVE-Bench color alteration, ISA records PSNR 15.91, SSIM 54.18, and Text Alignment 28.22, exceeding full-attention values of 15.34, 51.52, and 27.47.Its Model Consistency is 90.00, FiVE-Union is 74.76, and Accuracy is 83.18, versus 84.00, 73.74, and 79.77.
  • E.1. Evaluation on FiVE-Bench: For material modification and object addition, ISA improves Structure Distance to 73.09 and 54.79, and reduces LPIPS to 310.98 and MSE to 299.82 versus full-attention.ISA also achieves Motion Fidelity 76.39 for object addition.
  • E.2. Evaluation on IVE-Bench: On IVE-Bench, ISA achieves a leading Total score of 0.67 while maintaining a robust and balanced profile against eight state-of-the-art methods.The comparison includes Ditto, InsV2V, LucyEdit, and VACE.
  • E.2. Evaluation on IVE-Bench: ISA leads all four IVE-Bench video-quality sub-metrics: Subjective Quality 0.97, Background Preservation 0.98, Flickering 0.99, and Motion 1.00.The passage attributes the absence of temporal jitter and artifacts to sparse attention.
  • E.2. Evaluation on IVE-Bench: For IVE-Bench instruction and fidelity, ISA raises Instruction from 0.43 to 0.47 and achieves Fidelity 0.76 and Semantic consistency 0.91.It also records VTSS 0.040 and Quantity 0.40, while fidelity scores remain comparable to full-attention.
  • E.3. Evaluation on VIE-Bench: On VIE-Bench, ISA achieves average scores of 8.84 for Object Addition and 8.14 for Object Swapping, surpassing Omni-Video at 6.24 and Pika at 7.41.The evaluation covers five distinct editing tasks.

F. The Proof of Error Bound of 0-th Order Taylor Approximation

The proof rigorously links 0-th-order Taylor approximation error to attention sharpness through perturbation analysis. It combines a Taylor remainder bound, Jacobian stability, and Lipschitz decomposition into a tightened final bound.

  • Proof strategy: The analysis begins with first-order perturbation analysis to derive the relationship between approximation error and attention sharpness.
  • Step 1: Taylor error bound: Taylor’s theorem with integral remainder yields a second-order expansion of softmax around the approximate logits, with the Hessian bounded by ∥H∥∞≤1.
  • Step 1: Taylor error bound: The resulting value-level approximation error is strictly bounded for each token.
  • Step 2: Jacobian stability: The Jacobian satisfies ∥J∥2 = pmax(1 −pmax), and its stability is linked to the distribution energy and sharpness metric Mu.
  • Step 3: Lipschitz decomposition: A Lipschitz decomposition bounds deviations from the block mean, enabling the Taylor estimate and interaction-term control to combine into the final tightened bound.

G. The Algorithm Implementation of ISA

This section presents pseudocode implementations for ISA and Block-Wise 0-th Taylor Sparse Attention. For clarity, it assumes identical sequence lengths and head dimensions for Q, K, and V, while omitting batch size and head dimensions in the latter algorithm.

  • Algorithm presentation: The section provides pseudocode for ISA in Algorithm 1.
  • Algorithm presentation: It provides pseudocode for Block-Wise 0-th Taylor Sparse Attention in Algorithm 2.
  • Notation and assumptions: For clarity, the implementation assumes identical sequence lengths and head dimensions for Q, K, and V, omitting batch size and head dimensions in Algorithm 2.The omission simplifies the presentation of the Block-Wise 0-th Taylor Sparse Attention pseudocode.

H. Implementation of 0-th Taylor Sparse Attention: Triton Vs. TileLang · I. Additional Explanation

The appendix compares Triton and TileLang implementations of block-wise zeroth-order Taylor sparse attention and explains ISA’s performance behavior, approximation benefits, and comparison scope. For video-editing sequences of approximately 64K tokens, the authors use Triton for sparse-attention comparisons.

  • H. Implementation of 0-th Taylor Sparse Attention: Triton Vs. TileLang: Triton implements the forward and backward passes of block-wise zeroth-order Taylor sparse attention, while TileLang provides an alternative implementation for comparison.The comparison is conducted on a Hopper GPU.
  • H. Implementation of 0-th Taylor Sparse Attention: Triton Vs. TileLang: TileLang is more efficient at low No Sparsity Ratio and short sequence length, whereas Triton outperforms it as both increase.This performance crossover motivates selecting Triton for the target video-editing regime.
  • H. Implementation of 0-th Taylor Sparse Attention: Triton Vs. TileLang: Approximately 64K tokens correspond to 544 × 960 resolution on Wan-2.1/2.2, so the authors use Triton for comparisons with other sparse attention methods.The stated sequence length reflects typical video-editing workloads.
  • I. Additional Explanation: The authors exclude MLLM-based methods such as EditVerse, UniVideo, and InstructX because public implementations and standardized benchmarks are unavailable.They also identify these methods as operating under a distinct paradigm based on massive MLLM representational capacity.
  • I. Additional Explanation: ISA reconstructs full attention through context-token selection and block-wise zeroth-order Taylor sparse attention, yet can outperform full attention after training.The authors attribute this improvement to pruning irrelevant noise tokens.
  • I. Additional Explanation: ISA’s distinct attention paradigm adapts to sparsity during training, while local-mean approximation provides a secondary information-filtering mechanism.The reported advantage arises from the synergy between TopK pre-selection and Taylor-based aggregation.
  • I. Additional Explanation: The appendix reports that block variance does not reliably indicate Taylor approximation error magnitude.The visualization shows negligible correlation between approximation error Ei and block variance ∞, while the accompanying norm criterion also fails as a reliable indicator.
Loading 2605.04569v2…