PaperlayerView exact revision on arXiv

Source-linked AI summary

LIVEditor-14B: Lightning Unified Video Editing via In-Context Sparse Attention

Shitong Shao, Zikai Zhou, Haopeng Li, Yingwei Song, Wenliang Zhong, Lichen Bai, Zeke Xie

arXiv:2605.04569v2cs.CV

TL;DR

In-context video editing faces a computational bottleneck from attention over long sequences with context tokens alongside source tokens. LIVEditor-14B uses In-context Sparse Attention to prune redundant context and route queries between full and sparse attention, and the authors report strong performance in generation quality and inference latency across benchmarks.

  • Problem

    In ICL video editing, attention over long sequences with context tokens alongside source tokens creates a computational bottleneck and quadratic cost.

    Source: §1 · p. 2

  • Method

    ISA prunes redundant context and routes queries by sharpness to full attention or block-wise 0-th order Taylor sparse attention.

    Source: §3 · p. 5, §5 · p. 8

  • Results

    Across EditVerseBench and other benchmarks, LIVEditor-14B significantly outperforms state-of-the-art methods in generation quality and inference latency.

    Source: §5 · p. 8

  • Takeaways & Limitations

    The authors conclude that ISA's principles offer a scalable solution for ICL video editing.

    Source: §5 · p. 8

    The authors discarded first- and second-order Taylor variants because they were ill-suited for hardware acceleration and incurred prohibitive computational overhead.

    Source: §3 · p. 5

Abstract

from arXiv · show

Video editing has evolved toward In-Context Learning (ICL) paradigms, yet the resulting quadratic attention costs create a critical computational bottleneck. In this work, we propose In-context Sparse Attention (ISA), the first near-lossless empirical sparse framework tailored for ICL video editing. Our design is grounded in two key insights: first, context tokens exhibit significantly lower saliency than source tokens; second, we theoretically prove and empirically validate that Query sharpness correlates with approximation error. Motivated by these findings, ISA implements an efficient pre-selection strategy to prune redundant context, followed by a dynamic query grouping mechanism that routes high-error queries to full attention and low-error ones to a computationally efficient 0-th order Taylor sparse attention. Furthermore, we build \textbf{\texttt{LIVEditor-14B}} , a novel lightning video editing model via ISA and a proposed video-editing data pipeline that curated a 1.7M high-quality dataset. Extensive experiments demonstrate that LIVEditor-14B achieves a $\sim$60% reduction in attention-module latency while surpassing state-of-the-art methods across EditVerseBench, IVE-Bench, and VIE-Bench, delivering near-lossless acceleration without compromising visual fidelity.

1. Introduction

ICL video editing makes full attention costly because context and source tokens are concatenated, while existing sparse methods overlook their differences. ISA uses context-aware pruning and sharpness-guided computation, and powers LIVEditor-14B with a large curated editing dataset.

Empirical Success

  • ISA reduces attention-module latency by approximately 60% versus standard SDPA and FlashAttention v2/3, while LIVEditor-14B outperforms state-of-the-art models across EditVerseBench and VIE-Bench.The speedup relative to SDPA and FA2 becomes more pronounced as sequence length grows.Source: Figure 2 · p. 2, §1 · p. 2

Challenge

  • Video ICL attention scales quadratically with sequence length, and context tokens commensurate with source tokens quadruple its computational cost while increasing memory use and latency.Existing sparse attention methods designed for general video generation do not account for the distinction between context and source tokens.Source: §1 · p. 2

Key Finding and Theoretical Analysis

  • Context tokens typically contribute a negligible share of total attention scores, while Query sharpness is proportional to 0-th order Taylor approximation error.This motivates pruning redundant context and assigning high-sharpness queries precise computation while approximating low-sharpness queries.Source: §1 · p. 2

ISA

  • ISA retains critical context tokens, then routes high-error queries to full attention and low-error queries to block-wise 0-th order Taylor sparse attention.The sparse approximation uses tiled Key-Value means and reduces complexity from O(N 2D) to O(N 2D/b).Source: §1 · p. 2, Figure 3 · p. 3

LIVEditor-14B

  • LIVEditor-14B combines ISA with a proposed pipeline that constructed a dataset of over 1.7M high-quality video-editing pairs.The pairs cover style transfer, object swapping, and human editing, using an automated pipeline involving VLMs and diffusion models.Source: §1 · p. 2

2. Preliminary

The preliminaries define standard QKV attention and hardware-oriented block sparsification, with pooling attention selecting blocks efficiently. They also describe distinctive ICL attention patterns, including lower source-to-context scores than source-to-source scores.

Standard Attention

  • Standard attention computes scores from Q and K, applies softmax to obtain attention weights, then multiplies those weights by V.Source: §2 · p. 3

Block Sparse Attention

  • Block sparsification selects subsets of K and V; video models flatten contiguous spatiotemporal tiles into blocks, whose masks bypass score computation and value aggregation when entries are zero.Source: §2 · p. 3

ICL attention patterns

  • ICL attention exhibits four distinct regions, while source Queries typically score source Keys higher than context Keys, with the disparity increasing in deeper layers.Source: Figure 4 · p. 3, Figure 5 · p. 4

Pooling Attention

  • Pooling attention (Zhang et al.; 2025a) pools sequence dimensions into coarse Q, K, and V representations, then uses top-k coarse attention scores to derive the block mask.The coarse representations preserve local structure while remaining hardware-friendly, reducing complexity from O(NSD) to O(NQNKD).Source: §2 · pp. 3–4

3. In-Context Sparse Attention

ISA reduces in-context video-editing attention cost by pruning low-saliency context and routing queries between full attention and 0-th order Taylor sparse attention according to sharpness. LIVEditor-14B builds on ISA with staged training and a decoupled positional-embedding strategy.

Pre-Selection

  • Pooling attention ranks context Key/Value blocks, and retaining the top αs fraction reduces complexity from O(NSD) to O(N(Lsrc + αsLctx)D).Pre-selection uses the contiguous source/context token structure to identify context blocks for retention.Source: §3 · p. 4

Block-Wise 0-th Order Taylor Sparse Attention

  • Block-wise 0-th order Taylor sparse attention computes salient blocks exactly and approximates the rest, reducing interaction complexity from O(LQLKD) to O(LQD).First- and second-order Taylor variants were discarded because implementation trials found them ill-suited to hardware acceleration and computationally prohibitive.Source: §3 · pp. 4–5

Grouped Computation

  • Pooling-derived sharpness Mi correlates positively with Taylor error, so ISA routes high-sharpness queries to standard attention and low-sharpness queries to Taylor sparse attention, reaching 93.75% sparsity with negligible degradation.Theorem 3.1 bounds expected approximation error using block-wise sharpness; Figure 6 provides empirical support for the relationship.Source: Figure 6 · p. 4, §3 · p. 5

Implementation Detail and Analysis

  • Fine-tuning reduces ISA’s discrepancy from standard attention, while lowering αf and αns increases sparsity and yields monotonically greater computational speedup.The forward pass was implemented in Triton and TileLang, and the backward pass in Triton to make ISA trainable.Source: §3 · pp. 5–6, Figure 7 · p. 5, Figure 8 · p. 5

Results

  • On EditVerseBench, LIVEditor (ISA) outperforms related methods on nearly all metrics and full attention; training-free ISA beats sparse baselines on all metrics and full attention except Pick Score (Video).The table captions do not provide metric values.Source: Table 1 · p. 6, Table 2 · p. 6

LIVEditor-14B

  • LIVEditor-14B integrates ISA with pre-training on 1.7M mixed-quality samples, fine-tuning on 0.089M curated high-quality samples, and decoupled RoPE that resets source and context positions independently.The two-stage training balances generalization and precision, while decoupled RoPE addresses source/context video-length discrepancies and positional bias.Source: §3 · p. 6

4. Experiment

LIVEditor-14B (ISA) achieves leading results on EditVerseBench and best overall performance on VIE-Bench and IVE-Bench; experiments also compare ISA with other sparse attention methods and assess training stages and sparsity settings.

4.1. Setup

  • The experiments use four benchmarks and a two-stage post-training regimen on Wan 2.2, while the data pipeline combines self-constructed and public datasets.The self-constructed pipeline generates captions, editing tasks, instructions, and edited images; two training stages use 1.7M samples at learning rate 1e−5 and 0.089M high-quality samples at 1e−6, respectively.Source: §4.1 · pp. 6–7

4.2. Main Result

  • On EditVerseBench, LIVEditor-14B (ISA) scores 7.89 Quality, 20.09 Text Alignment, 27.19 Temporal Consistency, and 24.55 Editing Quality, exceeding prior bests on all four.Its Pick Scores are 99.32 on frames and 99.22 on video, exceeding previous highs of 98.56 and 98.44; it beats LIVEditor-14B (full-attn) on every metric except Video Pick Score.Source: §4.2 · p. 7
  • The paper reports that LIVEditor-14B (ISA) achieves the best overall performance, and separately evaluates it on VIE-Bench and IVE-Bench against an expanded set of state-of-the-art video-editing methods.Source: §4.2 · pp. 7–8
  • In training-free EditVerseBench comparisons, ISA significantly outperforms other sparse attention methods and LIVEditor-14B (full-attn), while being marginally slower than STA.The qualitative comparison reports that Radial Attention and Sparge Attention do not fully follow editing instructions, whereas ISA shows strong temporal consistency and executes the requested modifications.Source: §4.2 · p. 8, Figure 10 · p. 8

4.3. Ablation Studies

  • Stage II fine-tuning on a high-quality subset improves the model across all reported metrics on EditVerseBench and VIE-Bench.On EditVerseBench, the reported Stage I to Stage II values rise from 6.46 to 7.89 Quality, 19.50 to 20.09 Text Align, 25.27 to 27.19 Temporal Consistency, and 22.63 to 24.55 Editing Quality.Source: §4.3 · p. 8, §4.2 · p. 8, Table 4 · p. 8
  • ISA is most sensitive to Flat Ratio αf; its defaults are αf = 0.5, αns = 0.0625, and αs = 0.125, a configuration reported to yield 1.47x end-to-end speedup over full attention.Increasing sparsity by lowering αf significantly degrades performance; αns can be low without compromising performance, while αs shows an initial increase followed by a decrease on most metrics.Source: Figure 9 · p. 7, §4.3 · p. 8

5. Conclusion

LIVEditor-14B addresses the computational bottleneck of in-context video editing with In-context Sparse Attention (ISA), which uses query-sharpness analysis to enable experimentally lossless acceleration. Its data pipeline and benchmark evaluations support the model’s video-editing performance.

5. Conclusion

  • LIVEditor-14B significantly outperforms state-of-the-art methods in generation quality and inference latency on EditVerseBench and other benchmarks.The conclusion describes these as extensive evaluations.Source: §5 · p. 8
  • ISA dynamically routes queries and prunes redundant context tokens, drawing on theoretical analysis that links query sharpness to Taylor approximation errors.The conclusion characterizes the resulting acceleration as experimentally lossless.Source: §5 · p. 8
  • The work establishes a comprehensive data-processing pipeline to curate a high-quality video-editing dataset.Source: §5 · p. 8
  • The authors propose ISA’s principles as a scalable solution for future long-sequence video-generation tasks.Source: §5 · p. 8

A. Evaluation Benchmark … C.1. Data Processing Pipeline

The merged sections describe four-benchmark evaluation, ISA settings and training, and a four-stage synthetic video-editing data workflow.

A. Evaluation Benchmark

  • EditVerseBench has 200 instruction pairs across 20 categories, while IVE-Bench spans 600 prompts and 8 categories; experiments use only IVE-Bench’s short subset.EditVerseBench uses 100 videos split evenly between horizontal and vertical formats; IVE-Bench’s short subset contains 400 videos of 32–128 frames.Source: A. Evaluation Benchmark · p. 12
  • VIE-Bench evaluation omits reference-based editing, while FiVE-Bench assesses fine-grained object edits using 420 prompt pairs and the VLM-based FiVE-Acc metric.VIE-Bench uses GPT-4o to score instruction following, preservation, quality, and reference-task similarity; FiVE-Bench includes 100 videos and segmentation masks.Source: A. Evaluation Benchmark · p. 12

C.1. Data Processing Pipeline

  • The workflow generates target videos, filters them for source-target consistency and visual quality, then resamples and encodes source-target pairs and instructions into training-ready latents.It uses an internal 14-billion-parameter video diffusion model, YOLO, GroundingDINO, SAM, DOVER, VideoReward, VLMs, and Wan 2.1 VAE across generation, filtering, and encoding.Source: Figure 11 · p. 13, §C.1 · pp. 14–15

A. Evaluation Benchmark

  • On FiVE-Bench rigid and non-rigid object-replacement results, LIVEditor (ISA) outperforms LIVEditor (full-attn) on almost all metrics.The supplied tables cover rigid replacement (Tables 5 and 6) and non-rigid replacement (Tables 7 and 8).Source: Table 5 · p. 13, Table 6 · p. 14, Table 7 · p. 15

B. Hyperparameter Setting

  • Training uses approximately 1.7 million samples followed by 0.089 million fine-tuning samples, with independently reset RoPE positions for source and context tokens.The two stages use learning rates of 10^-5 and 5 × 10^-6, respectively; training uses 32 80GB GPUs and EMA decay 0.995 in stage two.Source: B. Hyperparameter Setting · p. 13
  • With a flat ratio of 0.5, no-sparsity ratio of 0.0625, and select ratio of 0.125, ISA achieves 1.47× speedup over FlashAttention-3 during full 32-step inference with CFG.The passage also reports superior performance under these settings.Source: B. Hyperparameter Setting · p. 13

C. Data Pipeline

  • The constructed dataset covers seven primary editing tasks, with global editing the largest category and fine-grained semantic edits also heavily represented.The distribution is intended to provide stylistic diversity and support instruction-following precision in localized regions.Source: Figure 12 · p. 14

C.1. Data Processing Pipeline

  • The data pipeline samples an edit task to create a target-image instruction, generates a target frame, and filters candidate frames using VLMs and image-quality metrics.Gemini-2.5-Pro or GPT-4o samples tasks and generates target-video prompts; Gemini 2.5 Image Preview synthesizes target frames, with HPSv2 and PickScore among the filtering signals.Source: §C.1 · p. 14

C.2. Data Distribution · C.3. Data Scheduling Process

The dataset spans seven editing modalities with over 1.7 million video-editing pairs, while training proceeds through mixed-quality pre-training and curated high-quality fine-tuning. The supplied FiVE-Bench captions also report that ISA outperforms full attention on almost all color-alteration metrics.

C.2. Data Distribution

  • Over 1.7 million video-to-video editing pairs cover seven primary modalities, including style transfer, object edits, human edits, and motion or viewpoint edits.Human Edit comprises Avatar Transformation and Face Detail Replacement; Other includes Motion and Viewpoint edits.Source: §C.2 · p. 15
  • Style Transfer forms 33.87% of the dataset, Human Edit 12.80%, and Other 1.09%.Human Edit is divided into Avatar Transformation and Face Detail Replacement; the smaller Other category includes complex motion and viewpoint tasks.Source: §C.2 · p. 15
  • The hierarchical distribution is intended to support generalization across video-editing requirements (Li et al., 2026; Liu et al., 2024; Shao et al., 2023; 2025; Shitong Shao, 2026; Yi et al., 2025).The passage highlights human priors and identity-preserving edits, as well as 3D geometry and temporal dynamics, as relevant capabilities.Source: §C.2 · p. 15

C.3. Data Scheduling Process

  • Pre-training combines approximately 1 million synthetic samples with 0.7 million Ditto samples and uses a mixture of high- and low-quality data.The model performed strongly on most tasks after this stage but showed limitations in object removal.Source: §C.3 · p. 15
  • Fine-tuning uses a final 0.089 million-sample set, including 0.03 million top-scoring samples selected from 0.06 million generated by Minimax Remover.The set also combines rigorously filtered public-dataset samples from Ditto, LoVoRA, and ReCo with internal data.Source: §C.3 · p. 15
  • The supplied FiVE-Bench color-alteration captions report that LIVEditor (ISA) outperforms LIVEditor (full-attn) on almost all metrics.This claim appears in both Table 9 and Table 10 captions; neither caption provides metric values.Source: Table 9 · p. 16, Table 10 · p. 16

D. Additional Analysis · E. Additional Experimental Result

The additional analysis identifies sharpness, Mi, as an effective proxy for Taylor approximation error: the reported bound is empirically dominated by this term rather than intra-block variance, which is computationally prohibitive to calculate.

D. Additional Analysis

E.1. Evaluation on FiVE-Bench · E.2. Evaluation on IVE-Bench · E.3. Evaluation on VIE-Bench

On FiVE-Bench, LIVEditor-14B (ISA) outperforms its full-attention counterpart across the reported editing tasks. On IVE-Bench it achieves a leading Total score, while on VIE-Bench it matches or surpasses full-attention across five editing categories.

E.1. Evaluation on FiVE-Bench

  • On FiVE-Bench rigid object replacement, ISA lowers Structure Distance to 71.37 from 81.31 and raises PSNR to 15.40 from 14.52 and Motion Fidelity to 58.48 from 55.73.FiVE evaluation also reports Accuracy of 50.71 versus 43.80 and FiVE-Union of 37.88 versus 25.86.Source: §E.1 · p. 16
  • For non-rigid object replacement, ISA lowers Structure Distance to 81.78 from 90.55 and LPIPS to 319.22 from 359.51, while raising Motion Fidelity to 57.99 from 54.59.FiVE metrics include Yes-No 57.00 versus 49.00, FiVE-Intersection 40.72 versus 40.21, and Accuracy 44.49 versus 42.38.Source: §E.1 · p. 16
  • On FiVE-Bench Color Alteration, ISA records PSNR 15.91, SSIM 54.18, Text Alignment 28.22, and FiVE Accuracy 83.18 versus full-attention's 79.77.Its Model Consistency is 90.00 versus 84.00 and FiVE-Union is 74.76 versus 73.74; the results are described as preserving video structure while modifying colors.Source: §E.1 · pp. 16–17
  • For Material Modification, ISA improves Structure Distance to 73.09 from 81.75, LPIPS to 310.98 from 386.07, and PSNR to 15.34 from 14.18.It also reaches Text Alignment 27.04 and Motion Fidelity 59.27; Table 11 covers Material Modification and reports ISA outperforming full-attention on almost all metrics.Source: §E.1 · p. 17, Table 11 · p. 17
  • For Object Addition, ISA lowers MSE to 299.82 from 394.66 and Structure Distance to 54.79 from 62.98, while Motion Fidelity rises to 76.39 from 57.05.ISA also attains PSNR 16.19 and CLIP-S 25.49; Table 12 reports it outperforming full-attention on almost all metrics.Source: §E.1 · p. 17, Table 12 · p. 17

E.2. Evaluation on IVE-Bench

  • On IVE-Bench, ISA achieves a Total score of 0.67, matching the best-performing baselines while offering a robust, balanced profile across evaluation dimensions.The supplied table caption characterizes LIVEditor (ISA) as the leading algorithm overall, with superior performance across 7 metrics.Source: §E.2 · p. 17, Table 13 · p. 18
  • On IVE-Bench, ISA scores 0.97 Subjective Quality, 0.98 Background Preservation, 0.99 Flickering, 1.00 Motion, and 0.47 Instruction versus full-attention's 0.43.It also records VTSS 0.040, Quantity 0.40, Fidelity 0.76, and Semantic consistency 0.91; Fidelity and Semantic consistency are comparable to full-attention and exceed AnyV2V and StableV2V.Source: §E.2 · pp. 17–18

E.3. Evaluation on VIE-Bench

  • On VIE-Bench, ISA averages 8.84 for Object Addition, 8.14 for Object Swapping, 8.16 for Video Stylization, and 8.12 for Hybrid (Bai et al., 2026).These exceed Omni-Video's 6.24, Pika's 7.41, InsV2V and Omni-Video on Video Stylization, and Omni-Video's 5.43, respectively; Object Removal scores are 8.57 for Background Preservation and 6.33 for Visual Quality, and ISA matches or surpasses full-attention across categories.Source: §E.3 · p. 18

F. The Proof of Error Bound of 0-th Order Taylor Approximation

The section derives a relationship between approximation error and attention sharpness, using Taylor analysis and a Lipschitz decomposition to establish a tightened bound.

F. The Proof of Error Bound of 0-th Order Taylor Approximation

  • A first-order perturbation analysis is used to rigorously relate approximation error to attention sharpness.The proof expands the softmax around approximate logits and uses a bounded Hessian to establish a strict value-level error bound for each token.Source: F. The Proof of Error Bound of 0-th… · p. 18
  • The proof links Jacobian stability to sharpness by adopting a conservative upper bound on the Jacobian norm in terms of the sharpness metric.This relation is stated for the active diffusion-model operating range, where distributions are concentrated but not yet one-hot.Source: F. The Proof of Error Bound of 0-th… · p. 18
  • The final tightened bound combines Taylor-based error control with a Lipschitz decomposition of the sharpness interaction.The proof identifies a bounded Lipschitz constant and uses the resulting decomposition to obtain the final bound.Source: F. The Proof of Error Bound of 0-th… · pp. 18–19

G. The Algorithm Implementation of ISA

This section presents pseudocode for ISA and Block-Wise 0-th Taylor Sparse Attention. It assumes identical sequence lengths and head dimensions for Q, K, and V, and omits batch size and head dimensions in the latter algorithm for clarity.

G. The Algorithm Implementation of ISA

  • The section provides pseudocode for ISA and Block-Wise 0-th Taylor Sparse Attention.The algorithms are presented as Algorithms 1 and 2, respectively.Source: G. The Algorithm Implementation of ISA · p. 19
  • The presentation assumes identical sequence lengths and head dimensions for Q, K, and V, and omits batch size and head dimensions in the Block-Wise algorithm.These simplifications are stated to clarify the pseudocode.Source: G. The Algorithm Implementation of ISA · p. 19

H. Implementation of 0-th Taylor Sparse Attention: Triton Vs. TileLang · I. Additional Explanation

The section explains why the authors use Triton for long video-editing sequences and why ISA can outperform full attention after training. It also describes ISA’s filtering mechanisms and clarifies the scope and limitations of its comparisons and error analysis.

H. Implementation of 0-th Taylor Sparse Attention: Triton Vs. TileLang

  • Triton outperforms TileLang as sparsity and sequence length increase; the authors use Triton for comparative analysis at video-editing sequence lengths of approximately 64K.The implementation comparison covers forward and backward passes of block-wise zeroth-order Taylor Sparse Attention on a Hopper GPU.Source: H. Implementation of 0-th Taylor… · pp. 19–20, Figure 17 · p. 23

I. Additional Explanation

  • The authors exclude recent MLLM-based methods and concurrent preprints from comparisons for implementation, benchmark, paradigm, and publication-status reasons.Excluded MLLM-based methods include EditVerse (Ju et al., 2025), UniVideo (Wei et al., 2025), and InstructX (Mou et al., 2025); LoVoRA (Xiao et al., 2025) and ReCo (Zhang et al., 2025c) are excluded as concurrent preprints not yet peer-reviewed.Source: §I · p. 20
  • After training, ISA outperforms standard full attention, which the authors attribute to pruning irrelevant noise tokens from the context.In in-context video editing, context tokens function primarily as conditional priors rather than direct generation targets.Source: §I · p. 20
  • ISA combines TopK context pre-selection with block-wise Taylor aggregation to isolate critical semantic information.TopK filters for semantic relevance, while the local-mean approximation acts as a secondary information-filtering mechanism.Source: §I · pp. 20–21
  • Block variance is not a reliable indicator of Taylor approximation error magnitude.The figure notes that its displayed error magnitudes are inflated without regularization and refers to the main-text results after regularization.Source: §I · p. 22, Figure 16 · p. 22
  • The ISA algorithm compresses blocks for coarse scoring, selects context blocks with TopK, splits queries by sharpness, and routes them to Taylor sparse or full attention.It reconstructs the output by scattering both query groups back and adding a coarse-output residual.Source: §I · p. 22

Related papers

Loading 2605.04569v2…