Source-linked AI summary
Sparkle: Realizing Lively Instruction-Guided Video Background Replacement via Decoupled Guidance
Ziyun Zeng, Yiqi Lin, Guoqiang Liang, Mike Zheng Shou
TL;DR
Background replacement is underexplored because existing datasets largely avoid the difficult combination of scene recreation, temporal consistency, and foreground preservation. Sparkle introduces a decoupled, quality-controlled data pipeline, a ∼140K-pair dataset, and Sparkle-Bench; its resulting model and data outperform existing baselines on the reported benchmarks.
Problem
Most existing video-editing datasets focus on local manipulation or style transfer, leaving background replacement underrepresented despite its demanding scene recreation and foreground-preservation requirements.
Method
Sparkle uses decoupled foreground and background guidance, compatible background-video generation, BAIT foreground tracking, and strict quality control to build training and evaluation resources.
Results
Sparkle comprises ∼140K high-quality video pairs, and its derivative Kiwi-Sparkle achieves substantial gains over existing baselines on OpenVE-Bench and Sparkle-Bench.
Takeaways & Limitations
The dataset, benchmark, and model provide open resources for more comprehensive research on instruction-guided background replacement.
Abstract
from arXiv · showhide
In recent years, open-source efforts like Senorita-2M have propelled video editing toward natural language instruction. However, current publicly available datasets predominantly focus on local editing or style transfer, which largely preserve the original scene structure and are easier to scale. In contrast, Background Replacement, a task central to creative applications such as film production and advertising, requires synthesizing entirely new, temporally consistent scenes while maintaining accurate foreground-background interactions, making large-scale data generation significantly more challenging. Consequently, this complex task remains largely underexplored due to a scarcity of high-quality training data. This gap is evident in poorly performing state-of-the-art models, e.g., Kiwi-Edit, because the primary open-source dataset that contains this task, i.e., OpenVE-3M, frequently produces static, unnatural backgrounds. In this paper, we trace this quality degradation to a lack of precise background guidance during data synthesis. Accordingly, we design a scalable pipeline that generates foreground and background guidance in a decoupled manner with strict quality filtering. Building on this pipeline, we introduce Sparkle, a dataset of ~140K video pairs spanning five common background-change themes, alongside Sparkle-Bench, the largest evaluation benchmark tailored for background replacement to date. Experiments demonstrate that our dataset and the model trained on it achieve substantially better performance than all existing baselines on both OpenVE-Bench and Sparkle-Bench. Our proposed dataset, benchmark, and model are fully open-sourced at https://showlab.github.io/Sparkle/.
1 Introduction
Background replacement remains underrepresented because it requires large-scale scene recreation and foreground preservation, while existing OpenVE-3M data often produces rigid or degraded edits. Sparkle addresses these issues with decoupled guidance, precise tracking, quality filtering, and a large dataset and benchmark.
- Most instruction-guided video datasets emphasize object manipulation or style transfer, neglecting large-scale background replacement for applications such as film production and advertising.
- OpenVE-3M models struggle to exceed 2.5/5.0 on OpenVE-Bench and often generate static backgrounds, rigid composites, or lost foreground subjects.
- OpenVE-3M’s synthesis pipeline lacks explicit background guidance, causing ignored background dynamics and structural collapse.
- Prompt misalignment and single-pass tracking can omit requested elements and lose fine-grained foreground details, such as changing an open hand into a fist.
- Sparkle generates compatible background videos independently, uses BAIT for high-precision foreground tracking, and synthesizes edits through decoupled foreground and background guidance.
- Sparkle contains ∼140K high-quality video pairs across five themes and introduces Sparkle-Bench with 458 videos across ∼100 scenes.
2 Related Work
Instruction-guided video editing datasets commonly use one-step V2V or two-step I2I + I2V generation, but these paradigms mainly serve simpler edits. Background replacement remains difficult because it demands scene recreation and foreground preservation.
- One-step V2V generation directly edits source videos and is mainly used for simpler tasks such as object removal.
- Two-step I2I + I2V generation edits a source frame first, then uses an in-context video generator to produce the video for more complex manipulations.
- These paradigms work well for local manipulation and style transfer because they avoid large-scale scene recreation and strict foreground preservation.
- OpenVE-3M extended the I2I + I2V paradigm to background replacement, but existing models still struggle because high-quality data remains scarce.
- Traditional video editing models use auxiliary signals such as Canny edges or depth maps, whereas newer instruction-guided models target natural-language editing.
3 Methodology
Sparkle constructs high-quality background-replacement data through staged filtering, independent background generation, precise foreground tracking, and decoupled guidance. The resulting dataset contains approximately 140K videos across five themes, while Sparkle-Bench provides a large dedicated evaluation benchmark.
- Pipeline Overview: Sparkle’s five-stage pipeline filters source videos, generates backgrounds independently, tracks foregrounds precisely, synthesizes edits with decoupled guidance, and filters outputs throughout.The pipeline uses fixed-camera videos, EditScore filtering, foreground removal and I2V background animation, BAIT tracking, and combined foreground/background Canny edges.
- Source Filtering: Fixed-camera filtering reduces the preliminary source pool from approximately 940K to approximately 260K videos for natural background detachment.Optical flow and homography estimation detect camera movement during coarse-to-fine filtering.
- Decoupled Guidance: Foreground-only guidance causes background structural collapse or motion loss because control models over-concentrate on the foreground without explicit background guidance.Sparkle addresses this failure by detaching foreground and background signals before synthesis.
- Individual Background Generation: Independent background generation removes the foreground before I2V animation, allowing the model to focus on required background dynamics such as waves, leaves, and clouds.The isolated background image is conditioned on a caption extracted from the editing prompt.
- BAIT Foreground Tracking: BAIT uses temporal bounding-box anchors, multiple forward and backward SAM3 passes, and majority pixel voting to produce precise foreground masks.Compared with single-frame tracking, consensus across temporal anchors suppresses missing foreground regions and background noise glitches.
- Decoupled Guidance: Sparkle combines source-video foreground edges and generated-background edges according to the BAIT mask, then uses the composite guidance with the edited first frame for synthesis.This creates a comprehensive control video containing decoupled foreground and background guidance.
- Dataset and Benchmark: Sparkle comprises approximately 140K videos across five relatively balanced themes, 22 subthemes, and approximately 100 diverse scenes.Sparkle-Bench contains 458 videos covering 97 distinct scenes across 21 subthemes and is described as the largest benchmark of its kind.
- Dataset and Benchmark: Sparkle-Bench evaluates background replacement across six criteria covering global, foreground, and background assessment, with scores from 1 to 5.The criteria include instruction compliance, visual quality, foreground integrity, foreground motion consistency, background dynamics, and background visual quality.
4 Experiments
Experiments assess Sparkle’s data quality, benchmark performance, synthesis choices, and generalizability. Sparkle improves data quality and enables Kiwi-Sparkle to outperform existing baselines across OpenVE-Bench and Sparkle-Bench.
- Data Quality Assessment: Sparkle achieves average gains of over 20% over OpenVE-3M across the OpenVE-3M subset and four additional themes.The largest improvements occur in Consistency & Detail Fidelity and Visual Quality & Stability.
- Performance on OpenVE-Bench: Kiwi-Sparkle gains 28% overall over Kiwi-Edit on OpenVE-Bench and surpasses competitors with three times more parameters.UniVideo and OmniVideo2 are cited as larger-parameter competitors.
- Performance on Sparkle-Bench: Sparkle-Bench evaluates broader background replacement capabilities across approximately 100 diverse scenes and four themes.The benchmark reports overall scores across Location, Season, Time, and Style, with theme-specific results in additional tables.
- Ablations: Copy-and-paste synthesis is tested against decoupled generation, while foreground-only guidance is tested against decoupled guidance.These comparisons evaluate whether direct compositing and omission of background guidance reduce video quality.
- Generalizability: Kiwi-Edit fine-tuned on the full Sparkle dataset is compared with models trained only on the OpenVE-3M subset to assess generalizability.The experiment tests whether the four themes beyond the OpenVE-3M subset provide broader improvements.
5 Conclusion
The paper identifies stale background edits as a consequence of mixed generation and addresses them with a five-stage decoupled pipeline. Sparkle, Sparkle-Bench, and Kiwi-Sparkle together provide improved data, evaluation, and model performance for background replacement.
- 5 Conclusion: The paper attributes stale edits to the conventional mixed generation paradigm and proposes a five-stage decoupled generation paradigm.The approach combines precise BAIT tracking, compatible background video generation, and strict quality control.
- 5 Conclusion: Sparkle provides a high-quality dataset, while Sparkle-Bench covers approximately 100 diverse scenes across 458 videos.The benchmark is presented as infrastructure for comprehensive evaluation of background replacement.
- 5 Conclusion: Kiwi-Sparkle demonstrates gains over existing baselines, extending the contribution from data and benchmarking to a trained derivative model.The paper positions this dataset, benchmark, and model as open infrastructure for future research.
A Coarse Camera Movement Filtering
The coarse filtering stage detects camera movement efficiently from optical flow and homography agreement, retaining sequences classified as static-camera.
- Coarse Filtering: The pipeline computes source-video optical flow at 2 FPS before applying coarse filtering to remove unqualified videos.Unimatch is used for optical-flow computation.
- Homography Estimation: A homography H models the dominant camera transformation between corresponding background points in consecutive frames.The formulation maps (x, y) to (x′, y′) using homogeneous coordinates.
- Homography Estimation: RANSAC estimates H from source points and flow-derived destinations, and aligned transformed points are classified as background.The ratio r measures the proportion of points satisfying the transformation.
- Movement Decision: A video is classified as having camera movement when r ≥50% and average motion magnitude m ≥1.Both conditions must hold between sampled frames to identify camera movement.
- Filtering Outcome: The filtering process reduces the source pool from 940K to 260K videos, retaining sequences whose sampled frames are free of camera movement.Sequences with no detected camera movement are classified as static-camera and retained.
B Detailed Evaluation Protocol on Sparkle-Bench
Sparkle-Bench uses Gemini-2.5-Pro to score background replacement across six dimensions on a 1-to-5 scale. The protocol emphasizes instruction compliance, foreground preservation and motion, background dynamics and quality, and foreground-background harmonization.
- Scoring Protocol: Gemini-2.5-Pro rates source and edited videos against the editing instruction across six dimensions on a 5-point scale.The protocol also requests a brief rationale before the final scores.
- Instruction Compliance: Instruction Compliance measures whether the edited background matches the requested content, style, mood, and dynamics while preserving the foreground.The scoring levels range from no relevant change or a failed edit to an exact prompt match with an untouched foreground.
- Overall Visual Quality: Overall Visual Quality includes global image quality and foreground-background harmonization of lighting, color temperature, and shadows.Time-of-day changes such as day to night require corresponding foreground lighting adaptation.
- Foreground Evaluation: Foreground Integrity and Foreground Motion Consistency assess preservation of the subject and fidelity of its temporal movement to the source.The motion scale ranges from completely different or scrambled movement to motion identical to the source.
- Background Evaluation: Background Dynamics evaluates whether motion intensity and character match the prompt rather than maximizing motion quantity.Gentle swaying grass and rushing water require different appropriate motion levels.
- Background Evaluation: Background Visual Quality rates structural sharpness, geometric coherence, texture stability, and temporal stability.The highest level describes a background sharp and stable enough to be on par with real footage.
- Scoring Constraints: Visual-quality and foreground/background criteria cannot exceed the Instruction Compliance score.This constraint prevents high visual scores from compensating for inaccurate instruction following.
C.1 Theme-specific Results on Sparkle-Bench
Theme-specific evaluation shows Kiwi-Sparkle is the only model surpassing 4.0/5.0 across all four reported themes, while Time remains the most challenging.
- Theme-specific results: Kiwi-Sparkle is the only model to surpass 4.0/5.0 across Location, Season, Time, and Style on Sparkle-Bench.Its high Background Dynamics and Background Visual Quality scores support accurate synthesis of the required elements.
- Theme-specific results: Foreground Integrity and Foreground Motion Consistency remain close to 4.0 for Kiwi-Sparkle.These scores indicate that the foreground remains well-preserved with consistent motion during editing.
- Theme-specific results: Time is the most challenging theme, with light and shadow adjustments remaining an area for improvement.Even on Time, Kiwi-Sparkle surpasses Lucy-Edit-1.1 by approximately 41%.
- Theme-specific results: Style is the highest-scoring theme for most models, suggesting that global style-transfer data can somewhat generalize to style-oriented background editing.The paper identifies this observation as warranting future investigation.
- Theme-specific results: Sparkle supports balanced refinement across themes and is suitable as a post-training corpus for background replacement.The paper attributes the more harmonious edits to its rigorous data pipeline.
C.2 Visualization
Visual comparisons show that Sparkle improves data quality and edited-video liveliness by supplying decoupled background guidance and regenerating full videos rather than relying on rigid compositing.
- Data visualization: OpenVE-3M frequently omits prompt-specific dynamic elements, including swaying curtains, flying seagulls, passersby, and floating motes.The comparison uses identical source videos and prompts across OpenVE-3M and Sparkle.
- Data visualization: Copy-and-Paste synthesis exhibits harsh contours and inadequate lighting and shadow adjustments, producing rigidly pasted results.The paper contrasts these outputs with more realistic videos generated through decoupled guidance and full-video regeneration.
- Data visualization: Removing background guidance causes severe structural collapse, high-frequency texture loss, and unstable lighting despite accurate foreground preservation.Examples include lost grass and flowers and sudden overexposure.
- Model visualization: Kiwi-Edit produces static, light-inconsistent backgrounds and misses required elements such as warm sunlight and falling snowflakes on OpenVE-Bench.Fine-tuning on Sparkle makes the edits more vibrant, lively, harmoniously lit, and motion-consistent without disturbing the foreground.
- Model visualization: On Sparkle-Bench, Kiwi-Edit shows static or light-inconsistent edits, whereas Kiwi-Sparkle achieves substantially stronger background dynamics and visual quality.The visual comparisons cover the Location and Season examples shown in the supplied passages.
- Foreground tracking: Kiwi-Sparkle also demonstrates foreground tracking and instruction-following by isolating foreground subjects in a minimalist clean white space.This test introduces the scene description as an editing category within the Style theme.
D License
Sparkle, Sparkle-Bench, and Kiwi-Sparkle are publicly released under CC-BY-4.0, while the code is released under Apache-2.0.
- Licensing: The dataset, benchmark, and model are released under the CC-BY-4.0 license, and the code is released under Apache-2.0.Source videos from OpenVE-3M remain subject to their original license and rights holders.