Source-linked AI summary
Soap2Soap: Long Cinematic Video Remaking via Multi-Agent Collaboration
Yiren Song, Huilin Zhong, Kevin Qinghong Lin, Haofan Wang, Mike Zheng Shou
TL;DR
Series-level cinematic remaking must preserve narrative, motion, and character identity across hundreds of shots, but existing pipelines accumulate drift and semantic inconsistency. Soap2Soap addresses this with multi-agent Dual-Bridge Consistency, visual memory, and batch keyframe synthesis, outperforming academic and commercial systems on identity, scene, and narrative consistency.
Problem
Series-level cinematic remaking lacks reliable mechanisms for preserving character identity, scene coherence, and narrative structure across hundreds or thousands of shots.
Method
Soap2Soap uses Dual-Bridge Consistency to couple scene-aware semantic screenplays with dynamically allocated visual anchors and contextual memory for each shot.
Results
Soap2Soap outperforms academic baselines and commercial video generation systems in identity stability, scene coherence, and narrative consistency on SoapBench.
Takeaways & Limitations
The results support explicit shot-level context allocation as important for maintaining character grounding and narrative continuity in long sequences.
Abstract
from arXiv · showhide
We study series-level cinematic remaking, a long-horizon video-to-video generation problem that localizes full episodes or films via stylization or actor replacement while strictly preserving narrative structure, motion choreography, and character identity across hundreds of shots. Existing video generation and editing pipelines often break down in this regime due to compounding identity drift, background mutation, and semantic erosion under large camera motions and viewpoint changes. We propose Soap2Soap, a multi-agent framework that enforces long-term language-visual consistency through a Dual-Bridge Consistency mechanism: a scene-aware JSON screenplay serving as a persistent semantic backbone, and dynamically allocated visual reference anchors at both scene and shot levels. To suppress drift before video synthesis, we introduce batch keyframe consistency, jointly generating multiple keyframes in a shared latent context via a grid-based formulation. A closed-loop verification agent further audits identity, stability, and alignment to trigger selective regeneration. Experiments on SoapBench demonstrate strong improvements over commercial video generation APIs in long-term consistency and narrative fidelity.
1 Introduction
Series-level cinematic remaking adapts films or series through stylization, actor replacement, or cultural re-contextualization while preserving narrative, choreography, emotion, and character consistency across hundreds to thousands of shots. Soap2Soap addresses this long-horizon challenge with explicit semantic and visual consistency mechanisms, evaluated on the SoapBench benchmark.
- Cinematic remaking localizes existing films or series through actor replacement, style adaptation, or cultural re-contextualization while preserving narrative, choreography, and emotional dynamics.Unlike short video generation, it spans hundreds of shots with complex camera language, multi-character interactions, and long-range narrative dependencies.
- Series-level remaking must explicitly enforce consistency across extremely long horizons, often spanning hundreds to thousands of shots.Physical production benefits from stable actors, coherent environments, continuous lighting, and camera setups, whereas generative remaking lacks these guarantees.
- The task couples shot-granular long-video understanding, viewpoint-robust character migration, and long-horizon consistency control.Character migration must preserve motion, interactions, and dramatic intent beyond face swapping, while maintaining stable character and environment appearance.
- Soap2Soap enforces consistency through a structured JSON screenplay, dynamically allocated visual anchors, contextual memory, and batch keyframe generation within a multi-agent framework.Its Video Understanding Agent extracts shot-level narrative events, character roles, cinematic intent, and shot-specific anchors for downstream generation.
- SoapBench evaluates long-video understanding and remaking consistency in multi-shot, multi-character scenarios, where Soap2Soap outperforms academic baselines and commercial systems.Reported advantages include identity stability, scene coherence, and narrative consistency in extensive experiments and human studies.
2 Related Works
Related work has advanced from frame-level analysis toward structured screenplay generation, while long-form video generation and multi-agent systems have evolved to address narrative complexity and task decomposition. However, existing approaches remain constrained by short clip durations and the challenge of cumulative errors.
- Long Video Reasoning: Long video reasoning now emphasizes structured screenplay generation and episodic analysis with Vid-LLMs rather than simple frame-level analysis.MM-VID and ScreenWriter use specialized vision-audio tools and MDL principles to segment scenes and identify characters in complex narratives.
- Long-Form Video Generation: Long-form video generation has progressed from U-Net diffusion architectures toward unified Diffusion Transformer frameworks, but computational limitations typically restrict outputs to 4–8 second clips.The passage describes this progression as occurring across several distinct phases.
- Multi-Agent Systems: Multi-agent video frameworks decompose complex generation into modular, role-specialized components, using hierarchical production roles and chain-of-thought planning to improve narrative logic.Recent systems also explore global asset memory and closed-loop planning to suppress cumulative error propagation.
3 Method
Soap2Soap treats cinematic remaking as a long-horizon video-to-video task that changes actor identities or visual styles while preserving narrative structure, motion choreography, and audiovisual coherence. Its multi-agent pipeline uses semantic and visual bridges, contextual references, joint keyframe synthesis, and closed-loop verification to maintain consistency across long sequences.
- Task formulation: Soap2Soap remakes source videos with new actor identities or visual styles while preserving storyline, shot order, motion choreography, and audiovisual coherence.The task explicitly evaluates semantic fidelity and identity correctness over extreme temporal horizons, where errors can accumulate into identity drift, scene mutation, or narrative corruption.
- Dual-Bridge Consistency: Dual-Bridge Consistency separates semantic and cinematic structure from visual appearance through a scene-aware JSON screenplay Sjson and visual memory M.Sjson records narrative events, character participation, camera language, actions, and storyline progression, while M stores scene and shot-level visual anchors.
- Contextual memory allocation: A context-aware allocation mechanism dynamically constructs compact shot-specific memory packages instead of loading the entire global context for every shot.Each package contains relevant semantic instructions and visual anchors, including shot descriptions, text-to-image and image-to-video prompts, and appearing-character references.
- Anchor-driven rendering: Anchor-driven rendering first jointly synthesizes 4 or 9 same-scene frames as a 2 × 2 or 3 × 3 grid, then generates and stitches 4–8 second shot clips.Shared attention during grid synthesis improves consistency under large viewpoint changes, while Veo 3 performs image-to-video generation conditioned on shot-level memory and keyframes.
- Closed-loop verification: The Verification Agent applies a closed-loop Critique–Correct–Verify process that audits quality, identity, environment and style, and plot consistency before selectively regenerating affected shots.Structured feedback updates screenplay and spatial controls, with configurable retry limits balancing generation quality and computational cost without full sequence rollbacks.
4 SoapBench: A Benchmark for Long-Video Remaking
SoapBench is a benchmark for evaluating training-free long-video remaking under complex multi-shot narratives. It provides separate tracks for long-video understanding and remaking, covering structured cinematic content and consistency across extended sequences.
- Benchmark Design: SoapBench evaluates long-video remaking through Long Video Understanding and Long Video Remaking tracks, targeting shot boundaries, scenes, identities, narrative structure, and camera motion.The benchmark is designed for complex multi-shot narratives and systematic evaluation of long-form cinematic remaking.
- Long Video Understanding: 10 movies with detailed screenplays yield 607 shots for understanding evaluation, with manually verified character annotations providing reliable ground truth.The movies and scripts are sourced from IMDb, and the annotations support shot-level character understanding evaluation.
- Long Video Remaking: 10 movies contribute continuous 1.5–5 minute remaking segments, including sequences of up to 42 consecutive shots.The remaking track evaluates cross-shot character identity and appearance consistency, scene consistency, and narrative fidelity.
- Remaking Scenarios: SoapBench covers real-to-stylized transformation and live-action re-casting while preserving original motion choreography and narrative structure.Examples of stylization include LEGO- or Disney-like rendering, while re-casting uses target character reference images for realistic re-shooting.
5 Experiments
Soap2Soap is evaluated on SoapBench for long-video understanding and remaking, testing shot-level screenplay alignment and long-horizon identity, scene, and narrative consistency. It outperforms Mocha, Kling O1, and Runway Gen4, while ablations show that dynamic memory allocation and closed-loop verification are essential for sustained consistency.
- Baselines: Soap2Soap is compared with Mocha, Kling O1, and Runway Gen4 using the same reference images for identity and style guidance.SeedDance 2.0 is excluded because API access was unavailable.
- Evaluation Tasks: SoapBench evaluates long-video understanding against aligned IMDb screenplays and long-video remaking for identity and scene consistency under long-horizon generation.Understanding is measured at shot level, while remaking assesses preservation of identity and scene consistency.
- Quantitative Evaluation: Soap2Soap achieves the best performance across all long-video remaking metrics, including identity, scene, narrative consistency, and visual similarity.The evaluation reports Gemini-based VLM scores for ID, Scene, and Plot, alongside CLIP Image Scores for identity and scene similarity.
- Ablation Studies: Disabling dynamic memory allocation drops IoU from 0.921 to 0.569 and F1 from 0.936 to 0.618, causing identity inconsistencies.The ablation replaces dynamic allocation with a static global description, showing that precise shot-level context allocation supports character grounding and narrative continuity.
- Ablation Studies: Removing the critique–correct–verify loop lowers F1 from 0.936 to 0.887 and ID-VLM from 9.17 to 8.91, while long-horizon identity and semantic errors accumulate.The Verification Agent is ablated by generating each chunk in a single pass without closed-loop auditing.
6 Conclusion
Soap2Soap is a multi-agent framework for cinematic remaking that addresses character identity drift and semantic erosion. It uses Dual-Bridge Consistency with scene-aware JSON screenplays and dynamically allocated visual anchors to support narrative fidelity and visual stability across hundreds of shots.
- Framework: Soap2Soap is a multi-agent framework designed to address character identity drift and semantic erosion in cinematic remaking.The framework targets these challenges directly.
- Dual-Bridge Consistency: Dual-Bridge Consistency decouples the remaking process through scene-aware JSON screenplays and dynamically allocated visual anchors.These components provide the framework’s semantic and visual consistency mechanism.
- Long-Horizon Consistency: The approach is designed to ensure narrative fidelity and visual stability across hundreds of shots.The stated goal is maintaining both properties over long cinematic sequences.
Appendix
The appendix covers the user study, multi-agent system implementation, structured JSON output format, grid joint synthesis visualization, and additional keyframe results. Together, these sections document evaluation protocols, system components, data structures, consistency strategies, and visual comparisons.
- Section A: User Study: Section A details the user-study protocol, including evaluation criteria, compared methods, questionnaire design, and preference results.
- Section B: Multi-Agent System Implementation Details: Section B describes the Video Understanding, Video Generation, and Verification Agents, plus contextual memory allocation and reference generation supporting Dual-Bridge Consistency.
- Section C: Output JSON Format: Section C presents the video-understanding module’s structured JSON format, including organization, character rosters, and shot-level cinematic and narrative analysis.
- Section D: Grid Joint Synthesis Visualization: Section D visualizes grid joint synthesis for intra-scene consistency through 2 × 2 and 3 × 3 grid-generation examples.
- Section E: More Keyframe Results: Section E provides 7 additional visual comparisons across videos ranging from 14 to over 40 shots, with keyframe-by-keyframe analyses of visual fidelity and consistency.
A User Study
A user study evaluates Soap2Soap’s perceptual quality for series-level cinematic remaking using participant preferences across character, scene, plot, and overall criteria. The method achieves the highest preference across all four criteria.
- Study design: The study collects 20 anonymous questionnaires from students and practitioners experienced in AI-based film, video generation, or video production.Each questionnaire presents 10 groups of remade samples.
- Evaluation criteria: Participants evaluate Character Consistency, Scene Consistency, Plot Consistency, and Overall Preference.These four criteria assess perceptual quality in series-level cinematic remaking.
- Results: Soap2Soap achieves the highest user preference across all four evaluation criteria.The result is reported in Fig. 6’s preference distributions.
B Multi-Agent System Implementation Details … C Output Json Format
Soap2Soap implements Dual-Bridge Consistency through coordinated language, visual, generation, and verification agents. A structured JSON screenplay, adaptive memory allocation, reference-driven synthesis, and selective regeneration preserve semantic fidelity and visual coherence across long videos.
- B Multi-Agent System Implementation Details: The system coordinates three collaborative agents around a Language Bridge and Visual Bridge to maintain semantic fidelity and visual coherence.The Language Bridge is the structured JSON screenplay, while visual references provide the Visual Bridge.
- B.1 Video Understanding Agent: The Video Understanding Agent converts unstructured video into Sjson using multimodal character identification, adaptive scene detection, environment-only descriptions, wardrobe DNA, and shot-level prompts.It combines on-screen labels, dialogue analysis, and visual feature matching, while encoding action flow, camera movement, temporal progression, and cinematographic parameters.
- B.2 Contextual Memory Allocation: Dynamic memory packages prevent redundant global-context loading by pairing semantic instructions with character, clothing, and environment visual anchors.Scene-level environment references are shared across shots, while shot-level character and clothing references adapt to appearances and disappearances.
- B.3 Reference Generation: Reference generation creates environment and clothing images as Visual Bridge carriers, preserving environment-only content, source aspect ratio, and target artistic style.Environment references contain spatial structure, lighting atmosphere, and background elements without character information.
- B.4 Anchor-Driven Visual Generation: The Video Generation Agent jointly synthesizes 4 or 9 scene-consistent keyframes in 2 × 2 or 3 × 3 grids to handle large viewpoint variation.It then uses Veo 3 for 4 to 8 second clips, combining scene, character, and keyframe references with Sjson-derived I2V prompts.
- B.4 Anchor-Driven Visual Generation: Generation prompts integrate visual DNA, narrative context, reference paths, and style prompts while enforcing aspect-ratio preservation, text removal, and quality control.Reference images follow a priority mechanism during conditioning.
- B.5 Verification Agent: The Verification Agent audits quality, identity and appearance, stability, and alignment, then sends structured feedback to update screenplay descriptions, spatial controls, memory packages, and I2V prompts.Selective regeneration targets affected shots rather than rolling back the full sequence.
- C Output Json Format: The Video Understanding Agent outputs Sjson as a four-part Language Bridge containing video metadata, a character roster, major scenes, and shot-level information.The format records technical specifications, character profiles, scene timing and environments, and related narrative and visual controls.
C.4 Shot-Level Analysis Structure
The shot-level structure combines technical cinematic parameters with narrative descriptions, action, dialogue, timestamps, and generation prompts. It specifies both static visual properties and dynamic shot behavior for consistent video synthesis.
- Technical Cinematic Parameters: Each shot records scene and technical parameters including lighting, color grading, composition, mood, shot size, camera angle, height, horizontal angle, and focal length.The example uses three-point lighting, warm high-contrast grading, rule-of-thirds composition, a medium close-up, eye-level camera, and a 50mm lens.
- Narrative Descriptions: Narrative fields encode the character’s action and associated dialogue within the shot.The example specifies that the character turns their head toward the door and includes spoken dialogue.
- Generation Prompts: Timestamped dialogue is paired with I2V and one-shot prompts that describe motion, expression, camera movement, duration, framing, lighting, and depth of field.The example uses a 4-second dynamic sequence in which the character turns toward the door, becomes concerned, moves subtly, and is filmed with a slight forward camera push.
D Grid Joint Synthesis Visualization
Soap2Soap generates multiple scene-consistent keyframes jointly on a shared grid canvas rather than sequentially, addressing identity and environmental inconsistencies caused by large viewpoint changes. The method uses 2 × 2 or 3 × 3 grids according to scene complexity and shot count.
- Challenge: Sequential keyframe generation can accumulate inconsistencies in character identity, clothing details, and environmental attributes during large viewpoint changes.The problem is especially pronounced in reverse-shot sequences that shift between over-the-shoulder and reverse angles.
- Grid Joint Synthesis: Grid joint synthesis generates 4 or 9 same-scene keyframes as one unified canvas in a single generation pass.Shared attention establishes cross-frame consistency during generation rather than treating each keyframe independently.
- Grid Joint Synthesis: Intrinsic consistency enforcement during denoising strengthens identity preservation and scene coherence across grid frames while improving computational efficiency over sequential generation.After generation, the complete grid is decomposed into individual keyframes for subsequent use.
- Grid Configurations: The 2 × 2 configuration targets shorter segments with 4 consecutive shots, whereas the 3 × 3 configuration supports complex scenes involving up to 9 shots.The choice depends on scene characteristics and the number of shots requiring consistent generation.
E More Keyframe Results
Figures 8–14 provide additional keyframe-by-keyframe comparisons across benchmark test videos, illustrating visual fidelity and consistency against original frames across multiple source films and styles.
- Keyframe Comparisons: Additional keyframe comparisons demonstrate visual fidelity and consistency against original video frames across the benchmark test videos.The comparisons are presented in Figures 8 through 14.
- Film and Style Coverage: The examples include Braveheart, Forrest Gump, and The Pursuit of Happiness in realistic or anime styles.Figure 8 uses Realistic styling, Figure 9 uses Anime styling, and Figure 10 uses Realistic styling.
- Film and Style Coverage: Additional examples cover Avengers: Infinity War, The Shawshank Redemption, Green Book, and Infernal Affair in Lego, Clay, or realistic styles.Figures 11–14 show Lego, Clay, Clay, and Realistic styles, respectively.