Source-linked AI summary
MuSS: A Large-Scale Dataset and Cinematic Narrative Benchmark for Multi-Shot Subject-to-Video Generation
Haojie Zhang, Di Wu, Bingyan Liu, Linjie Zhong, Yuancheng Wei, Xingsong Ye, Nanqing Liu, Yaling Liang
TL;DR
MuSS addresses the lack of datasets and benchmarks for coherent multi-shot cinematic narratives and non-copy-paste Subject-to-Video generation. Its dual-track dataset, cross-shot matching, and Cinematic Narrative Benchmark improve storytelling effectiveness, structural grounding, and identity consistency over current baselines.
Problem
Existing video datasets and benchmarks largely focus on isolated single-shot generation, lacking authentic multi-shot narrative logic and evaluation of spatiotemporal storytelling.
Method
MuSS combines a large-scale dual-track cinematic dataset with progressive captioning, cross-shot subject matching, and a visual-logic-driven Cinematic Narrative Benchmark.
Results
MuSS-augmented models achieve state-of-the-art storytelling effectiveness, structural grounding, and cross-shot identity consistency, while current baselines struggle with cinematic multi-shot scenarios.
Takeaways & Limitations
MuSS provides a unified resource for training and evaluating continuous cinematic storytelling and 3D-consistent Subject-to-Video generation.
Takeaways & Limitations
Some baselines lack external reference inputs, making ACP-Var and Copy-Paste Rate structurally inapplicable for their identity evaluation.
Abstract
from arXiv · showhide
While video foundation models excel at single-shot generation, real-world cinematic storytelling inherently relies on complex multi-shot sequencing. Further progress is constrained by the absence of datasets that address three core challenges: authentic narrative logic, spatiotemporal text-video alignment conflicts, and the "copy-paste" dilemma prevalent in Subject-to-Video (S2V) generation. To bridge this gap, we introduce MuSS, a large-scale, dual-track dataset tailored for multi-shot video and S2V generation. Sourced from over 3,000 movies, MuSS explicitly supports both complex montage transitions and subject-centric narratives. To construct this dataset, we pioneer a progressive captioning pipeline that eliminates contextual conflicts by ensuring local shot-level accuracy before enforcing global narrative coherence. Crucially, we implement a cross-shot matching mechanism to fundamentally eradicate the S2V copy-paste shortcut. Alongside the dataset, we propose the Cinematic Narrative Benchmark, featuring a visual-logic-driven paradigm and a novel Anti-Copy-Paste Variance (ACP-Var) metric to rigorously assess continuous storytelling and 3D structural consistency. Extensive experiments demonstrate that while current baselines struggle with continuous narrative logic or degenerate into trivial 2D sticker generators, our MuSS-augmented model achieves state-of-the-art narrative effectiveness and cross-shot identity preservation.
1 Introduction
MuSS addresses the limitations of single-shot video datasets and current S2V methods by introducing a large-scale cinematic multi-shot dataset with progressive annotation and cross-shot subject matching. It also proposes a Cinematic Narrative Benchmark centered on visual logic and the Anti-Copy-Paste Variance metric for evaluating storytelling and subject consistency.
- Motivation: Existing datasets and frameworks mainly support isolated single-shot videos with simple single-subject actions, whereas cinematic production requires complex multi-shot sequencing.The introduction identifies scarce narrative logic and conflicts between local shot accuracy and global caption coherence as key dataset-construction challenges.
- Dataset: MuSS is derived from over 3,000 movies and contains approximately 700K filtered shots and more than 30K captioned multi-shot sequences for multi-shot video and S2V generation.The sequences are selected through multi-dimensional filtering involving aesthetics, motion, and semantic consistency.
- Benchmark: The Cinematic Narrative Benchmark is a dual-track, visual-logic-driven evaluation suite designed to measure storytelling under realistic cinematic conditions rather than only global quality and basic text alignment.Its stated contributions include Multi-Dimensional Visual Logic and the Anti-Copy-Paste Variance (ACP-Var) metric.
- Dataset Construction: MuSS combines progressive VLM annotation with precise cross-shot subject matching, using alternate-shot references to encourage novel views and prevent the S2V “copy-paste” shortcut.This addresses the tendency of existing personalized S2V models to replicate a reference subject’s pose and lighting, degrading generalization to novel views across shots.
2 Related Work
Prior work has advanced coherent long-video generation through long-range attention, hierarchical sequencing, and memory-based continuity, while customized video generation increasingly targets identity preservation across views and scenes. However, existing S2V datasets emphasize isolated single-shot actions, encouraging copy-paste shortcuts and leaving multi-shot 3D identity preservation insufficiently benchmarked.
- Long-video generation: Long-video generation has progressed from temporal extrapolation toward narrative modeling through consistent self-attention, hierarchical visual sequencing, and memory-based contextual continuity [60] [58] [54].StoryDiffusion [60] supports long-range generation, MovieDreamer [58] models coherent visual sequences hierarchically, and LetsTalk [54] mitigates error accumulation in extended talking videos.
- Customized video generation: Customized generation increasingly addresses strict identity preservation across varying views and scenes using spatial priors and video diffusion transformers [46] [7] [27] [55].WithAnyone [46], MultiRef [7], and OpenSubject [27] provide image-level spatial priors extended to video, while Magic Mirror [55] uses video diffusion transformers.
- Customized video generation: Existing S2V datasets mainly cover isolated single-shot actions, encouraging copy-paste shortcuts and failing to rigorously benchmark 3D identity preservation across dynamic multi-shot cinematic transitions.This limitation motivates evaluation protocols that test identity consistency beyond isolated actions and across cinematic transitions.
3 MuSS Dataset Construction
MuSS is constructed from over 3,000 movies through two phases: coherent multi-shot dataset creation and cross-shot-matched Subject-to-Video pair curation. The pipeline combines shot filtering, progressive narrative captioning, and zero-shot subject extraction to improve alignment and 3D identity consistency.
- Dataset Construction: MuSS contains approximately 700K filtered single shots, more than 30,000 professionally captioned multi-shot clips, and over 1,000 hours of video from over 3,000 movies.The dataset statistics include clip durations, caption lengths, caption vocabulary, and clips per source video.
- Multi-Shot Foundation: The construction pipeline first preprocesses movies, detects shot boundaries with TransNetV2, and filters motion to retain usable cinematic dynamics.Preprocessing removes watermarks and black borders; motion filtering excludes frozen or excessively chaotic shots while retaining low-motion dialogue and establishing shots.
- Progressive Coherent Captioning: Progressive captioning first recaptions individual shots and removes VideoCLIPXL [47] pairs scoring below 0.20, then aggregates consecutive shots into narrative-coherent sequences.A film-director-assistant VLM globally refines captions using keyframes and initial captions under entity, coreference, contextual-consistency, and structured-formatting constraints.
- Subject-to-Video Pair Construction: S2V curation uses zero-shot subject extraction and cross-shot matching to prevent the “copy-paste” shortcut while preserving 3D identity consistency.Qwen2.5-VL-7B [2] and DeepSeekV3 [23] produce subject tags, GroundingDINO [25] detects initial-frame objects, and SAM 2.1 [34] generates pixel-level masks.
4 Cinematic Narrative Benchmark
The Cinematic Narrative Benchmark is a dual-track MuSS suite designed to assess storytelling, cross-shot stability, spatiotemporal control, and genuine 3D consistency beyond coarse single-shot metrics. Its Visual-Logic Driven paradigm combines LMM reasoning with domain-expert perceptual models, while specialized metrics expose temporal, identity, and copy-paste failures.
- Benchmark Overview: The benchmark addresses gaps in existing single-shot evaluations by measuring storytelling, cross-shot visual stability, and spatiotemporal control through two MuSS-derived tracks.Existing benchmarks rely on coarse global assessment and do not adequately capture these cinematic properties.
- Evaluation Paradigm: Its Visual-Logic Driven paradigm combines Gemini-2.5-Flash reasoning with DINOv2, TransNet V2, RAFT, YOLOv11, and SAM for human-level structural assessment without generic global text priors.The approach combines visual reasoning with domain-specific perceptual fidelity because perfect global captions are impractical at scale.
- Track 1: Track 1 measures local shot-text alignment, transition precision, global visual-world preservation, motion dynamics, and narrative rhythm using Txt.Align, Scene.Con, and Con.Gap.RAFT filters out static slideshow-like generations before Jensen-Shannon Distance against professional film-edit coherence distributions computes Con.Gap.
- Track 2: Track 2 separates reference fidelity from internal identity preservation and evaluates authentic spatiotemporal grounding without penalizing legitimate cinematic camera changes.Ref-Sub.Con measures fidelity to the external reference, whereas Inter-Sub.Con measures identity consistency across generated shots.
- Track 2: ACP-Var, Simpose, CP-Rate, and Act.Str detect 2D sticker-like copying by assessing structural and pose diversity, keypoint similarity, feature-entropy collapse, and action tempo.A near-zero DINOv2 similarity entropy indicates trivial 2D duplication, while Procrustes-aligned keypoints expose rigid reference-posture collapse.
- Evaluation Setup: The study manually verifies 100 MuSS test prompts and evaluates storyboard, native multi-shot, customized S2V, and physical copy-paste baselines across stratified cinematic conditions.The prompts cover shot count, scene type, dialogue/action rhythm, framing, and subject count, with 50 prompts per track.
5 Experiments
Experiments show that MuSS improves continuous multi-shot narrative logic and subject identity preservation, overcoming spatial hallucinations and copy-paste behavior that limit existing baselines. Human evaluations further support the benchmark metrics’ alignment with professional judgments.
- Track 1: Narrative Effectiveness: MuSS-augmented models sweep Track 1’s visual-logic metrics while maintaining multi-shot consistency and competitive spatiotemporal transition precision.The benchmark evaluates narrative effectiveness across continuous storytelling and cinematic transitions.
- Track 2: Subject Consistency: MuSS breaks the S2V copy-paste shortcut while achieving state-of-the-art grounding and internal identity preservation among customizable baselines.Phantom and VACE show high reference-subject consistency but poor inter-subject consistency, revealing rigid 2D image-translation behavior; ACP-Var and CP-Rate are unavailable for methods without external references.
- Track 1: Narrative Effectiveness: MuSS preserves spatial topology and cast continuity across perspective-changing transitions, whereas existing baselines hallucinate backgrounds, shift lighting or architecture, and lose characters.Concatenation-based methods struggle with continuous storytelling, while native multi-shot models can fail four-dimensional visual-logic tests.
- Metric Validation: Fifteen professional filmmakers’ ratings of 200 generated sequences support strong alignment between ACP-Var, visual-logic metrics, and judgments of motion naturalness, perspective richness, and visual continuity.ACP-Var penalizes rigid 2D sticker effects that traditional metrics overlook, while Scene.Logic aligns with expert assessments of continuity.
6 Conclusion
The work addresses incoherent multi-shot cinematic narratives and the copy-paste shortcut in Subject-to-Video synthesis through MuSS and a Cinematic Narrative Benchmark. MuSS combines progressive captioning with cross-shot matching to support authentic identity preservation.
- MuSS targets two bottlenecks: incoherent multi-shot cinematic narratives and the pervasive copy-paste shortcut in Subject-to-Video synthesis.
- The dataset uses progressive captioning to address the challenges of multi-shot video generation.
- A rigorous cross-shot matching mechanism is designed to guarantee authentic identity preservation in Subject-to-Video synthesis.
- The work also introduces the Cinematic Narrative Benchmark for evaluating the proposed multi-shot and Subject-to-Video generation setting.
Supplementary Material … B.2 Prompt Templates for Progressive Captioning
The supplementary material specifies MuSS’s reproducible data access, cascaded filtering, cross-shot reference construction, and progressive-captioning prompt templates. Together, these details operationalize dataset curation while addressing quality, identity reliability, and spatiotemporal alignment.
- A Data Source and Access: MuSS distributes YouTube video identifiers and timestamp annotations rather than raw videos, leaving source copyright and licensing with providers and rights holders.The dataset is built from publicly accessible YouTube videos, but the raw files are not directly distributed.
- A Data Source and Access: Automated download and trimming scripts reproduce the dataset locally from released identifiers and annotated multi-shot boundaries, with users responsible for lawful source-content use.The repository provides download_muss.sh and download_muss.py using open-source fetching and trimming tools.
- B Extended Dataset Construction Details: The supplementary section provides operational parameters for reproducing the MuSS curation pipeline introduced in the main manuscript.These details cover extended dataset construction, filtering, cross-shot references, and captioning prompts.
- B.1 Multi-Dimensional Cascaded Filtering Thresholds: MuSS applies cascaded filtering to single-shot candidates before sliding-window sequence construction, retaining low-motion dialogue and establishing shots while excluding frozen or excessively chaotic content.The filtering pipeline also targets severe intra-shot semantic drift, poor visual aesthetics, and inadequate temporal quality; Table S1 lists the empirical thresholds.
- B.1 Multi-Dimensional Cascaded Filtering Thresholds: Cross-shot references are selected from the same storyline but outside target clips using subject tags, entity captions, GroundingDINO and SAM 2.1 masks, visual matching, and verifier checks.Candidate pairs must meet a multimodal text/image similarity threshold of 0.6 before further verification.
- B.2 Prompt Templates for Progressive Captioning: Table S2 documents the exact System and User prompt templates used with Large Multimodal Models for Progressive Two-Stage Coherent Captioning.The templates are provided to improve transparency and reproducibility when resolving spatiotemporal text-alignment conflicts.
C Extended Benchmark Implementation Details
The benchmark uses transparently documented, stratified LMM prompts and visually grounded MDVL evaluation. Gemini-2.5-Flash scores four cinematic-continuity dimensions from uniformly sampled keyframes paired with global and local prompts.
- Prompt Construction: MDVL prompts are sampled from MuSS and balanced across tracks, shot counts, scene types, dialogue/action rhythm, composition, and subject configurations, with manual inspection.The prompt set is designed to cover diverse benchmark conditions while supporting reproducibility.
- MDVL Evaluation: Gemini-2.5-Flash [9] evaluates four cinematic-continuity dimensions using a 2×N visual grid of two uniformly sampled keyframes per generated sub-shot.The grid is provided alongside the global narrative description and local shot prompts to enforce visually grounded reasoning.
D Implementation & Training Details
The MuSS-augmented baseline builds on EchoShot [40], using latent sequence concatenation to inject identity priors. Training uses specified optimization, video-processing, and compute settings to reach convergence.
- Model Construction: The MuSS-augmented baseline is built on EchoShot [40] with latent sequence concatenation for identity-prior injection.
- Optimization: Training uses AdamW with β1 = 0.9, β2 = 0.999, weight decay = 10−4, a 1×10−5 learning rate, 2,000-step linear warmup, and 50,000 total steps.
- Video Processing: Videos are processed at 832 × 480 resolution and 16 fps, with 161-frame temporal context through a multi-shot sliding-window approach.
- Compute and Preparation: Training runs on 32 NVIDIA H20 GPUs and requires approximately 3.5 days to reach convergence, while reference preparation uses automated curation.
E User Study Details · F Extended Dataset Visualizations and Limitations · F.1 More Visualizations of the Dataset
A blind expert study evaluates MuSS-generated sequences against cinematic criteria using a standardized 1–5 rubric, while extended visualizations illustrate the dataset’s genre diversity, visual fidelity, and two narrative tracks.
- E User Study Details: 15 professional filmmakers with at least three years of industry experience blindly evaluated 200 randomly sampled generated sequences.Participants were unaware of the generating model.
- E User Study Details: The evaluation covered temporal narrative coherence, visual continuity across cuts, motion naturalness, and overall subject consistency.
- E User Study Details: Experts used a strict 1–5 Likert rubric ranging from cinematic-grade continuity and identity preservation to complete multi-shot failure.The highest rating denotes professional-looking temporal continuity, identity preservation, and spatial transitions; the lowest denotes arbitrary subject mutation or slideshow-like degradation.
- F Extended Dataset Visualizations and Limitations: MuSS’s extended visualizations demonstrate broad cinematic genre coverage with complex lighting, varied spatial layouts, and dynamic subject motions.
- F.1 More Visualizations of the Dataset: The visualizations cover two core narrative tracks: complex cinematic narratives and subject-centric narratives.Four detailed visualizations are used to highlight these tracks.
- F.1 More Visualizations of the Dataset: Complex-narrative examples show montage transitions across establishing, over-the-shoulder, and multi-character shots, with keyframes, progressive captions, and raw screenshots.
- F.1 More Visualizations of the Dataset: Subject-centric examples pair a core reference subject with diverse multi-view target shots to illustrate identical identity across shots.
F.2 Limitations and Future Work · G Ethical Considerations
MuSS reduces copy-paste shortcuts but remains vulnerable to difficult visual conditions that disrupt subject extraction. Its consistent subject-driven generation also presents dual-use risks, motivating watermarking, logging, and provenance safeguards for deployment.
- F.2 Limitations and Future Work: MuSS curation and auditing identified 86 difficult cases, chiefly involving occlusion or crowds (31%), viewpoint changes or extreme close-ups (24%), low light or motion blur (22%), identity switches (19%), and formatting or tool errors (4%).These cases can perturb the zero-shot subject extraction pipeline.
- F.2 Limitations and Future Work: The progressive captioning pipeline targets precise spatiotemporal text alignment in complex cinematic narratives.Curated keyframes are paired with progressive multi-shot captions for Track 1.
- G Ethical Considerations: Robust Subject-to-Video generation poses risks of deepfake generation and deceptive synthesis.The authors advocate invisible watermarking such as SynthID, comprehensive generation logging, and robust provenance tracking for public deployment.
- G Ethical Considerations: MuSS is intended for academic research and uses curation and evaluation metrics that penalize superficial copying to encourage structurally grounded, physics-aware synthesis.The paper nevertheless acknowledges the dual-use nature of highly consistent subject-driven generation.
- G Ethical Considerations: Track 2 structures isolated reference subjects alongside dynamic target shots and captions that force novel-view synthesis rather than pixel copying.This design addresses the subject-to-video copy-paste shortcut through cross-shot variation.
- G Ethical Considerations: Raw cinematic data include montage transitions across diverse genres and subjects shown consistently across changing camera angles, lighting conditions, and backgrounds.These examples illustrate the structural and cross-shot consistency targeted by MuSS.