Source-linked AI summary
ContextMaster: Interactive Multi-Shot Video Creation via Fixed-Budget Sparse Context Routing
Xu Guo, Zhengxuan Wei, Xinghui Li, Hanzhuo Huang, Xinyu Liu, Xiangyang Luo, Min Wei, Yiran Zhu, Qiulin Wang, Yulong Xu, Xintao Wang, Pengfei Wan, Qi Fan, Xiangwang Hou
TL;DR
Interactive multi-shot video creation requires composing generation, reference conditioning, and editing over growing visual histories without increasing denoising-time context reads. ContextMaster addresses this with role-aware representations, fixed-budget sparse routing, and privileged context distillation, improving task fulfillment and cross-shot consistency while reaching 16 FPS on one GPU.
Problem
Interactive multi-shot video creation lacks a unified way to compose generation, reference conditioning, and editing over evolving histories while keeping denoising cost predictable.
Method
ContextMaster combines role-aware context representations, reusable clean states, fixed-budget sparse routing with ConstraintSink, and privileged context distillation for sparse few-step inference.
Results
ContextMaster improves task fulfillment and cross-shot consistency across three primitive tasks and compositional trajectories while reaching 16 FPS on a single GPU.
Takeaways & Limitations
The model supports flexibly composed multi-shot creation across generation, reference conditioning, editing, and continuation over persistent visual history.
Takeaways & Limitations
Throughput decreases approximately 0.4 FPS per additional shot because full accumulated history is still prefetched to construct the context cache.
Abstract
from arXiv · showhide
Recent video models increasingly support generation, reference conditioning, and editing within a single model, yet typically expose them as separate operations over fixed inputs. Practical creation unfolds across multiple shots, requiring one model to generate from text, follow a reference, or edit source footage while maintaining shared history. We formalize this setting as interactive multi-shot video creation (IMVC) and introduce ContextMaster, a unified model with a role-aware context representation for these operations. An interactive model must retain access to an expanding history without allowing the context read cost at each denoising step to grow. ContextMaster combines reusable clean context states with fixed budget sparse context routing and uses ConstraintSink to keep task constraints visible. To address the dual challenges of sparse context access and inference with few denoising steps, we propose a two-stage privileged context distillation framework, which transfers full context behavior from a dense teacher through consistency distillation and then refines deployment rollouts with distribution matching. Experiments on the three primitive tasks demonstrate improved task fulfillment and consistency across shots over specialized baselines. User studies further validate flexibly composed workflows, while the model reaches 16 FPS on a single GPU.
1 Introduction
ContextMaster formalizes interactive multi-shot video creation as stateful composition of generation, reference conditioning, and editing over evolving visual history. It addresses heterogeneous context, fixed-budget sparse access, and few-step inference while improving fulfillment and consistency at 16 FPS.
- Interactive multi-shot video creation: IMVC lets one model generate, reference, or edit shots in arbitrary order while updating shared visual history with accepted results.The setting supports workflows that interleave generation, editing, and continuation.
- Results: 16 FPS on a single GPU accompanies improved task fulfillment and cross-shot consistency across T2MV, R2MV, V2MV, and compositional X2MV workflows.The three primitive capabilities are evaluated quantitatively, while compositional multi-shot creation combines them over evolving visual history.
- Interactive multi-shot video creation: Role-aware rotary coordinates unify heterogeneous visual context while preserving multi-shot alignment.The context includes reference images, historical shots, source video, and noisy targets with distinct roles.
- Cacheable fixed-budget context: Cacheable fixed-budget context bounds active reads as history grows using reusable clean states, query-dependent block-sparse reads, and ConstraintSink.The design preserves explicit reference and source correspondences without discarding them.
- Privileged context distillation: A dense full-context teacher initializes a sparse few-step student through consistency distillation, followed by distribution matching on deployment-matched rollouts.This addresses both sparse context selection and few-step denoising while reducing errors accumulated in generated history.
2 Related Work
Prior work addresses unified and interactive video creation, heterogeneous visual context, and efficient few-step diffusion through complementary interfaces, positional schemes, sparsity methods, and distillation objectives. ContextMaster extends these directions toward a shared role-aware interface with growing history and fixed-budget context access.
- Unified and interactive video creation: Unified video creation spans holistic multi-shot synthesis with bidirectional cross-shot access and streaming next-shot generation conditioned on historical memory.Unified models and editors also support reference-conditioned generation and diverse video manipulations within shared models.
- Heterogeneous and growing visual context: Prior systems distinguish shot identities and in-context conditions with discontinuous, phase-shifted, or group-specific rotary coordinates.ContextMaster extends this direction with role-aware coordinates for reference, history, source, and target streams, while prior growing-history systems use sparse sampling, rolling windows, or fixed sinks.
- Sparse and few-step video diffusion: Sparse video diffusion exploits spatiotemporal patterns or content-relevant block retrieval, while BLADE combines adaptive block sparsity with sparsity-aware step distillation.Consistency and distribution matching objectives support few-step generation, and ShotStream uses self-forcing to reduce causal rollout errors.
3 Methodology
ContextMaster unifies interactive generation and editing over evolving shot histories with role-aware representations, cacheable fixed-budget context access, and privileged distillation. Its routing preserves explicit reference and source constraints while keeping denoising-step context-read cost independent of history length.
- Interactive interface: The framework processes each interaction using an instruction, optional reference image, optional source video, and finalized shot history through a shared generation-and-editing interface.Accepted candidates enter persistent history and are re-encoded for the next round, while rejected candidates can remain request-local sources for further editing.
- Role-aware representation: Role-aware encoding separates reference appearance, historical identity and narrative state, source editing structure, and target latents within one concatenated representation.Video VAE latents are concatenated by role, while role–shot RoPE offsets distinguish segments and shared frame coordinates align source and target frames during editing.
- Cacheable sparse context attention: The context branch is prefilled once per interaction round as a reusable KV cache, while context self-attention and target self-attention remain dense and only target-to-context reads are sparsified.Cached context states are independent of denoising step and CFG branch, enabling reuse throughout sampling.
- ConstraintSink and budgeted dynamic routing: ConstraintSink makes every reference block visible to each target block and preserves exactly aligned source–target blocks, with all retained blocks sharing the same context budget.Remaining capacity is routed adaptively using content-dependent retrieval, with separate source and history competition and work-conserving transfer when roles or candidates are missing.
- Budgeted dynamic routing: Each target query activates at most B blocks, or Bm context tokens, reducing dense target-to-context cost from O(NXNCD) to O(NXBmD) per layer independently of history length.The bound targets latency-critical reads repeated at every denoising step, while context states are encoded once per round and reused.
- Privileged consistency distillation: Privileged consistency distillation uses a frozen dense teacher with unrestricted context and a sparse student to transfer full-context behavior and consistency across adjacent noise levels.An exponential-moving-average target matches clean predictions at two endpoints, making sparse predictions invariant to a full-context teacher transition.
4 Experiments
Experiments evaluate ContextMaster across multi-shot generation, reference conditioning, editing, and interactive workflows using task-specific benchmarks, comparisons, user studies, and ablations. The model shows strong cross-shot performance, favorable human ratings, and benefits from role-aware routing and two-stage distillation.
- Benchmarks: Task-specific benchmarks cover T2MV, R2MV, V2MV, and interactive X2MV trajectories spanning generation, reference conditioning, editing, and composed operations.T2MV contains 100 prompts, R2MV 50 five-shot cases, V2MV 50 source-video cases, and X2MV 20 interactive trajectories.
- T2MV comparison: ContextMaster achieves the strongest overall T2MV performance, including an Inter-Shot improvement from 0.808 to 0.836, while remaining competitive on intra-shot consistency.The comparison includes MultiShotMaster, LongLive, ShotStream, and Infinity-RoPE.
- R2MV comparison: On R2MV, ContextMaster’s largest gains are in Inter-Shot and TF, reaches 16.74 FPS, and preserves reference appearance across changing views.Phantom retains higher DD and slightly higher intra-shot subject consistency, while qualitative results show less stable reference appearance across close-up and wider views.
- V2MV comparison: On V2MV, ContextMaster leads or ties on eight of nine reported metrics, although VideoCoF retains a small AQ advantage.The comparison uses the same source shots and instructions against VideoCoF, LucyEdit, StreamEdit, and LiveEdit.
- User study: User studies find ContextMaster highest in IF and CC across the three primitive tasks, with favorable scores maintained across mixed X2MV creation trajectories.Ten video professionals rated VQ, IF, TC, and CC on a 1–5 scale against the strongest baseline for each primitive task.
- Ablations: Ablations show that role-aware RoPE and mandatory ConstraintSink routing benefit task performance, while PCD primarily improves Inter-Shot and TF and DMD contributes more strongly to appearance quality.The architecture ablation fixes the active-read budget at 6 FE; the training ablation compares Vanilla sparse, PCD only, DMD only, and sequential training.
5 Conclusion · Appendix
ContextMaster formalizes interactive multi-shot video creation as stateful composition over persistent visual history and introduces role-aware, fixed-budget sparse context mechanisms. Its design preserves heterogeneous visual roles and explicit correspondences while bounding active context reads.
- 5 Conclusion: ContextMaster formalizes interactive multi-shot video creation as stateful composition over persistent visual history.The composition includes generation, reference conditioning, editing, and continuation.
- 5 Conclusion: The framework unifies generation, reference conditioning, editing, and continuation within one persistent visual history.
- 5 Conclusion: Role-aware rotary coordinates distinguish heterogeneous visual roles in the shared context representation.
- 5 Conclusion: Asymmetric clean-context prefill and fixed-budget sparse routing bound each target query’s active reads.
- 5 Conclusion: ConstraintSink preserves explicit reference and source correspondences within the same context budget.
- 5 Conclusion: Privileged context distillation trains the sparse few-step model by transferring full-context teacher behavior.The supplied passage introduces this training approach but truncates its description after “through cons”.
A Interactive Director · B Implementation Details
The Interactive Director is a tool-augmented VLM agent that converts concise requests into ordered multi-shot scripts and prepares operation-specific conditions. It dispatches these instructions through ContextMaster’s interfaces and updates shared visual history with accepted outputs for subsequent planning.
- A Interactive Director: The Interactive Director supports interactive multi-shot creation by coordinating generation, reference conditioning, and editing within one evolving project.It is introduced as a tool-augmented VLM agent inspired by Aurora (Yu et al., 2026).
- A Interactive Director: Given a concise request, the Director expands it into an ordered multi-shot script.The script specifies the sequence of operations for the requested creation workflow.
- A Interactive Director: The Director prepares the conditions required by each operation before execution.These conditions support generation, reference conditioning, and editing tasks.
- A Interactive Director: It can retrieve missing reference images from the Internet through the Serper API.This retrieval capability supplies absent references needed by the planned workflow.
- A Interactive Director: Instructions, references, and source segments are dispatched to ContextMaster through its generation, reference, and editing interfaces.The same interfaces are used for the primitive tasks.
- A Interactive Director: Accepted outputs are committed to the shared visual history, allowing subsequent operations to use the updated project state.This creates an iterative planning loop across turns.
B.1 Training Configuration · B.2 Training Data
Training uses two-stage privileged context distillation, beginning with a dense full-context teacher and refining a fixed-budget sparse student for four-step deployment. The data pipeline combines public single-shot datasets with approximately one million internal multi-shot videos spanning T2MV, R2MV, and V2MV.
- B.1 Training Configuration: The dense full-context teacher is trained with AdamW at a learning rate of 1 × 10^-5, 200 warmup steps, 0.01 weight decay, and mixed precision.PCD initializes both the sparse student and its exponential-moving-average target from this teacher.
- B.1 Training Configuration: PCD uses a 2 × 10^-6 learning rate, 0.99 EMA decay, 50-level teacher noise discretization, and the deployment’s same four-step schedule.The teacher uses classifier-free guidance at scale 6.0, absorbed into the student for single-branch inference.
- B.1 Training Configuration: DMD initializes its generator from the PCD student, pairing a dense-context guided real-score model with fixed-budget-context generator and fake-score models.The generator and fake-score learning rates are 2 × 10^-6 and 4 × 10^-7, respectively, with five fake-score updates per generator update.
- B.1 Training Configuration: 64 NVIDIA H200 GPUs are used for all experiments.This hardware configuration supports the reported training experiments.
- B.1 Training Configuration: Interactive trajectories are evaluated from references, ordered instructions, and resulting videos using Visual Quality, Instruction Following, Temporal Consistency, and Cross-Shot Consistency.These metrics assess perceptual quality, instruction completion, within-shot stability, and preservation of identities, attributes, and accepted history.
- B.2 Training Data: Teacher training starts from public HuMo, Kiwi-Edit, and Ditto-1M data to establish single-shot reference-conditioned generation and video-editing capabilities.The passage names the cited datasets and their initial role in teacher training.
- B.2 Training Data: Approximately one million internal multi-shot videos are used to construct training examples for T2MV, R2MV, and V2MV.For R2MV, identity-matched cross-pair reference images are retrieved according to ID similarity.
C Additional Ablation Studies · D Evaluation Details · D.1 User Study
The ablation identifies 6 FE as the quality–efficiency operating point: increasing the active context budget from 2 to 6 FE substantially improves key metrics, while 8 FE offers only marginal gains and reduces throughput. Evaluation details also include a human-rating interface covering four criteria.
- C Additional Ablation Studies: The active-budget ablation keeps the checkpoint fixed across configurations.This controls the comparison while assessing the deployed 6-FE setting.
- D Evaluation Details: The active-budget ablation uses a common evaluation protocol across configurations.The protocol is applied to five-shot T2MV sequences.
- C Additional Ablation Studies: 6 FE is selected as the quality–efficiency operating point for active context routing.The ablation uses the same checkpoint and evaluation protocol on five-shot T2MV sequences.
- C Additional Ablation Studies: Increasing the active context budget from 2 to 6 FE substantially improves AQ, Inter-Shot consistency, and TF.The comparison is conducted on five-shot T2MV sequences under a shared checkpoint and evaluation protocol.
- C Additional Ablation Studies: Expanding the budget from 6 to 8 FE yields only marginal gains while reducing throughput by 0.56 FPS.This trade-off motivates choosing 6 FE rather than the larger budget.
- D.1 User Study: The user-study interface collects human ratings across four evaluation criteria.Figure 6 presents the interface used for collecting these ratings.
D.2 Gemini Task-Fulfillment Evaluation
Figure 7 presents the task-specific prompts used by Gemini 2.5 Pro to evaluate task fulfillment.
- D.2 Gemini Task-Fulfillment Evaluation: Figure 7 specifies task-specific prompts for Gemini 2.5 Pro’s task-fulfillment evaluation.The prompts are designed for evaluating whether tasks are fulfilled.
E Limitations
ContextMaster’s fixed-budget routing limits repeated target-to-context reads, but full-history bidirectional prefill still causes throughput to decline as shots accumulate.
- E Limitations: Throughput decreases by approximately 0.4 FPS per additional shot because the context branch prefills the full accumulated history to construct its KV cache.Future work will explore persistent per-shot caches and compact shot summaries.
F Additional Qualitative Results
Figures 8–10 present additional qualitative results for ContextMaster across text-to-multi-shot video, video-to-multi-shot video, reference-to-multi-shot video, and editing tasks.
- Additional qualitative results: Additional qualitative results cover T2MV, V2MV, R2MV, and X2MV.The results are shown in Figures 8–10.
- T2MV: Figure 8 shows additional qualitative results for T2MV using a four-shot chef-and-bread-roll sequence.The sequence depicts kneading dough, close-up hand movements, bread rolls baking, and the chef smiling at the camera.
- V2MV: Figure 9 shows additional qualitative results for V2MV on replacing an old man with a red headscarf by an old woman with a blue headscarf.The instruction specifies the identity and headscarf change in the video.
- R2MV and X2MV: Figure 10 shows additional qualitative results for R2MV and X2MV, including a three-shot young-man work sequence.The sequence shows laptop work beside papers, adjusting round glasses, and stretching after working.