PaperlayerView exact revision on arXiv

Source-linked AI summary

UnityShots: Memory-Driven Multi-Shot Audio-Video Generation with Boundary-Aware Gating

Jiehui Huang, Yuechen Zhang, Bin Xia, Jiahao Wang, Xu He, Zhenchao Tang, Meng Chu, Xin Tao, Pengfei Wan, Jiaya Jia

arXiv:2606.21661v1cs.CV

TL;DR

Multi-shot audio-video generation needs cross-shot memory to preserve subjects, scenes, and speaker identity across cuts. UnityShots uses fixed video memories and reference-speaker anchoring, leading open-source baselines on cross-shot coherence across I2V, T2V, and R2V while matching the strongest closed-source system on multi-shot axes.

  • Problem

    Existing multi-shot generators lack scalable, structured mechanisms for preserving subject appearance, scene context, and speaker identity across independently generated shot boundaries.

  • Method

    UnityShots combines long- and short-term fixed video memory slots, boundary-conditioned gating, per-shot speaker anchors, and a cut-type prior across I2V, T2V, and R2V generation.

  • Results

    UnityShots leads open-source baselines on every cross-shot coherence metric across I2V, T2V, and R2V, and matches the strongest closed-source system on multi-shot axes.

  • Takeaways & Limitations

    The system provides a unified multi-shot generation approach that maintains cross-shot coherence across conditioning modes while exposing transition strength as an inference-time control.

  • Takeaways & Limitations

    The authors report occasional subtitle instability across shots and visual degradation in scenes with multiple simultaneous speakers or compositionally complex layouts.

Abstract

from arXiv · show

Generating a coherent multi-shot video requires structured cross-shot memory. Subject appearance, scene context, and speaker identity must persist across cuts. Existing approaches either train end-to-end over fixed-length sequences and cannot scale, generate shot-by-shot with memory banks that grow linearly, or orchestrate pretrained generators under an LLM planner without a multi-shot-aware backbone. We present UnityShots, a memory-driven multi-shot audio-video generation system built on LTX-2.3, trained on annotated cinematic and music-video shots. The video stream maintains two fixed-size slots, a long-term memory (LTM) slot anchored to the opening shot and a short-term memory (STM) slot holding the immediately preceding tail, both updated at every cut by a boundary-conditioned gate that fuses visual cut probability and beat-tracker signals. The audio stream injects a reference speaker token at every shot to preserve vocal timbre without a sliding audio bank. A discrete cut-type prior, learned through AdaLN, becomes an inference-time control knob over transition strength. We release a benchmark of $200$ multi-cultural multi-shot sequences spanning six ethnic regions and ten or more languages, with per-shot reference identities, reference audio, and per-boundary transition labels. Evaluated across I2V, T2V, and R2V conditioning modes, UnityShots leads open-source baselines on every cross-shot coherence metric and matches the strongest closed-source system on the multi-shot axes.

1. Introduction

UnityShots addresses the need for explicit cross-shot memory by separating persistent narrative identity from recent continuity and updating both through boundary-aware audio-visual signals. Built on LTX-2.3, the system supports I2V, T2V, and R2V control and enables composable per-shot generation over arbitrarily long sequences.

  • Motivation: The design targets coherent persistence of subject appearance, scene context, and speaker identity across shot boundaries.It explicitly combines opening-shot identity for narrative anchoring with recent motion and scene state for local continuity.
  • Related work: Earlier multi-shot approaches have structural limitations, including sequence-length and cost constraints when entire clips are processed as one denoising pass.The introduction identifies end-to-end, shot-by-shot, and other existing families as unable to fully provide scalable multi-shot generation.
  • Contribution: UnityShots uses two fixed-size video memory slots: a long-term slot anchored to the opening shot and a short-term slot containing the immediately preceding tail.Both slots are updated at every shot boundary by a gate that fuses visual cut probability with beat-tracker signals.
  • Contribution: The model is trained for image-to-video (I2V), text-to-video (T2V), and reference-identity-to-video (R2V) conditioning.These modes allow steering with a visual first frame, a text prompt alone, or a reference portrait.
  • Contribution: A companion agent system exposes per-shot generation as composable tools that an external planner can invoke across sequences of arbitrary length.This supports both manual and agent-driven production pipelines.

2. Releated Work

Prior work extends video generation to longer clips, but typically lacks discrete shot-boundary modeling and persistent cross-shot audio-visual memory. Existing alternatives trade scalability, controllability, or multimodal consistency against sequence-length limits, while agent pipelines inherit coherence limits from their underlying generators.

  • Video generation: Long-clip video diffusion models [4] [16] [34] [49] [55] [21] [32] [41] do not model shot boundaries as discrete events.These systems include open-source and large-scale models targeting minute-scale generation.
  • Video generation: End-to-end methods [12] [29] [43] [46] [47] produce coherent sequences but are limited by GPU memory and training context, whereas shot-by-shot generation lifts the length cap.Cinematic design methods [42] [48] add 3D-aware scene composition and per-shot camera specification.
  • Persistent memory: Persistent-memory research uses small fixed-size token sets as stable global attractors, while UnityShots uses recurrent O(1)-cost memory updated by joint audio-visual boundary signals rather than retrieval.The design covers both video and audio streams; Figure 2 compares it visually with prior designs.
  • Audio-video generation: Dual-stream architectures [11] [13] improve multimodal alignment, but DreamID-Omni [11] targets single shots without cross-shot memory and existing methods lack a persistent cross-shot speaker anchor.Foley-Crafter synthesizes video-conditioned ambient audio from text-to-audio models [25].
  • Agent-orchestrated pipelines: Agent-orchestrated pipelines [23] [24] [56] offer practical LLM-planned workflows, but their pretrained generators were not trained for cross-shot coherence or require closed-source paid APIs.This limits clean propagation of multi-shot signals through the planner.

3. Method

UnityShots extends LTX-2.3 with fixed long- and short-term video memory, boundary-aware gating, and per-shot speaker anchoring to preserve appearance, scene continuity, and vocal timbre across cuts. Strata-RoPE separates memory from current-shot tokens, while transition priors and multimodal boundary signals control memory reset strength.

  • Overview: UnityShots generates synchronized multi-shot audio-video clips from per-shot prompts, a reference image, and reference audio while conditioning each shot on a structured summary of preceding shots.The system uses LTX-2.3 as a 22B dual-stream diffusion-transformer backbone and supports I2V generation with the previous shot’s tail supplying memory.
  • Video memory: The video memory bank maintains fixed-size LTM and STM slots, anchoring identity to the opening shot while retaining the immediately preceding shot’s tail for local continuity.The LTM contains two latent frames, and each new shot receives the previous shot’s Pv-frame tail through the STM pathway.
  • Audio reference anchor: A fixed reference speaker token is injected at every shot to preserve character vocal timbre without maintaining a sliding audio memory bank.Background music is treated separately because it evolves continuously and is not tied to visual edit boundaries.
  • Boundary gating: Boundary-aware gates combine visual-cut probability, RMS energy change, and beat position, while a discrete transition prior controls how strongly each cut resets memory.The CONTINUE prior prevents over-resetting the LTM anchor after a visual-detector misfire; monotone gates keep the LTM contribution nonzero and allow STM weight to approach full strength at hard cuts.
  • Token layout and training: Strata-RoPE assigns LTM, STM, and current-shot video tokens to separated temporal strata so rotary attention attenuates cross-stratum interaction without extra depth, while training unrolls memory recurrence over 3–9-shot chunks.Stage 2 mixes I2V (p = 0.5), T2V (p = 0.3), and R2V (p = 0.2) conditioning, and the R2V variant replaces per-shot first-frame conditioning with a concatenated reference-identity token.

4. Experiment

UnityShots outperforms open- and closed-source baselines across conditioning modes on multi-shot coherence, with the largest gains emerging in longer sequences. Its dual-tier memory is complementary, though memory augmentation trades off cultural-authenticity ratings in I2V ablations.

  • Evaluation setup: The benchmark contains 200 multi-shot sequences spanning six cultural groups and ten or more languages, with reference identities, audio, and boundary labels.Sequences contain 3–9 shots and are evaluated across I2V, T2V, and R2V conditioning modes using video, audio, and multi-shot coherence metrics.
  • Sequence-length analysis: On sequences of six or more shots, UnityShots avoids identity drift by combining persistent LTM opening-shot context with STM immediate-boundary continuity.The monotone LTM gate preserves a non-zero opening-shot contribution, and UnityShots reaches perfect NC, Story, and Char ratings versus MOVA’s 4.71 NC and 4.50 Story.
  • Benchmark results: UnityShots leads open-source baselines on every I2V video and coherence metric, exceeding the strongest baseline by +0.40 NC and +0.69 Story.Against Kling, it leads NC by +0.13, Story by +0.10, and AES-A by +0.47 despite lower per-frame fidelity.
  • Benchmark results: UnityShots(T2V) beats HoloCine [29] on every shared metric, while UnityShots(R2V) beats DreamID-Omni [11] on 8 of 9 metrics.The T2V gains are +0.62 NC and +0.61 Story; R2V gains include +0.52 NC, +0.48 Story, +0.54 Pace, +1.01 AES-A, and +0.008 CLAP.
  • Memory ablation: Table 3 shows that both memory slots are necessary: only the full system makes all four evaluated metrics exceed the no-memory backbone.Single-slot variants lower NC relative to no-memory, while improving Story and Char; in R2V, the full system gains +0.15 NC and +0.12 Char over the best single-slot variant.
  • Limitations: The no-memory I2V baseline leads on Cult, scoring 4.20 versus 3.27–3.33 for memory-augmented variants, indicating a cultural-authenticity trade-off.The passage attributes this to fixed identity tokens suppressing cultural-specific visual details inferred from the prompt.

5. Conclusion

UnityShots is a memory-driven multi-shot audio-video generation system with fixed-size, boundary-aware video memory and inference-time control over transition strength. Its current limitations include subtitle instability and visual degradation in complex multi-speaker scenes, motivating iterative per-shot refinement as future work.

  • Conclusion: UnityShots uses two fixed-size video memory slots, anchored to the opening shot and preceding tail, updated at each cut by a gate combining visual and audio boundary signals.A discrete cut-type prior learned through AdaLN provides explicit inference-time control over transition strength.
  • Conclusion: The system is built on a 22B dual-stream diffusion transformer for memory-driven multi-shot audio-video generation.
  • Limitations and future work: Current limitations are occasional subtitle instability across shots and visual degradation in scenes with multiple simultaneous speakers or compositionally complex layouts.These issues are expected to improve with greater base-model capacity and a more mature companion agent system.
  • Limitations and future work: Future work will use the agent system for iterative per-shot frame refinement after initial generation, toward a fully automated pipeline.

Boundary-Aware Gating · Appendix · A. Training and Architecture Details

The appendix specifies the training setup for Stage 2 multi-shot fine-tuning and notes that Stage 1 uses the same base model, optimiser, and learning rate on single-clip identity data.

  • A. Training and Architecture Details: Stage 2 multi-shot fine-tuning uses the key hyperparameters listed in Table 1.The passage identifies Table 1 as the source of these hyperparameters but does not provide their values.
  • A. Training and Architecture Details: Stage 1 identity foundation training uses the same base model, optimiser, and learning rate as Stage 2.Stage 1 trains on single-clip identity data.

B. Benchmark Summary

The benchmark comprises 200 culturally diverse multi-shot sequences spanning six ethnic regions and at least ten languages, with structured identity, audio, and shot-level annotations.

  • B. Benchmark Summary: The benchmark contains 200 multi-cultural multi-shot sequences with 3–9 shots each, covering six ethnic regions and ten or more languages.Cultural quotas allocate 25% each to East Asian and Western sequences, 15% each to South Asian and African, and 10% each to Latin and Middle Eastern sequences.
  • B. Benchmark Summary: Each sequence includes reference identity images, 6–12 s reference audio with content distinct from shot dialogue, and per-shot first frames conditioned on character portraits.

C. Training Data and Benchmark Construction

This section describes seven-stage construction of the multi-shot training corpus and five-stage construction of a 200-sequence, culturally balanced evaluation benchmark. The pipelines annotate shot boundaries, subtitles, narratives, character identity, speaker links, and quality-related metadata for training and evaluation.

  • Training data construction: The training corpus is built through a seven-stage pipeline covering shot segmentation, annotation, character anchoring, record merging, and quality filtering.TransNetv2 supplies per-boundary visual cut probabilities and inferred CONTINUE, DISSOLVE, and HARD transition types for the cut-type prior; later stages consolidate per-shot outputs under hierarchical priority handling.
  • Training data construction: Character anchoring extracts identity-preserving reference frames and links WhisperX acoustic speaker labels to visual characters through temporal-overlap matching.The result is a per-shot speaker-to-character map for training records.
  • Benchmark construction: The benchmark contains 200 unique multi-shot story sequences balanced across six ethnic regions and ten or more languages.The regional allocation is 25% East Asian, 25% Western, 15% South Asian, 15% African, 10% Latin, and 10% Middle Eastern.
  • Benchmark construction: Benchmark stories include per-character descriptions and per-shot captions, with controlled coverage of dialogue-heavy and interview-format narratives to stress audio continuity.These stories are produced within a five-stage benchmark-construction pipeline.

D. Extended Comparison with Published Multi-Shot Methods

On a 200-sequence multi-cultural benchmark, UnityShots is compared with published T2V multi-shot methods under a shared protocol. Its dual-tier memory bank removes the trade-off between short-range conditioning and long-range coherence described for existing baselines.

  • Comparison with Published Multi-Shot Methods: Table 2 evaluates StoryMem, MultiShotMaster, HoloCine, and UnityShots in T2V mode on 200 multi-cultural sequences using the main paper’s evaluation protocol.StoryMem represents shot-by-shot methods, whereas MultiShotMaster and HoloCine represent end-to-end methods.
  • Comparison with Published Multi-Shot Methods: UnityShots eliminates the baselines’ coherence trade-off with its dual-tier memory bank, gaining +0.48 NC and +0.42 Story.StoryMem preserves TSIM through preceding-clip conditioning but accumulates identity drift beyond its conditioning window, while MultiShotMaster’s fixed context window misses long-range distinctions.
  • Comparison with Published Multi-Shot Methods: Among baselines, MultiShotMaster achieves the strongest TA, but trails on NC and TSIM because its fixed context window does not distinguish long-range from short-range context.Its stronger TA is attributed to larger-scale training.

E. Baseline Evaluation Protocol and Standard Benchmark Results · F. Pseudocode for Memory Update · G. Benchmark Prompt Examples

UnityShots is evaluated on a 200-sequence multi-cultural benchmark against open- and closed-source systems, outperforming baselines on key VBench coherence metrics. The accompanying pseudocode specifies boundary-aware memory updates and inference inputs, while benchmark prompts instantiate diverse opening-shot conditions.

  • E. Baseline Evaluation Protocol and Standard Benchmark Results: The evaluation compares MOVA [38], LTX-2 [13], ID-LoRA [7], OVI [26], and DreamID-Omni [11] using official checkpoints or official interfaces.Open-source systems use recommended settings and DDIM with 50 steps when applicable, while service-only systems receive identical benchmark prompts and reference inputs.
  • E. Baseline Evaluation Protocol and Standard Benchmark Results: UnityShots leads baselines by +0.006–+0.014 on VBench Subject Consistency, while also improving Background Consistency and Motion Smoothness on the 200-sequence benchmark.Table 3 scores videos with all shots concatenated at inferred boundaries; the LTM slot anchors opening-shot identity through every denoising pass.
  • F. Pseudocode for Memory Update: Algorithms 1 and 2 provide complete pseudocode for applying the boundary-aware gate and running the full UnityShots inference loop.The pseudocode covers both per-shot memory updates and end-to-end inference.
  • F. Pseudocode for Memory Update: The memory gate extracts the previous latent tail, computes boundary-conditioned STM and LTM coefficients, and applies content-aware refinement before scaling STM.The listed coefficients are g_stm = 0.3 + 0.7b and g_ltm = 0.1 + 0.6b, with refinement ρ obtained through a sigmoid over pooled latent features and the boundary embedding.
  • F. Pseudocode for Memory Update: The inference context concatenates the reference image, LTM, gated STM, reference audio, and current noisy video latent using Strata-RoPE bands.The pseudocode returns the denoised video latent after constructing this multimodal context.
  • G. Benchmark Prompt Examples: Benchmark prompt examples cover diverse opening shots featuring East Asian, South Asian, Latin, Western, and African subjects across office, desk, kitchen, exterior, garden, and café settings.Each sequence uses an opening caption with [subject] replaced by the reference identity image at inference.
  • G. Benchmark Prompt Examples: Algorithm 2 defines inference inputs as shot prompts, boundary strengths, a reference image, and reference audio for multi-shot generation.These inputs correspond to the benchmark’s shot-wise conditioning and boundary-control setup.

H. Per-Category Ablation Results · I. Companion Agent System

UnityShots’ ablations show that rule-based long-term gating and learned short-term refinement are complementary, while regional analyses reveal consistent coherence gains and identity-related cultural-detail trade-offs. Its companion agent exposes reference-controlled multi-shot generation through an open, cloud-only MCP workflow.

  • H. Per-Category Ablation Results: At 6+ shots, replacing the rule-based LTM gate with a learned gate reduces NC by 0.29 and Story by 0.50, destabilizing the opening-shot anchor.The comparison covers the full UnityShots design, learned LTM plus learned STM, and rule-based LTM plus rule-based STM across sequence-length brackets.
  • H. Per-Category Ablation Results: Replacing the learned STM gate with a fixed rule-based schedule hurts boundary smoothness across all lengths, showing that content-adaptive refinement benefits short timescales.The results support predictable rule-based behavior for the long-range anchor but learned refinement for short-term updates.
  • H. Per-Category Ablation Results: UnityShots leads MOVA on NC in every region, with the largest gaps in Middle Eastern and African sequences at +1.13 and +1.09.Table 5 reports regional NC and Cult comparisons on the 200-sequence benchmark; Story and Char follow the same regional pattern as NC.
  • H. Per-Category Ablation Results: MOVA reaches its highest NC on Western sequences at 4.15, where denser pretraining coverage reduces the benefit of additional memory.Greater scene diversity in Middle Eastern and African categories amplifies the persistent opening-shot anchor’s benefit.
  • H. Per-Category Ablation Results: The Cult gap between the no-memory backbone and UnityShots remains 0.81–0.96 across all six regions, indicating a systemic identity-conditioning trade-off rather than a regional artifact.The finding indicates that identity conditioning suppresses cultural visual detail at similar rates across ethnic backgrounds.
  • I. Companion Agent System: UnityShots provides a Claude-Code-compatible wrapper exposing per-shot reference editing as MCP tools for Claude Code, Claude Desktop, and VS Code clients.The wrapper installs as a Claude Code plugin and runs cloud-only without requiring a local GPU.
  • I. Companion Agent System: The planner-plus-worker architecture converts a story prompt into structured shot plans, synthesizes per-shot reference images, invokes UnityShots, and concatenates clips with FFmpeg.The planner is the conversational agent, while a vision-language model creates shot references before per-shot generation.
  • I. Companion Agent System: The MCP server exposes asynchronous story-to-multi-shot creation and fine-grained single-shot generation with optional reference images and audio.The released package includes MCP definitions, cloud connectors, example shot plans, a cloud-only Claude Code workflow, and a self-hostable full-stack implementation.
Loading 2606.21661v1…