Source-linked AI summary
Bridging Your Imagination with Audio-Video Generation via a Unified Director
Jiaxu Zhang, Tianshu Hu, Yuan Zhang, Zenan Li, Linjie Luo, Guosheng Lin, Xin Chen
TL;DR
Existing video-generation systems often separate script drafting from keyframe design, limiting narrative logic and visual consistency in multi-shot work. UniMAGE unifies text and image generation with interleaved and disentangled training, and demonstrates stronger coherence and consistency than existing agent-based and unified models.
Problem
Many AI video systems treat script drafting and keyframe generation as separate tasks, limiting narrative logic and visual consistency in multi-shot video creation.
Source: §1 · p. 1Method
UniMAGE combines text and image generation in a unified director model, using interleaved concept learning and disentangled expert learning for scripts and keyframes.
Source: §1 · pp. 2–3Results
UniMAGE demonstrates stronger narrative coherence, character stability, and image consistency in long-form, multi-scene script creation than existing agent-based and unified models.
Source: §1 · p. 3Takeaways & Limitations
UniMAGE generates long-context scripts and keyframes that can guide existing audio–video generation models.
Source: §3 · p. 4, §5 · p. 12UniMAGE does not yet fully address emotional pacing, stylistic cinematography, or fine-grained control over directorial intent.
Source: §5 · p. 12
Abstract
from arXiv · showhide
Existing AI-driven video creation systems typically treat script drafting and key-shot design as two disjoint tasks: the former relies on large language models, while the latter depends on image generation models. We argue that these two tasks should be unified within a single framework, as logical reasoning and imaginative thinking are both fundamental qualities of a film director. In this work, we propose UniMAGE, a unified director model that bridges user prompts with well-structured scripts, thereby empowering non-experts to produce long-context, multi-shot films by leveraging existing audio-video generation models. To achieve this, we employ the Mixture-of-Transformers architecture that unifies text and image generation. To further enhance narrative logic and keyframe consistency, we introduce a ``first interleaving, then disentangling'' training paradigm. Specifically, we first perform Interleaved Concept Learning, which utilizes interleaved text-image data to foster the model's deeper understanding and imaginative interpretation of scripts. We then conduct Disentangled Expert Learning, which decouples script writing from keyframe generation, enabling greater flexibility and creativity in storytelling. Extensive experiments demonstrate that UniMAGE achieves state-of-the-art performance among open-source models, generating logically coherent video scripts and visually consistent keyframe images.
1 Introduction
UniMAGE brings script drafting and keyframe generation into a unified director model to address limited narrative logic and visual consistency in multi-shot filmmaking. Its training combines text–image interleaving with separate script and keyframe expertise to support coherent long-context storytelling.
1 Introduction
- Existing systems often separate script drafting from keyframe generation, limiting narrative logic and visual consistency in multi-shot video creation.Source: §1 · p. 1
- UniMAGE is a unified director model that coordinates narrative logic and visual composition to bridge user intent with multimodal scripts.Source: §1 · pp. 2–3
- Built on Mixture-of-Transformers, UniMAGE integrates an LLM and diffusion model in one transformer framework for joint text–image reasoning and generation.The model must handle film scripts’ lengthy, context-rich sequences.Source: §1 · p. 2
- Interleaved Concept Learning jointly optimizes text and image generation from interleaved scripts and keyframes, while ID prompting marks keyframe indices and character identities.These mechanisms support narrative-concept understanding and stable multi-subject appearances across long sequences.Source: §1 · pp. 2–3
- Disentangled Expert Learning trains a dedicated script expert while retaining interleaved keyframe generation; Pre-Context Script Splitting teaches continuation from prior context or a user prompt.The strategies target coherent script logic and flexible continuation.Source: §1 · pp. 2–3
- UniMAGE demonstrates stronger narrative coherence, character stability, and image consistency in long-form, multi-scene script creation than existing agent-based and unified models.Source: §1 · p. 3
2 Related Work
Prior film-generation systems combine language-model planning and image synthesis, but many separate script creation from visual generation or handle only short contexts. UniMAGE uses a unified multimodal framework for long-form narrative and visual generation.
2 Related Work
- Earlier film-generation work includes LLM-based script expansion, agent-based filmmaking workflows, and diffusion methods for character consistency across image sequences.Examples include Anim-Director, FilmAgent [33], MovieAgent [30], and StoryDiffusion.Source: §2 · p. 3
- Many film-generation systems decouple script generation and visual synthesis, contributing to weak narrative–visual alignment and limited long-range coherence; UniMAGE instead learns both in one framework.Source: §2 · p. 3
- Unified multimodal models have explored autoregressive text–image tokens and shared-transformer diffusion, while Bagel’s Mixture-of-Transformers enables modality-specific parameters with shared self-attention.The related work notes that existing unified models mainly handle short contexts without mechanisms for long-sequence narrative or visual coherence.Source: §2 · p. 3
- Audio–video systems have improved cross-modal alignment and generalization, yet current methods lack planning and coherence across multi-shot narrative sequences.Source: §2 · pp. 3–4
3 Method
UniMAGE turns user prompts into structured illustrated scripts by unifying text understanding and image generation in a Mixture-of-Transformers model. Its training and inference strategies target coherent long-form scripts, consistent keyframes, and continuation.
3 Method
- UniMAGE transforms user prompts into scripts containing global descriptions, shot-level content, and keyframe images, using a Mixture-of-Transformers model with separate understanding and image-generation experts.The experts process a shared token sequence; text prediction uses next-token prediction and visual prediction uses Rectified Flow.Source: §3 · p. 4
3.1 Preliminaries
- The model’s text expert uses next-token prediction, while its image-generation expert uses Rectified Flow for visual token prediction.The next-token objective conditions each prediction on preceding tokens; Rectified Flow transports samples along learned straight trajectories.Source: §3 · p. 4, §3.1 · p. 4
3.2 Script Structure
- The script comprises global character and environment descriptions, content descriptions, and keyframe images, with frame descriptions for static layouts and video descriptions for temporal and narrative elements.Special tokens identify script elements and subject appearances; indicator symbols mark dialogue and environmental sound effects.Source: §3.2 · p. 5, Figure 2 · p. 5
3.3 Interleaved Concept Learning
- Interleaved Concept Learning jointly optimizes both experts on interleaved script text and keyframes, while In-Context ID Prompting links image tokens to frame, character, and environment IDs for visual consistency.The interleaved stage is intended to help the model understand lengthy, context-rich scripts; identity prompting addresses consistency across multiple characters and scenes.Source: §3.3 · pp. 5–6
3.4 Disentangled Expert Learning
- Disentangled Expert Learning trains the understanding expert on pure-text scripts while optimizing the generation expert on text–image data with the understanding branch frozen.The approach also incorporates text–image pairs to improve visual fidelity.Source: §3.4 · p. 6
- Pre-Context Script Splitting trains prompt-based extension and in-context continuation by inserting prompts at split points in complete scripts.Extension uses a new user prompt; continuation uses a system prompt before the script’s last shot.Source: §3.4 · pp. 6–7
3.5 Inference of UniMAGE
- At inference, UniMAGE first generates a multi-shot text script, supports prompted extension or continuation, then generates keyframes for the completed script.The text and image generation processes are separated within one model.Source: §3.5 · p. 7
4 Experiments
Experiments assess UniMAGE through qualitative script comparisons, ViStoryBench scores, ablations, and a user study; results report advantages in consistency, alignment, and long-form storytelling.
4 Experiments
- The training dataset combines multi-shot text–image scripts, long-form text scripts, and single-shot text–image pairs; evaluation includes qualitative scenarios and six ViStoryBench metrics.The dataset subsets support interleaved concept learning, disentangled expert learning, and generation-expert training, respectively.Source: §4 · pp. 7–8
4.1 Qualitative Results
- Across multiple shots, UniMAGE maintains character identities and visual coherence, while baseline appearances vary in facial structure, hairstyle, or clothing across scenes and angles.The comparison covers multi-character scripts and character traits across different environments and shot configurations.Source: Figure 5 · p. 8, §4.1 · p. 9
- In long-form sequences, UniMAGE follows plot progression with consistent style and character appearance, while baselines show repetitive visuals, unstable identities, or fragmented transitions.The comparison reports improved coherence, scene diversity, and cinematic storytelling for UniMAGE.Source: Figure 6 · p. 9, §4.1 · p. 9
- Against SEED-Story, UniMAGE produces more coherent plots and higher-quality keyframes, with generalization across genres, visual styles, and storytelling formats.The passage attributes SEED-Story's weak generalization to training on only three animation datasets.Source: §4.1 · p. 9, Figure 7 · p. 10
- Without In-Context ID Prompting, identity cues become ambiguous during scene transitions or long spans, producing mixed and inconsistent character depictions.Figure 8 marks failure cases without the strategy; the passage describes explicit ID conditioning as crucial for multi-character consistency.Source: Figure 8 · p. 10, §4.1 · p. 10
- Without Pre-Context Script Splitting, interleaved-only training often produces repetitive descriptions and weaker narrative flow during script extension and continuation.The strategy trains the understanding branch to continue narratives from partial context and follow existing plot structures.Source: Figure 8 · p. 10, §4.1 · p. 11
4.2 Quantitative Results
- On ViStoryBench, UniMAGE achieves the highest CIDS (59.2) and OCCM (88.07), plus Alignment (80.8), with balanced performance across consistency, alignment, and visual quality.Some baselines lead isolated metrics, but the passage reports no balanced alternative across these dimensions.Source: Table 1 · p. 11, §4.2 · p. 11
- In a study of 50 volunteers ranking 40 scripts, UniMAGE received the highest preference across criteria, including a GSB score of 0.72 for narrative logic.Figure 9 summarizes the aggregated rankings; lower average rankings correspond to higher quality.Source: §4.2 · p. 11, Figure 9 · p. 11
5 Conclusions
UniMAGE unifies script drafting and keyframe generation in a single framework for long-context, multi-shot narratives. The paper reports strong capabilities and state-of-the-art results among open-source systems, while noting that several higher-level filmmaking dimensions remain unaddressed.
5 Conclusions
- UniMAGE integrates script drafting and keyframe generation, using Mixture-of-Transformers and two training paradigms to support long-context narratives with logical and visual coherence.Interleaved Concept Learning uses text–image interleaving; Disentangled Expert Learning decouples script and keyframe generation.Source: §5 · p. 12
- UniMAGE achieves state-of-the-art results among open-source systems and maintains character identity, storyline continuity, and visual alignment across extended sequences.The conclusion also notes support from In-Context ID Prompting and Pre-Context Script Splitting.Source: §5 · p. 12
- Emotional pacing, stylistic cinematography, and fine-grained control over directorial intent are not yet fully addressed.The paper identifies richer cinematic understanding and more expressive narrative control as future directions.Source: §5 · p. 12
- The work is solely for research purposes; the described technology is not incorporated into ByteDance products, and the figures, scripts, and human faces are AI-generated.Source: §5 · p. 12
A Mixture-of-Transformers Architecture
The architecture combines separate understanding and generation experts within a shared multimodal Transformer framework. ViT and VAE handle visual input and generation, while modality-interleaved tokens use generalized causal attention.
A Mixture-of-Transformers Architecture
- Text, ViT, and VAE tokens are interleaved according to the multimodal input structure and divided into sequential modality-specific splits.Source: A Mixture-of-Transformers Architecture · p. 16
- Each modality split can attend to earlier splits, with causal attention for text tokens and bidirectional attention for vision tokens.Source: A Mixture-of-Transformers Architecture · p. 16
- MoT uses two Transformer experts for understanding and generation, with all tokens processed through shared multimodal self-attention in each block.The transformer parameters are initialized from Qwen 2.5, and each attention block includes QK-Norm for training stability.Source: Figure 10 · p. 16, A Mixture-of-Transformers Architecture · p. 16
- ViT handles visual understanding and captures semantic content, while VAE handles image generation and low-level information.The architecture uses SigLIP2-so400m/14 for ViT initialization and a pre-trained FLUX VAE for generation.Source: Figure 10 · p. 16, A Mixture-of-Transformers Architecture · p. 16
B Supplementary on Script Dataset
The supplementary describes a text–image interleaved script dataset built from filtered multi-shot videos, along with prompt styles and structured annotation instructions. The annotations cover visual and audio content in a coordinated format.
B Supplementary on Script Dataset
- The dataset pipeline curates approximately 800k multi-shot video clips, captions them with Gemini 2.5 Pro, selects a keyframe from each clip, and retains approximately 450k clips after quality filtering.Q-Align assesses keyframe image quality and aesthetics.Source: B Supplementary on Script Dataset · pp. 16–17
- User prompts are generated in four styles: simple narrative, abstract concept, phrase splicing, and spoken expression.Source: B Supplementary on Script Dataset · pp. 17–18
- Figure 11 provides an example of the text–image interleaved script dataset.Source: B Supplementary on Script Dataset · p. 17
- The annotation prompt requests JSON with first-frame visual descriptions and video–audio descriptions for sorted shots.The requested descriptions include camera angle, visual style, characters, environment, objects, and audio content.Source: B Supplementary on Script Dataset · p. 17
- The instructions require video and audio to be described simultaneously in chronological order and spoken content to remain in its original language.Source: B Supplementary on Script Dataset · p. 18
C Supplementary on Experiments
The supplementary provides complete versions of the paper’s generated multimodal results and additional generated scripts. These materials are included because of main-text page limits and to further demonstrate generalizability.
C Supplementary on Experiments
- Main-text page limits prevent displaying the complete generated multimodal scripts.Source: C Supplementary on Experiments · p. 18
- The supplementary provides full versions of all results presented in the main paper.Source: C Supplementary on Experiments · p. 18
- Additional generated scripts are included to further demonstrate UniMAGE’s generalizability.Source: C Supplementary on Experiments · p. 18
D Supplementary on Audio-Video Generation
This supplement presents sample multi-shot audio-video stories across several narrative prompts, pairing frame or keyframe descriptions with shot-by-shot action, dialogue, and sound cues. It includes two demonstrations generated using Veo3 I2V and illustrates consistency limitations in current audio-video models.
Supplementary demonstrations
- The two supplementary videos use Veo3 I2V with scripts, character lines, and sound-effect descriptions provided by UniMAGE.The paper names the files UniMAGE_Demo_1.mp4 and UniMAGE_Demo_2.mp4.Source: D Supplementary on Audio-Video… · p. 18
Limitations
- Current audio-video models may vary characters’ timbre and facial features in long multi-camera videos, an issue left for future work.The paper identifies these as limitations of current audio-video generation models.Source: D Supplementary on Audio-Video… · p. 18
Multi-shot example
- The archaeologist story develops across shots from discovering an artifact to revealing a celestial map and awakening stone guardians.The sequence includes dialogue, camera directions, and sound or music cues; the tablet’s symbols form a cosmic map before guardians rise from the chamber floor.Source: D Supplementary on Audio-Video… · p. 21
D Supplementary on Audio-Video Generation
- A boy’s magic spell summons a playful ghost, whose meeting with him ends as it opens a shimmering doorway through the curtains.The sequence moves from surprise and tentative contact to a glowing threshold that hints at an adventure.Source: D Supplementary on Audio-Video… · p. 26
- A teacher encourages two shy students to draw, then uses an art book to invite them to make something together.The classroom sequence follows each child’s hesitant or quiet engagement before the teacher brings them together around a shared creative activity.Source: D Supplementary on Audio-Video… · p. 27
- At an international market, a Qatari diplomat, Japanese martial arts master, and American tourist discover a cryptic note linked to a regional guild mark and hidden passage.Separate observations—a misplaced trinket and a note found in an ornate box—bring the three characters together to investigate.Source: D Supplementary on Audio-Video… · p. 28
- At an auction, a critic outbids a suspicious rival for “Midnight Bloom,” and their tense exchange leaves the victory ambiguous.The sequence then extends into a hidden-passage mystery in which the painting is revealed as a map and the pair flee approaching pursuers.Source: D Supplementary on Audio-Video… · pp. 29–30
- The auction story continues with the critic and rival entering a concealed study, learning the painting is a map, and fleeing when someone finds them.The critic’s flight ends with a newly determined resolve to protect the secret.Source: D Supplementary on Audio-Video… · pp. 30–31
- A historian’s public lecture moves from ancient ingenuity and written language to shared human challenges and the continuing legacy of past civilizations.Her address uses examples including pyramids, Mesopotamian waterwheels, and early writing before concluding that ancient legacies echo in the modern world.Source: D Supplementary on Audio-Video… · p. 32
Audio-Video Generation Examples
- The biologist discovers a rare frog, uncovers tampering in its genetic data, and works with a colleague to identify the pathogen and trace its source.The shots move from rainforest discovery to lab investigation and a joint resolution.Source: D Supplementary on Audio-Video… · pp. 33–34
- A professor and student find an ancient pot whose symbols match old texts, present their findings, and then see the artifact begin to glow.The example shifts from an archaeological dig to a university lecture hall before ending on the anomalous artifact.Source: D Supplementary on Audio-Video… · pp. 34–35
- A street performer’s juggling and magic catch a scholar’s attention, turning her focused walk into a shared moment of amusement.The sequence follows her from initial distraction to watching, smiling, and laughing.Source: D Supplementary on Audio-Video… · p. 36
- A mother and son enter a fantasy world through a living storybook, explore a magical forest, and approach a glowing doorway that opens onto their own story.The shots progress from reading together through the portal journey and a tender exchange in the forest.Source: D Supplementary on Audio-Video… · p. 37
- An old fisherman’s walk home ends with his sack spilling old coins and a carved wooden box, which he opens to reveal a golden glow.A continuation prompt leads to a close-up of his tearful, joyful reaction to the discovery.Source: D Supplementary on Audio-Video… · pp. 38–39
- A final prompt introduces a worried young farmer whose cow is sick and shows him standing somberly in a field at sunset.The supplied example begins the farm scenario but does not yet show the veterinarian treating the cow.Source: D Supplementary on Audio-Video… · p. 40
Calf and veterinarian
- A sick calf’s owner seeks help, and a veterinarian examines the animal with reassuring dialogue and hopeful music.The sequence progresses from the calf’s illness and the owner’s worry to veterinary care.Source: D Supplementary on Audio-Video… · p. 40
Architect and street artist
- An architect confronts a street artist over unauthorized painting, then collaborates with him on a mural respecting the historic building.The music shifts from clashing themes to a collaborative tone as their disagreement gives way to shared design work.Source: D Supplementary on Audio-Video… · pp. 41–42
Fisherman
- A fisherman patiently casts and retrieves his net, celebrates a generous catch, then settles into a peaceful close.The sequence includes dawn scenery, the sounds of waves and a net, and music that shifts from hopeful to celebratory.Source: D Supplementary on Audio-Video… · p. 43
Photographer and ranger
- A nature photographer shares a rare bird sighting with a park ranger, who checks a bird guide before they agree to protect it.The scenes move from forest observation to identification in the park office and a shared conservation resolve.Source: D Supplementary on Audio-Video… · p. 44
Young musician
- A young musician discovers an old song in an attic, performs it at a town festival, and receives an emotional response from the crowd.The sequence accompanies his guitar practice and performance with evolving music, applause, and cheers.Source: D Supplementary on Audio-Video… · p. 45