Source-linked AI summary
Bernini: Latent Semantic Planning for Video Diffusion
Bernini Team, Chenchen Liu, Junyi Chen, Lei Li, Lu Chi, Mingzhen Sun, Zhuoying Li, Yi Fu, Ruoyu Guo, Yiheng Wu, Ge Bai, Zehuan Yuan
TL;DR
Combining MLLM understanding with diffusion synthesis for unified video generation and editing remains an open question. Bernini uses MLLM semantic planning in ViT embedding space to guide a diffusion renderer and achieves state-of-the-art results across diverse video benchmarks.
Problem
Effectively combining MLLM multimodal reasoning with diffusion-based photorealistic synthesis for unified understanding, generation, and editing remains unresolved.
Method
Bernini uses an MLLM planner to predict target semantics in ViT embedding space, which conditions a DiT renderer, with source VAE features added for editing detail preservation.
Results
Bernini achieves state-of-the-art performance across video generation, editing, and subject-to-video benchmarks, including OpenVE-Bench, OpenS2V-Eval, and Bernini-Bench.
Takeaways & Limitations
Using the MLLM’s ViT embedding space as a semantic bridge transfers pretrained understanding into generation and supports generalization across diverse video tasks.
Takeaways & Limitations
Bernini remains limited by its foundation models, depends on a strong LLM rewriter for complex edits, and has lower visual quality than stronger closed-source systems.
Abstract
from arXiv · showhide
Multimodal large language models (MLLMs) and diffusion models have each reached remarkable maturity: MLLMs excel at reasoning over heterogeneous multimodal inputs with strong semantic grounding, while diffusion models synthesize images and videos with photorealistic fidelity. We argue that these two families can be unified through a simple division of labor: MLLMs perform semantic planning, while diffusion models render pixels from high-level semantic guidance and low-level visual features. Building on this idea, we propose Bernini, a unified framework for video generation and editing. An MLLM-based planner predicts the target semantic representation directly in the ViT embedding space, and a DiT-based renderer synthesizes pixels conditioned on this plan, augmented by text features and, for editing, source VAE features for detail preservation. Because semantics serve as the interface, the planner and renderer can be trained separately and only lightly co-trained, preserving the pretrained strengths of both components while keeping training efficient. To better handle multiple visual inputs, we introduce Segment-Aware 3D Rotary Positional Embedding (SA-3D RoPE), and further incorporate chain-of-thought reasoning in the planner to better transfer understanding into generation. Bernini achieves state-of-the-art performance across a wide range of video generation and editing benchmarks, with the MLLM's pretrained understanding translating into strong generalization on challenging editing tasks.
1 Introduction
Bernini unifies MLLM semantic reasoning with diffusion-based video synthesis and editing through the MLLM’s ViT embedding space as an interface. Its shared protocol, data pipelines, and architectural components support diverse tasks and strong benchmark performance.
- Motivation: Bernini combines MLLM semantic reasoning with diffusion synthesis, assigning high-level understanding to the planner and detail preservation to the renderer.This division reflects the complementary strengths of MLLMs in multimodal reasoning and diffusion models in photorealistic generation.
- Semantic interface: The MLLM planner predicts target visual representations in its continuous ViT embedding space, creating a semantic bridge to the diffusion generator.A lightweight ViT embedding decoder progressively fills masked target tokens during inference.
- Architecture: The DiT renderer synthesizes video conditioned on the semantic plan, text features, and source VAE features for editing detail preservation.Bernini’s renderer performs flow matching in VAE latent space, while visual and text inputs are serialized into a unified sequence.
- Unified design: Bernini uses a shared input protocol across text-to-video, subject-to-video, and editing, with SA-3D RoPE for multiple visual sources.The framework also introduces multi-task data construction pipelines spanning video-image pretraining, video editing, and reference-guided generation.
- Results: Bernini achieves state-of-the-art performance across video generation, editing, and subject-to-video benchmarks, including OpenVE-Bench, OpenS2V-Eval, and Bernini-Bench.The reported benchmarks include the newly proposed Bernini-Bench.
2 Methods
Bernini unifies multimodal video generation and editing through an MLLM planner that predicts target semantic representations and a DiT renderer that synthesizes outputs from those representations and visual conditions. Its methods include unified multimodal serialization, masked semantic planning, source-detail preservation, segment-aware positional encoding, and separately specified training objectives.
- Framework Overview: Bernini uses an MLLM-based planner and DiT-based renderer, with an MLP connector mapping planner hidden states into renderer conditioning features.The renderer also uses text features and source visual conditions when available.
- Unified Input Formulation: A unified token sequence represents text-to-video, text-to-image, subject-to-video, and image or video editing tasks using textual and visual tokens.The sequence includes source visual inputs and target-output visual tokens.
- Mask-based Semantic Planning: Masked semantic planning trains the MLLM to infer randomly masked dense target visual embeddings from visible tokens and multimodal context, then progressively refines them during inference.The masking ratio is sampled from Beta(α, β), and inference begins with all target visual tokens masked before K refinement steps.
- DiT-based Rendering: The DiT renderer performs diffusion in VAE latent space conditioned on MLLM hidden states, incorporating source VAE features to preserve low-level detail and source consistency.The resulting target latent is decoded into the final output.
- Segment-Aware 3D RoPE: SA-3D RoPE adds segment-index encoding through a segment-dependent global phase modulation, enabling attention to distinguish visual tokens from different segments while preserving spatiotemporal modeling.The method augments the temporal, vertical, and horizontal positional encoding used by standard 3D RoPE.
- Training Objectives: Training combines next-token prediction for the MLLM with flow-matching objectives for the ViT embedding decoder and DiT renderer.The visual and rendering objectives operate in continuous ViT embedding space and VAE latent space, respectively, with corresponding loss weights.
3 Data
Bernini is trained on diverse data spanning video-pair, image-pair, interleaved image-text, propagation-based, human motion-aware, and reference-video-guided tasks. Its data construction addresses limited video-editing supervision through large-scale pairing, image-to-video transfer, motion-aware synthesis, and explicit reasoning supervision.
- Video-pair Data: Bernini constructs 20 million video pairs from general T2V corpora to address the limited scale and quality of existing video-editing data.The pipeline filters raw-video clips by similarity, duration, and content balance to produce diverse, balanced pairs.
- Image-pair Data: A nearly 30 million-pair image-manipulation dataset from over 300k tutorial videos provides diverse supervision for transferring mature image-editing knowledge to video editing.The data captures naturally occurring visual transformations, while image-to-video reformulation offers a practical route around the difficulty of collecting video-to-video data at scale.
- Propagation-based Data Boosting: Propagation-based boosting combines a base propagation model with a strong image-editing model to generate higher-quality addition, removal, replacement, and style-transfer data.This addresses artifacts and limited edit diversity in initial DiffuEraser- and VACE-based data.
- Human Motion-aware Data: A dual-branch human motion-aware synthesis framework fuses I2V motion adaptation with V2V source-motion consistency, enabling a controllable trade-off between action adaptation and motion preservation.The framework targets editing scenarios where object changes alter human pose and motion.
- Reference-video-guided Data: Reference-video-guided motion-transfer data is constructed as ⟨reference video, image, target video⟩ triplets using DWPose extracted from real videos.The setting animates a person in an image using motion from a reference video.
- Chain-of-Thought Data: Explicit self-text and self-vision-text Chain-of-Thought reasoning supplies structured prompts and intermediate visual states, improving editing fidelity and temporal coherence.Self-vision-text reasoning decomposes editing into image-level reasoning and video-level generation, providing explicit visual grounding.
4 Training and Inference
Bernini uses three-stage training to separately develop semantic planning and visual rendering before lightly co-training both components. At inference, the MLLM plans target ViT semantics iteratively, which condition the DiT renderer alongside text and source visual features.
- Three-stage training: Bernini trains the MLLM planner and DiT renderer separately before light co-training aligns semantic planning with visual rendering.This preserves the pretrained strengths of both components while avoiding excessive interference.
- Stage I: MLLM pretraining: Stage I progressively expands from text-to-image to text-to-video, image-pair, and video-pair data, while task-dependent masking supports heterogeneous generation, editing, and understanding.The mask ratio is sampled from a task-dependent Beta distribution, with more informative editing inputs shifting the ratio toward 1.0.
- Stage II: DiT pretraining: Stage II pretrains the DiT renderer on mixed generation and editing data, conditioning it on text and source VAE features for high-fidelity synthesis and source-preserving editing.Pair-data sampling is linearly decayed to reduce instruction-following and non-edited-region inconsistencies, with training at 480P and 16 fps.
- Stage III: Joint training: Stage III lightly co-trains both modules on image/video generation, editing, and understanding data, connecting ViT-space semantic predictions with VAE-latent rendering.Only a relatively small number of co-training steps is used, preserving strong understanding, reasoning, generation, and editing capabilities.
- Inference: During inference, the MLLM predicts masked target semantic tokens over 25 iterative planning steps, then the DiT renderer denoises using target semantics, text, and source visual features.The planner’s iterative inference adds negligible runtime relative to DiT rendering, while the renderer uses multi-source guidance in latent space.
5 Infrastructure
Bernini’s infrastructure addresses long-context training through memory optimization, specialized parallelism, sequence packing, and workload balancing. Multi-GPU inference and two-stage distillation further improve efficiency while preserving generation quality.
- Memory Optimization: Per-GPU memory fell from 72 GB to 40 GB through FSDP optimization and direct index-scattering into pre-allocated input buffers.The input pipeline avoided concatenating all visual and textual tokens before scattering them for sequence parallelism.
- Parallelism Strategy: Extending Ulysses sequence parallelism for the MLLM planner achieved 2× throughput at SP degree 4.FSDP weight sharding and Ulysses-style sequence parallelism were used across both DiT and MLLM components.
- Sequence Packing and Batch Forward: Sorting samples by sequence length for each sequence-parallel group achieved a 2× throughput speedup.Batch forward was also introduced for both MLLM and diffusion components using variable-length and joint processing strategies.
- Dataloader Balance: The load-balanced dataloader reduced the maximum-to-minimum workload ratio below 1.01 and improved throughput by approximately 15%.Greedy bin-packing redistributed heterogeneous workloads across nodes at each iteration.
- Inference Optimization: Multi-GPU inference achieved a speedup of over 7.2× using asynchronous communication for DiT QKV tensors and temporal context parallelism for the VAE.The VAE used asynchronous convolution-cache transmission along the temporal dimension.
- Sampling Efficiency: Two-stage CFG distillation and ReFlow reduced diffusion sampling cost while preserving generation quality.CFG distillation enabled single-pass prediction of the CFG-combined output, halving per-step compute by removing dual conditional and unconditional evaluations.
6 Experiments
Bernini is evaluated across video editing and generation, including the newly introduced Bernini-Bench, and achieves strong results against leading baselines. Ablations and qualitative analyses attribute performance to reasoning, semantic interfaces, and segment-aware positional encoding, while broader tests show generalization beyond training distributions.
- Bernini-Bench: Bernini-Bench contains 300 test cases across 22 editing categories, including action, position, causal-reasoning, and camera-focus edits absent from other benchmarks.The benchmark uses rich editing instructions and diverse source videos collected from free, open-source stock-media platforms.
- Video Editing: On Bernini-V2V, Bernini raises the overall score from 3.30 to 3.49 compared with Wan2.7 and consistently outperforms Kling O3 across evaluation dimensions.Human evaluation finds Bernini competitive with Wan2.7 and particularly stronger in Video Consistency.
- Video Editing: Bernini outperforms VINO on OpenVE with an overall score of 4.04 vs. 3.18 and reaches 8.02 on EditVerse editing quality.It also delivers strong performance across the diverse evaluation settings of OpenVE, EditVerse, and FiVE.
- Ablations: Enriching the reasoning context consistently improves Bernini-V2V performance, with self-text reasoning outperforming prompt enhancement from the planner’s initialization model.The prompt-enhancer variant provides only slight baseline improvements and performs worse than self-text reasoning.
- Video Generation: Bernini reaches a Total score of 84.64 on VBench, essentially matching Wan2.2-A14B at 84.79 after extending it to a unified editing framework.On OpenS2V-Eval, it achieves the highest Total score of 62.94 and a FaceSim score of 78.20 versus Kling O3’s 57.20.
- Generalization and Ablations: Bernini generalizes to stylization, animation, weather, effects, motion, focus, position, and causal edits, including inferring that prolonged heavy rain extinguishes a fire.Removing the ViT semantic interface weakens instruction following, while removing both ViT and MLLM further degrades precise, faithful editing; SA-3D RoPE is separately evaluated against two positional baselines.
7 Related Work
Related work follows two main strategies for connecting multimodal understanding with visual generation: unified multimodal backbones and separate MLLM–diffusion systems. The latter differ mainly in which MLLM representation serves as the conditioning interface, from text or query tokens to hidden states and video-specific multimodal inputs.
- Joint Multimodal Backbones: Joint multimodal backbones process text and visual tokens together in a unified sequence, as exemplified by Emu3 and Janus.Emu3 tokenizes text, images, and videos into a shared discrete vocabulary and trains a single transformer from scratch with pure next-token prediction; Janus decouples visual encoders for understanding and generation.
- MLLMs as Conditioners for Visual Generation: Separate MLLM–diffusion systems primarily differ in the representation passed from the MLLM to the visual generator.Narrow interfaces use output text tokens or learnable query tokens, while wider interfaces use hidden states directly.
- MLLMs as Conditioners for Visual Generation: Video extensions couple MLLMs with video diffusion backbones and condition generation on hidden states, optional query or multimodal context, and reference VAE latents.UniVideo and VInO use this interface to support video generation settings including subject-to-video.
8 Conclusion and Limitations
Bernini unifies semantic planning and pixel rendering for video generation and editing, achieving state-of-the-art results while retaining limitations from its foundation models and reliance on detailed LLM-rewritten instructions.
- Contributions: Bernini decouples semantic planning from pixel rendering for unified video generation and editing.An MLLM planner predicts targets in native ViT embedding space, while a DiT renderer synthesizes pixels conditioned on the plan, text, and source VAE features.
- Contributions: SA-3D RoPE and a latent chain-of-thought planner support multi-segment disambiguation and semantic transfer into generation.The planner and renderer can be trained largely independently while preserving their pretrained strengths.
- Results: Bernini achieves state-of-the-art results across video editing and subject-to-video benchmarks and generalizes beyond standard training cases.Its reported generalization includes challenging video editing instructions.
- Limitations: Bernini remains limited by its adopted MLLM and DiT foundation models and depends on a strong LLM rewriter for complex edits.This dependence indicates that its native reasoning ability is not yet fully sufficient for challenging editing scenarios.
- Limitations: Although Bernini achieves state-of-the-art consistency in subject-to-video generation, its visual quality falls short of stronger closed-source systems.The limitation concerns visual quality rather than consistency.
9 Contributions and Acknowledgements · Appendix · A MLLM Prompts for Bernini-Bench Evaluation
The paper records contribution roles and acknowledgements, then specifies MLLM-based evaluation prompts for instruction-driven and reference-guided video editing. These prompts score compliance, consistency, reference matching where applicable, generation quality, and overall performance on a 1–5 scale.
- 9 Contributions and Acknowledgements: Algorithm authors contributed equally and are listed alphabetically by first name, with Lu Chi marked as Project Lead.The algorithm contributors are Chenchen Liu, Junyi Chen, Lei Li, Lu Chi†, Mingzhen Sun, and Zhuoying Li.
- 9 Contributions and Acknowledgements: The authors thank Ruibiao Lu, Mingyang Zou, and Zhen Ye for supporting the project.
- A MLLM Prompts for Bernini-Bench Evaluation: The Bernini-Bench prompt evaluates original-versus-edited videos across four dimensions: Instruction Compliance, Video Consistency, Generation Quality, and Overall Performance.Each dimension uses a 1–5 scale, with non-execution assigned Instruction Compliance 1 and Overall Performance 1 while consistency and quality are marked N/A.
- A MLLM Prompts for Bernini-Bench Evaluation: Video Consistency assesses preservation of non-edited regions, identity, layout, style, and motion while excluding changes that are necessary consequences of the instruction.
- A MLLM Prompts for Bernini-Bench Evaluation: Both prompts evaluate Generation Quality through artifacts, temporal stability, seamless integration, and physical plausibility, and Overall Performance through holistic user satisfaction.The reference-guided prompt outputs valid JSON containing brief reasoning and the five dimension scores, including N/A when non-execution applies.
- A MLLM Prompts for Bernini-Bench Evaluation: The Bernini-V2V prompt adds Reference Image Consistency to the four core dimensions, evaluating appearance resemblance contextually rather than by pixel-level copying.It considers shape, colour, texture, pattern, style, material, and identity while requiring natural integration into the video scene.
- A MLLM Prompts for Bernini-Bench Evaluation: Instruction Compliance measures whether the correct target and action are executed with the specified structural and semantic requirements, independently of reference-image appearance.
B Experimental Results · B.1 More Qualitative Comparison with SoTA Methods
Bernini’s qualitative comparisons show more precise instruction following than competing methods on video-to-video and reference-video-to-video editing tasks. It preserves unrelated content while correctly modifying motion, focus, hairstyle, background details, and reference-image style.
- B.1 More Qualitative Comparison with SoTA Methods: On V2V editing, Bernini correctly changes the white tiger’s motion to scratching, unlike Wan-2.7 and Kling-O3, while VINO produces an unnatural paw interaction.Figure 19 reports this additional V2V comparison.
- B.1 More Qualitative Comparison with SoTA Methods: Bernini alone shifts focus to the girl in the background while blurring the foreground in the V2V focus-editing case.The supplied passage identifies this as the middle V2V case in Figure 19.
- B.1 More Qualitative Comparison with SoTA Methods: In the V2V comparisons, Bernini follows the requested edits more faithfully than Wan-2.7, Kling-O3, and VINO across motion and focus changes.The comparison covers additional V2V results shown in Figure 19.
- B.1 More Qualitative Comparison with SoTA Methods: On VR2V editing, Bernini modifies only the girl’s hairstyle, whereas VINO and Kling alter facial features and Wan-2.7 changes overall lighting.Figure 20 presents these additional qualitative VR2V comparisons.
- B.1 More Qualitative Comparison with SoTA Methods: In the VR2V background-editing case, Bernini alone removes the snow beneath the person’s feet from the original video.The supplied passage identifies this as the middle VR2V case in Figure 20.
- B.1 More Qualitative Comparison with SoTA Methods: In the rightmost VR2V case, only Bernini accurately applies the reference image’s style.Figure 20 presents this additional qualitative comparison.
B.2 Qualitative Comparison of Video Editing with Reasoning
Figure 21 shows that self-generated textual reasoning improves instruction following in complex video editing, while LLM prompt rewriting further enhances structure and precision. Intermediate chain-of-thought images provide essential visual grounding.
- Qualitative comparison: Figure 21 shows that the baseline struggles with complex layout adjustments and imaginary state changes.These limitations are observed in qualitative comparisons across reasoning variants.
- Qualitative comparison: Self-generated textual reasoning yields notable improvements in instruction following for complex video editing.The improvement is shown through qualitative comparisons across different reasoning variants.
- Qualitative comparison: An LLM rewriter that refines input prompts further improves the structure and precision of generated results.Prompt refinement enhances performance beyond incorporating self-generated textual reasoning alone.
- Qualitative comparison: Intermediate chain-of-thought images provide essential visual grounding for video editing with reasoning.The passage identifies these intermediate images as an additional component of the reasoning process.
B.3 More Generalization Results
Bernini generalizes effectively to diverse video editing instructions absent from its V2V training data, indicating transferable instruction-following. These include expression, perspective, spatial and temporal reasoning, effects, atmosphere, interaction, and composed editing.
- Generalization to Diverse Editing Instructions: Bernini executes diverse editing instructions absent from V2V training, including expression, perspective, spatial and temporal reasoning, effect addition, atmosphere rendering, subject interaction, and composed editing.Figure 22 presents additional generalization examples across these instruction types.