PaperlayerView exact revision on arXiv

Source-linked AI summary

Making MLLM Latents Matter: Factor-Aware Predictive Learning for Cinematic Video Generation

Songlin Yang, Zhe Wang, Xiaotong Zhao, Xi Wang, Ran Zhang, Shuai Li, Taiyi Wu, Songlian Li, Eric Liu, Alan Zhao, Anyi Rao

arXiv:2610.30926v1cs.CV

TL;DR

Cinematic video generation must infer temporal details from limited inputs while controlling character, scene, action, and camera, but MLLM representations are not automatically effective for generation. FAPL grounds the DiT in MLLM latents and uses factor-aware and predictive learning signals; it improves performance across sparse-to-video, factor-aware editing, and multi-shot generation tasks.

  • Problem

    Cinematic video generation requires temporal inference from sparse guidance and compatible control of character, scene, action, and camera, while native conditioning pathways can let the DiT bypass MLLM latents.

    Source: §1 · pp. 1–2

  • Method

    FAPL grounds the DiT in MLLM latents, uses paired-video factor-span replacement to supervise changes and preservation, and trains the MLLM and connector through a frozen DiT with shared video targets.

    Source: §1 · p. 3, §3.3 · p. 7

  • Results

    Across sparse-to-video, factor-aware editing, and multi-shot generation, FAPL improves action ordering and camera fidelity, requested factor changes, and cross-shot Subject and Scene Consistency.

    Source: §4.3 · p. 8

  • Takeaways & Limitations

    The findings suggest multimodal reasoning and generation can reinforce each other: reasoning guides controllable synthesis, while generation provides grounded supervision for inferring missing content from limited observations.

    Source: §5 · p. 10

    Performance remains bounded by the underlying models: current MLLMs struggle with 3D blockouts and complex camera motion, while the pretrained DiT limits visual fidelity and complex factor realization.

    Source: §5 · p. 10

Abstract

from arXiv · show

Cinematic video generation requires both temporal inference from limited inputs (i.e., sparse rather than frame-by-frame guidance) and factor-aware control of character, scene, action, and camera. Multimodal large language models (MLLMs) offer useful priors for both capabilities, but their representations are not automatically actionable for video generation: a Video Diffusion Transformer (DiT) may bypass them through native conditioning pathways, and using them does not ensure either factor-aware control or generation-compatible temporal prediction. We propose Factor-Aware Predictive Learning (FAPL), which establishes effective MLLM latent use as a shared prerequisite and then targets the two capabilities with complementary learning signals. First, latent-only generator grounding trains the DiT with a frozen MLLM and connector, strengthening the DiT's use of MLLM latents for generation. Second, for factor-aware control, paired videos specify a target-factor change and non-target preservation, and factor-span replacement trains the DiT to realize this relationship in its output. Third, for temporal inference, text-only and text-plus-complete-video observations share a video target, and the trained DiT is then frozen while the MLLM and connector are optimized through it. This final stage trains limited-input representations for generation while retaining complete-observation reconstruction as a complementary training mode. Extensive experiments demonstrate our consistently better performance on three cinematic video generation tasks (i.e., sparse-to-video, factor-aware editing, and multi-shot generation) over baselines and validate FAPL's contributions to MLLM feature use, temporal control, and factor coordination.

1 INTRODUCTION

Cinematic video generation must infer temporal details from sparse inputs while controlling character, scene, action, and camera compatibly. FAPL addresses the gap between MLLM understanding and effective generation by training the DiT to use MLLM latents and learning factor-aware and predictive signals.

1 INTRODUCTION

  • Cinematic video generation requires plausible action sequences and camera motion from sparse guidance, with changes to one factor preserving the requested content of others.The central challenge combines inference from limited inputs with selectively controllable generation.Source: §1 · p. 1
  • MLLMs offer visual priors and semantic reasoning for interpreting scripts and references, inferring temporal details, and coordinating cinematic factors.Recent methods condition Video Diffusion Transformers on projected MLLM hidden states.Source: §1 · p. 2
  • Native text-encoder and VAE-reference pathways can reduce generation loss without substantial reliance on MLLM latents, while existing supervision lacks task-structured signals.Factor control lacks explicit links between a span intervention, intended change, and non-target preservation; sparse-input adaptation around a frozen MLLM does not train it to produce temporally informative representations.Source: §1 · p. 2
  • FAPL first trains the DiT with frozen MLLM latents as sole content conditions, then uses paired videos and factor-span replacement to supervise target changes and preservation.For temporal inference, it freezes the trained DiT and optimizes the MLLM and connector using text-only and text-plus-complete-video observations with a shared video target.Source: §1 · p. 3
  • FAPL links limited-input prediction and complete-observation reconstruction through shared video targets, and supervises factor-level change and preservation without assuming fully disentangled representations.These are two of the paper’s stated contributions alongside translating MLLM understanding into actionable guidance for limited-input inference and factor-aware control.Source: §1 · p. 3

2 RELATED WORK

Prior work addresses limited-input cinematic generation and factor control through separate methods, while MLLM-conditioned systems commonly use video-level supervision. FAPL instead uses latent-only training, factor-span replacement, and prediction through a fixed video generator.

Cinematic Video Generation

  • Cinematic video studies address limited-input content inference and factor-aware control, with camera motion, character replacement, action transfer, and scene modification often handled separately.Reference-driven and multi-shot methods also target identity and scene consistency from given references.Source: §2 · p. 3

MLLM-Conditioned Video Generation and Editing

  • MLLM-conditioned video systems commonly combine MLLM features with text-encoder and VAE-reference features, while video-level supervision does not specify each condition’s contribution or which factor should change.Paired editing examples are typically additional supervised samples rather than explicit signals for both change and preservation.Source: §2 · p. 3
  • FAPL strengthens MLLM latent use through latent-only training and uses factor-span replacement between paired examples to supervise factor-aware changes and preservation.Its three-stage framework grounds the DiT, trains factor-aware control, then uses a fixed DiT while optimizing the MLLM and connector across shared targets.Source: §2 · p. 4, Figure 3 · p. 4

Predictive Representation Learning

  • Predictive representation methods often use task-specific spaces without high-fidelity video decoders or require separately trained decoders; FAPL instead predicts through visual reconstruction with a fixed generator.This reuses a pretrained MLLM and Video DiT while training the MLLM to infer video content from limited observations.Source: §2 · p. 4

3 METHOD

FAPL trains MLLM representations to support cinematic video generation through latent-only generator grounding, factor-span interventions, and prediction through a fixed generator. Its paired observations and edits structure training around shared video targets, selected factor changes, and preservation of other content.

3 METHOD

  • FAPL combines MLLM-only generator grounding, factor-span interventions, and shared-target prediction through a fixed generator to train representations for limited-input inference and factor-aware control.All three stages use MLLM latents as the only content conditions.Source: §3 · p. 4

3.1 PRELIMINARY: FRAMEWORK AND PAIRED DATA

  • Paired observations and paired edits provide signals for DiT use of MLLM information and factor-aware responses without introducing a new representation network.Source: §3.1 · p. 5

3.1.1 FRAMEWORK: MLLM-CONDITIONED VIDEO DIFFUSION

  • MLLM-conditioned video diffusion combines text-encoder, MLLM, and VAE-reference features as concatenated DiT conditions, but reconstruction alone does not establish effective use of MLLM latents.The DiT may reduce generation loss by relying primarily on text and VAE features.Source: §3.1.1 · p. 5

3.1.2 PAIRED DATA

  • Observation pairs use text-only and text-plus-complete-video inputs with the same target, while editing pairs change one structured prompt field among character, scene, action, and camera.Factor spans are located in projected latents, need not have equal lengths, and are not assumed to be fully disentangled.Source: §3.1.2 · pp. 5–6

3.2 STAGE I: MLLM-ONLY GENERATOR GROUNDING

  • Stage I freezes the MLLM and connector and trains the DiT on MLLM latents alone across text-only and text-plus-complete-video observations.Disabling separate text-encoder and VAE-reference conditions requires the DiT to use MLLM latents across both modes.Source: §3.2 · p. 6

3.3 STAGE II: FACTOR-AWARE CONTROL IMPROVEMENT

  • Stage II freezes the MLLM and connector and trains the DiT on text-only latent conditions formed by replacing a selected factor span between paired descriptions.The target video supervises the replaced span’s effect and preservation of remaining factors, without a feature-separation objective.Source: §3.3 · pp. 6–7

3.4 STAGE III: PREDICTION FROM LIMITED INPUTS

  • Stage III freezes the DiT and jointly fine-tunes the MLLM and connector using text-only and text-plus-complete-video inputs supervised by the same video target.Gradients pass through the fixed DiT to adapt representations for reconstructing semantic, motion, and temporal content from limited evidence.Source: §3.4 · p. 7

3.5 INFERENCE AND CINEMATIC CONTROL

  • At inference, factor-aware control replaces only the edited factor’s span, while multi-shot generation uses shared character and scene references and varies action and camera instructions by shot.Script-and-reference generation does not require a complete target video.Source: §3.5 · p. 7

4 EXPERIMENTS

The experiments evaluate FAPL on sparse-to-video generation, factor-aware editing, and multi-shot generation, then ablate its training stages. FAPL improves the reported cross-task outcomes, with stage-specific gains in factor editing and temporal control.

4.1 IMPLEMENTATION

  • The 13B DiT training uses 10K, 15K, and 10K steps for Stages I–III, respectively, with 100K quality-filtered source–target video pairs shared across stages.Stages I and III use text-only and text-plus-complete-video observations at a 1:1 ratio; Stage II uses text-only inputs with factor-span replacement.Source: §4.1 · pp. 7–8

4.3 COMPARISON AND ANALYSIS

  • Across three tasks, FAPL improves action ordering, camera fidelity, factor-editing ∩-Acc, and cross-shot consistency; FaceSim-Arc is 0.523 versus UniVideo’s 0.451.The other FaceSim-Arc comparisons are 0.412 for Omni-Video 2 (1.3B) and 0.468 for Omni-Video 2 (14B).Source: Table 1 · p. 8, §4.3 · p. 8

4.2 EVALUATION DATA, TASKS, AND METRICS

  • Evaluation uses 5K text-only sparse-to-video cases, 5K factor-editing cases, and 100 multi-shot sequences, with task-specific temporal, consistency, and editing metrics.The multi-shot sequences contain two to four shots; factor-editing cases are evenly split across four factors.Source: §4.2 · p. 8

4.3 COMPARISON AND ANALYSIS

  • Examples show FAPL following text-specified appearance, scene, action, and camera details while changing requested factors and preserving unrelated content.UniVideo misses details such as hair length, hand gestures, or an orbiting camera, and can alter non-target content during edits.Source: §4.3 · p. 8, Figure 4 · p. 9, Figure 5 · p. 9

4.4 ABLATION STUDY

  • Stage I improves action ordering and both camera errors over the pretrained model, while Figure 7 illustrates latent-enhancement gains across three MLLM–DiT models.The illustrated models are Omni-Video 2 (1.3B and 14B) and UniVideo (13B).Source: §4.4 · p. 9, Figure 7 · p. 10
  • Stage II factor swapping achieves 57% ∩-Acc, compared with 48% without factor swapping under the same paired data and optimization budget.Motion Order and camera errors do not improve over the no-swapping control.Source: §4.4 · p. 9, Table 2 · p. 10
  • Mixed-observation Stage III raises Motion Order from 57% to 61% and reduces translation error from 0.0629 to 0.0586 and rotation error from 1.26 to 1.02 radians.Editing accuracy remains comparable to Stage II, changing from 57% to 58%.Source: §4.4 · p. 9, Table 2 · p. 10
  • Figure 6 illustrates consistent multi-shot generation without cross-shot memory.Source: Figure 6 · p. 10

5 CONCLUSION

FAPL makes MLLM representations actionable for temporal inference and factor-aware control by grounding generation in latents, supervising factor changes and preservation, and learning prediction from limited inputs. Its performance remains bounded by the underlying MLLMs and pretrained DiT.

5 CONCLUSION

  • FAPL makes MLLM representations actionable for temporal inference and factor-aware control in cinematic video generation.Source: §5 · p. 10
  • The findings connect multimodal reasoning with generation: reasoning guides controllable synthesis, while generation provides grounded supervision for inferring missing content from limited observations.Source: §5 · p. 10
  • Current MLLMs struggle with 3D blockouts and complex camera reasoning, while the pretrained DiT limits fidelity and complex factor realization such as realistic acting.Source: §5 · p. 10

Related papers

Loading 2610.30926v1…