MLLM-Guided Semantic Correction for Text-to-Video Generation

Junhao Chen; Zheqi Lv; Keting Yin; Shengyu Zhang; Zhou Zhao; Feiyang Chen; Xinyu Duan; Baoxing Huai; Fei Wu

Source-linked AI summary

MLLM-Guided Semantic Correction for Text-to-Video Generation

Junhao Chen, Zheqi Lv, Keting Yin, Shengyu Zhang, Zhou Zhao, Feiyang Chen, Xinyu Duan, Baoxing Huai, Fei Wu

arXiv:2608.16513v1Aug 17, 2026cs.CVcs.AI

TL;DR

Text-to-video diffusion models struggle to detect and correct semantic deviations during generation. This paper introduces a training-free MLLM-guided mid-generation correction framework that improves semantic alignment, visual fidelity, and temporal consistency without modifying model parameters.

Problem
Text-to-video diffusion models lack mechanisms to detect and correct intermediate semantic deviations such as missing objects, incorrect attributes, and mismatched actions.
Method
The training-free framework uses MLLM feedback through Semantic Assessment Supervisor previews and Semantic Modification Assistant interventions within the diffusion sampling loop.
Results
SASMA improves semantic alignment and perceptual quality across three architectures while enhancing temporal consistency and visual-semantic alignment across diverse scenes.
Takeaways & Limitations
External multimodal reasoning during sampling can enhance semantic coherence in text-to-video generation without architecture changes or task-specific training.
Takeaways & Limitations
The method depends partly on the underlying MLLM’s reliability and generalization, which can lead to suboptimal corrections for some attributes or object categories.

Abstract

from arXiv · show

Recent advances in diffusion models and Transformer architectures have led to significant progress in text-to-video generation. However, these models often suffer from semantic errors such as missing objects, incorrect attributes, or mismatched actions. Although some semantic correction methods perform optimization before sampling or refinement after sampling, how to detect and correct semantic deviations during the video generation process remains underexplored. In this paper, we introduce a training-free, interpretable mid-generation correction framework that integrates multimodal large language model (MLLM) feedback directly into the diffusion sampling loop. Our framework achieves diffusion trajectory correction by injecting semantic evaluation signals during video synthesis, enabling the model to optimize the generated content through continuous self-reflection. We propose two key modules: a Semantic Assessment Supervisor that generates intermediate preview frames for semantic evaluations and deviation diagnostics, and a Semantic Modification Assistant that corrects semantic drift during inference via a controllable latent trajectory intervention. Our method improves semantic alignment, visual fidelity, and temporal consistency without modifying model parameters. We validate the effectiveness of our approach through extensive experiments across multiple benchmarks.

I. INTRODUCTION … III. METHOD

The paper presents SASMA, a training-free and interpretable framework that uses MLLM feedback during diffusion sampling to detect and correct semantic drift in text-to-video generation. It combines intermediate semantic assessment with controllable latent intervention to improve semantic alignment, visual fidelity, and temporal consistency without modifying model parameters.

I. INTRODUCTION: Text-to-video diffusion models can produce semantic errors, including missing objects, incorrect attributes, and mismatched actions, despite recent progress in prompt adherence and video quality.These errors motivate correction mechanisms that operate during generation rather than relying solely on fixed guidance or endpoint refinement.
I. INTRODUCTION: Existing correction strategies include non-self-correcting guidance, starting-point optimization, and endpoint refinement, but they cannot reliably prevent semantic drift throughout sampling.Static high CFG scales can reduce visual fidelity and motion diversity while amplifying accumulated errors; pre-sampling correction cannot address later drift, and post-generation refinement reacts only after errors occur.
I. INTRODUCTION: SASMA embeds MLLM semantic evaluation signals directly into the diffusion sampling loop for training-free, interpretable, online trajectory correction.The framework treats generation as an evolving process that can be inspected and adjusted during synthesis.
I. INTRODUCTION: The Semantic Assessment Supervisor interprets intermediate states, while the Semantic Modification Assistant performs controllable corrective interventions during inference.The supervisor uses selected sampling steps and preview representations for evaluation; the assistant applies enhancement and suppression cues without changing generator parameters.
I. INTRODUCTION: Experiments across multiple text-to-video benchmarks show improved semantic consistency, visual fidelity, and temporal coherence from MLLM-guided feedback during diffusion sampling.The reported validation covers the framework’s claimed benefits without modifying model parameters.
A. Text-to-Video Generation: Text-to-video research comprises training-based and training-free paradigms, with training-based methods learning spatial and temporal correlations from paired video–text data.The related-work discussion situates the proposed method within the training-free line of text-to-video generation research.
B. LLM-assisted Video Generation: LLM-assisted video-generation methods commonly use language models as high-level planners that convert textual prompts into structured scene descriptions or multiscene scripts.Examples include LVD’s detailed scene layouts and VideoDrafter’s multiscene scripts for guiding diffusion-based synthesis.
III. METHOD: The method formalizes diffusion-based generation and introduces two components that jointly enable dynamic reasoning and semantic correction during inference.The framework is organized around the Semantic Assessment Supervisor and Semantic Modification Assistant, which integrate MLLM feedback into ongoing denoising.

A. Background and Preliminaries … B. Semantic Assessment Supervisor

The framework models text-to-video diffusion in latent space and introduces MLLM feedback for semantic reasoning during sampling. The Semantic Assessment Supervisor exposes intermediate semantics through predicted clean latents and selectively diagnoses inconsistencies for later correction.

1) Diffusion model:: The text-to-video diffusion model generates video latents conditioned on an embedding c = E(p) derived from prompt p.The pretrained text encoder maps the textual prompt into a semantic feature space.
1) Diffusion model:: Sampling proceeds from Gaussian noise xT ∼N(0, I) through latent states {xt}T t=0, producing a clean latent x0.The deterministic DDIM formulation defines the denoising trajectory, with ˆx0 estimating the corresponding clean latent.
1) Diffusion model:: After denoising, the clean latent x0 is decoded by D(·) into a sequence of N video frames.The decoder converts the final latent representation into the generated video.
2) Incorporating multimodal reasoning:: At arbitrary timesteps, the framework partially decodes xt into a coarse visual representation and combines it with prompt p for MLLM semantic reasoning.The MLLM produces semantic feedback St containing multimodal understanding and potential corrective signals.
B. Semantic Assessment Supervisor: Because intermediate xt states are dominated by stochastic noise and lie far from an interpretable visual manifold, direct reasoning over them provides little semantic insight.The supervisor therefore exposes evolving semantics without disrupting the generative trajectory.
B. Semantic Assessment Supervisor: The supervisor reinterprets DDIM’s predicted clean latent ˆx0 as the model’s instantaneous hypothesis of the underlying clean sample.This provides the basis for semantic assessment during the diffusion trajectory.
B. Semantic Assessment Supervisor: At scheduled timesteps t = tstart + k · ∆, the supervisor decodes low-fidelity previews that remain semantically aligned with the evolving generation state.Each preview and prompt are evaluated by the MLLM for inconsistencies such as object incompleteness or attribute mismatch.
B. Semantic Assessment Supervisor: If visual-semantic consistency is satisfied, correction terminates early; otherwise, structured diagnostics generate corrective prompts and embeddings for subsequent diffusion updates.This early termination avoids unnecessary semantic injection and reduces computational overhead.

DIFFUSION SAMPLING

The diffusion sampling procedure estimates the clean sample and updates the latent iteratively using the model’s noise prediction. At selected timesteps, it applies an additional correction update before continuing the sampling loop.

Sampling update: The sampler estimates the clean sample ˆx0 from the current latent xt, timestep t, conditioning c, and predicted noise ϵθ(xt, t, c).The estimate is computed as (xt − √(1 − αt)ϵθ(xt, t, c))/√αt.
Sampling update: It computes the preceding latent xt−1 by combining ˆx0 with the predicted noise under the schedule coefficients αt−1.The update is xt−1 ← √αt−1 ˆx0 + √(1 − αt−1)ϵθ(xt, t, c).
Conditional correction: When t belongs to the selected timestep set T, the procedure performs an additional latent and clean-sample update using the modified latent ˜xt and conditioning change ∆c±.The correction is embedded within the sampling loop and is followed by continuation of the timestep iteration.

C. Semantic Modification Assistant · 1) Semantic Dilution: · 2) Semantic Injection:

The Semantic Modification Assistant performs controllable bidirectional refinement within the diffusion trajectory, diluting accumulated semantic bias before injecting corrected guidance and resuming normal denoising. This three-step process integrates high-level semantic feedback while maintaining temporal coherence, preserving model priors, and keeping correction stable and interpretable.

C. Semantic Modification Assistant: Semantic injection performs controllable bidirectional refinement within the diffusion trajectory while maintaining temporal coherence and preserving model priors.The process is designed to incorporate corrective feedback directly into the trajectory rather than applying correction only before or after sampling.
C. Semantic Modification Assistant: The mechanism partially reverses accumulated semantic bias before reintroducing corrected guidance under updated conditions.Together, the two components form the full algorithm presented in Algorithm 1.
C. Semantic Modification Assistant: The semantic injection process consists of three sequential steps applied during diffusion sampling.These steps are semantic dilution, semantic injection, and trajectory resumption.
1) Semantic Dilution:: Given latent x_t−1, a one-step back transition produces intermediate latent x̃_t by weakening previously accumulated conditional influence.This operation initiates the semantic dilution stage.
1) Semantic Dilution:: The dilution step uses the unconditional case to remove condition-induced semantic information, producing a semantically neutral latent for feedback-guided correction.The neutral latent serves as a clean basis for subsequent semantic modification.
2) Semantic Injection:: Modified conditioning signals are used in a feedback-guided denoising step to inject corrected semantic information into the diffusion trajectory.This step applies the updated semantic guidance after dilution.
2) Semantic Injection:: The denoised estimate under modified guidance yields refined latent x̃_t−1 incorporating high-level feedback from the preview representation.The resulting latent carries the corrected semantic information forward in the trajectory.
2) Semantic Injection:: The corrected latent re-enters the normal diffusion trajectory and continues denoising with the original conditioning c.Resuming normal denoising ensures generation proceeds on the corrected trajectory.

IV. THEORETICAL FOUNDATIONS · A. Semantic Correction as a Three-Step Diffusion Operator

The section theoretically characterizes SASMA as a semantic-feedback modification of the diffusion trajectory and analyzes when it reduces denoising error relative to standard DDIM inference. It formulates correction as three sequential operations: semantic dilution, semantic injection, and trajectory resumption.

IV. THEORETICAL FOUNDATIONS: SASMA’s theoretical analysis characterizes how semantic feedback injection alters diffusion trajectories and establishes conditions for strictly smaller denoising error than standard DDIM inference.
IV. THEORETICAL FOUNDATIONS: One semantic correction cycle consists of three sequential operations, which yield an exact four-term update decomposition for comparison with baseline diffusion.
A. Semantic Correction as a Three-Step Diffusion Operator: At timestep t, SASMA modifies the standard diffusion trajectory through semantic dilution, semantic injection, and trajectory resumption.
A. Semantic Correction as a Three-Step Diffusion Operator: Semantic dilution forwards-diffuses x_t−1 into an intermediate noisy state, creating a re-noised representation for non-local semantic-feedback influence.
A. Semantic Correction as a Three-Step Diffusion Operator: Semantic injection backward-diffuses the intermediate state under refined semantic guidance ∆c±, directly incorporating corrections into denoising and producing refined latent x̃_t−1.
A. Semantic Correction as a Three-Step Diffusion Operator: Trajectory resumption continues diffusion under the original prompt condition c after semantic correction.

B. Exact Algebraic Reformulation

This section derives an exact algebraic reformulation by successively substituting estimator definitions, expanding terms, and collecting them into a four-term decomposition. The reformulation further exploits a coefficient cancellation, with all coefficients determined solely by the noise schedule.

Equation (12) is first rewritten by substituting the definition of ˆx0.
Equation (13) is similarly reformulated, after which the result is substituted into Equation (15) and expanded.
Substituting ˜xt from Equation (11) and collecting terms produces a four-term decomposition.
The decomposition coefficients η1–η4 depend only on the noise schedule, and η2 + η3 = 0 enables a further rewrite.

C. Comparison with Standard DDIM · D. Denoising Error Analysis

SASMA differs from standard DDIM by adding an explicit semantic correction term that compensates for discrepancies between original and refined noise predictions. Its denoising-error analysis derives a corrected bound and concludes that SASMA can achieve lower error under mild, practically satisfied assumptions.

C. Comparison with Standard DDIM: Standard DDIM provides the reference update from timestep t−1 to t−2.The update is introduced as the baseline for comparison.
C. Comparison with Standard DDIM: SASMA adds a semantic correction term that explicitly compensates for discrepancies between original and refined noise predictions.This distinguishes its update from standard DDIM.
D. Denoising Error Analysis: The denoising-error analysis defines standard DDIM’s error relative to the true noise ϵ at timestep t−1.The true-noise reference establishes the error quantity used for comparison.
D. Denoising Error Analysis: For SASMA, the analysis separately derives the corresponding denoising error after semantic correction.This provides the corrected counterpart to the standard DDIM error.
D. Denoising Error Analysis: The analysis introduces a state correction gain and a semantic correction magnitude to characterize SASMA’s intervention.These quantities support the subsequent error-bound derivation.
D. Denoising Error Analysis: The triangle-inequality analysis yields the bound δSASMA ≤|η4|(∥ϵθ(xt−1, t −1, c) −ϵ∥−∆state) + |η3|Csem.The bound combines the baseline denoising discrepancy, state correction, and semantic correction contributions.
D. Denoising Error Analysis: Under the stated enhanced-semantic-guidance condition, SASMA has lower denoising error than standard DDIM, completing the theoretical justification.The conclusion is framed as holding under mild and practically satisfied assumptions.

V. EXPERIMENTS … 2) Qualitative Results:

Experiments evaluate SASMA on VBench and ChronoMagic-Bench against several video diffusion baselines under controlled inference settings. Results indicate improved semantic alignment, perceptual quality, temporal consistency, and visual-semantic alignment, with interpretable mid-generation feedback across challenging scenarios.

1) Benchmarks and Evaluation Metrics:: Experiments use VBench and ChronoMagic-Bench to assess subject consistency, temporal coherence, text-video alignment, temporal reasoning, and longhorizon consistency.VBench provides diverse fine-grained dimensions, whereas ChronoMagic-Bench emphasizes generative temporal reasoning and longhorizon consistency.
2) Comparison Baselines:: SASMA is compared with CogVideoX1.5, HunyuanVideo, and AnimateDiff using default settings, benchmark prompts, and multiple samples per prompt.Multiple samples measure the stability and consistency of semantic correction performance.
3) Implementation Details:: All methods use NVIDIA RTX 3090 GPUs, DDIM sampling with T = 50 steps, and identical random seeds and diffusion hyperparameters.Semantic injection runs from ts = 0.1T to te = 0.9T with interval ∆= 5, using VideoLLaMA3-7B in a training-free inference setting.
1) Quantitative Results:: VBench results report seven per-dimension metrics and three aggregated scores across CogVideoX1.5, HunyuanVideo, and AnimateDiff.The aggregated metrics are Quality Score, Semantic Score, and Total Score.
1) Quantitative Results:: SASMA improves overall performance across three architectures, spanning semantic alignment and perceptual quality dimensions.Its gains adapt to baseline strength, with substantial structural and aesthetic improvements for CogVideoX1.5 and finer improvements for HunyuanVideo.
2) Qualitative Results:: SASMA enhances temporal consistency and visual-semantic alignment in text-to-video generation scenarios.The qualitative comparison attributes these improvements to MLLM-guided feedback introduced during diffusion, without retraining.
2) Qualitative Results:: SASMA provides transparent and traceable feedback during diffusion, with qualitative coverage spanning multi-object arrangements, spatial relationships, dynamic human activities, and natural environments.Figure 4 illustrates semantic evaluation and correction along the denoising trajectory through interpretable diagnostic and corrective feedback signals.
2) Qualitative Results:: The experiments include an ablation study of SASMA modules on ChronoMagic-Bench-150 with CogVideoX1.5.The supplied passage identifies the study but does not provide its numerical findings.

C. Method Analysis and Ablation … 3) Effect of Evaluation Interval and Stage Scheduling:

The ablations show that SASMA modules improve semantic and perceptual quality, while multi-round feedback and evaluation scheduling determine correction stability and effectiveness. Three-round feedback is adopted for balanced robustness, and interval=5 is selected for full-process evaluation because it balances correction capability, efficiency, and temporal consistency.

1) Ablations on the component of SASMA:: SASMA modules consistently improve semantic and perceptual quality over the standard baseline.Semantic Injection embeds corrective signals into the diffusion trajectory, reducing semantic drift and improving frame-level consistency.
2) Effect of Multi-round Evaluation Strategies:: The study compares one-round, two-rounds, and three-rounds schedules for ordering assessment, prompt polishing, and negative-prompt refinement.The configurations distribute corrective operations differently during inference.
2) Effect of Multi-round Evaluation Strategies:: The two-rounds configuration achieves the best subject consistency and perceptual scores, whereas three-rounds produces higher overall consistency and more balanced temporal and semantic improvements.One-round still improves substantially over baseline but is less effective for refining semantic fidelity.
2) Effect of Multi-round Evaluation Strategies:: Three-round feedback separates prompt enhancement from negative-prompt refinement, reducing conflicts between complementary corrective operations.The strategy first corrects positive semantic guidance before applying negative constraints, although two-rounds achieves marginal gains on specific metrics.
2) Effect of Multi-round Evaluation Strategies:: Three-rounds is adopted by default for superior overall consistency and robustness, with early stopping allowing satisfactory samples to bypass additional correction rounds.The refinement process terminates once satisfactory semantic alignment is detected.
3) Effect of Evaluation Interval and Stage Scheduling:: The scheduling ablation varies early steps 1-24, late steps 25-49, or full steps 1-49 with evaluation intervals 3, 5, and 10 across 50 sampling steps.These settings represent high-frequency, mid-frequency, and lowfrequency scheduling.
3) Effect of Evaluation Interval and Stage Scheduling:: For the early stage, interval=3 performs best for subject consistency, motion smoothness, and aesthetic quality, while late-stage performance remains stable across frequencies.Frequent intervention benefits semantic-structure formation early, whereas excessive late evaluation can introduce unnecessary perturbations.
3) Effect of Evaluation Interval and Stage Scheduling:: For full-process evaluation, interval=5 provides the best balance between correction capability, computational efficiency, semantic correction, and temporal consistency.A fixed mid-frequency strategy achieves comparable effectiveness to dense-to-sparse scheduling with simpler implementation and stable performance across metrics.

4) Effect of MLLM Scale: … 3) MLLM Feedback Quality Analysis:

The analysis shows that MLLM scale, intermediate previewing, and feedback quality jointly affect semantic correction. The 7B model offers the strongest practical balance, while previewing enables earlier and more reliable semantic assessment than raw decoding.

4) Effect of MLLM Scale:: The 72B model leads temporal and semantic consistency, whereas the 7B model surpasses it in aesthetic quality, imaging quality, and overall consistency.The 72B model achieves the highest motion smoothness and subject consistency, while 7B performs better on the listed perceptual metrics.
4) Effect of MLLM Scale:: The 3B model achieves the highest dynamic degree because its more conservative corrections preserve motion dynamics while improving semantic alignment.Larger models provide more detailed and stringent feedback, which can over-constrain generation and suppress motion.
4) Effect of MLLM Scale:: The 7B MLLM achieves the best overall trade-off between semantic correction quality, perceptual quality, and computational efficiency.The 72B model uses approximately 9× the VRAM of 7B, 155GB versus 17GB, while providing only marginal gains on some metrics.
1) Effect of the Intermediate Preview Mechanism:: Intermediate previewing replaces noisy latent decodings with cleaner approximations of the final video, allowing MLLMs to detect objects, attributes, and actions more reliably.Without previews, diffusion noise makes semantic judgments difficult; preview videos provide actionable feedback for subsequent correction.
2) Quantitative Validation of Preview Effectiveness:: Preview-based evaluation exposes semantic structure much earlier than raw decoding during the 50-step sampling process.Raw decoding produces nearly zero MATCH predictions until t/T ≈0.5, while preview reaches 65% at t/T = 0.14 and exceeds 90% after t/T = 0.24.
2) Quantitative Validation of Preview Effectiveness:: Although raw decoding and preview reach agreement rates above 85% at later timesteps, preview provides semantically valid responses at earlier stages.Agreement measures consistency between Qwen2.5-VL-7B and Qwen2.5-VL-72B, while MATCH measures semantic alignment with the prompt.
3) MLLM Feedback Quality Analysis:: Feedback quality is evaluated by generating diagnostic, corrective, and constraint signals with three 7B or 8B MLLMs and verifying them with Qwen2.5-VL-72B.The evaluated generators are Qwen2.5-VL-7B, VideoLLaMA3-7B, and InternVL2.5-8B; Figure 8 compares their feedback quality.

E. Efficiency and Inference Time Analysis … VII. LIMITATIONS

The framework adds moderate inference overhead but uses staged correction and early stopping to limit MLLM calls and amortized cost. It provides training-free semantic control without parameter updates, while remaining partly dependent on reliable MLLM judgments.

E. Efficiency and Inference Time Analysis: Semantic feedback increases runtime and raises peak memory usage to 30.9 GB because the MLLM is loaded during inference.Memory remains constant across configurations.
1) Inference Cost and Early Stopping Analysis:: Multi-round correction has only slightly higher runtime than one-round correction, while introducing more stages reduces the average number of MLLM calls.Progressive correction also reduces over-correction and enables finer-grained adjustments.
1) Inference Cost and Early Stopping Analysis:: Early stopping evaluates semantic alignment at scheduled timesteps and skips later correction operations when the current video already matches the prompt.Otherwise, the corresponding staged operations proceed.
1) Inference Cost and Early Stopping Analysis:: 78.79% of VBench samples converge within three MLLM calls, with an average of 3.168 calls.Additionally, 34.96% terminate after one evaluation and 66.95% stop within two calls.
2) Component-wise Efficiency Breakdown:: Semantic injection accounts for 41.73% of overhead, preview generation for 38.59%, and MLLM inference for 19.68%.These measurements cover 10 semantic evaluation steps on CogVideoX1.5-5B.
VI. CONCLUSION: The training-free framework separates semantic interpretation from trajectory manipulation, using intermediate previews and diagnostic signals for assessment and parameter-free trajectory correction for modification.The design enables precise semantic control and improves semantic accuracy across diverse settings.
VII. LIMITATIONS: The method’s robustness is partly contingent on MLLM assessment quality because prejudice in low-fidelity previews can produce untrustworthy diagnostics and suboptimal corrections.This limitation is especially relevant for certain attributes, object categories, and edge cases.