Source-linked AI summary
Self-correction Optimization for Interleaved Multimodal Generation
Xin You, Zhiwei Ning, Zukai Chen, Minghui Zhang, Xuanke Shi, Hanxiao Zhang, Jingsong Liu, Jie Yang, Quan Wang, Yun Gu
TL;DR
Interleaved generation remains difficult because systems must maintain temporal and visual-subject consistency, while existing approaches can require costly additional training. SCO minimally corrects the classifier-free guidance update using new-event and state-preserving constraints; experiments report improvements across interleaved image–text benchmarks and physically grounded video-generation scenarios.
Problem
Interleaved generation systems may require computationally expensive modeling, data, or architecture and can fail to correct conflicts between current instructions and inherited visual states.
Source: §1 · p. 1Method
SCO applies bounded minimal self-correction to the original CFG update by jointly optimizing complementary new-event and state-preserving constraints.
Source: §1 · p. 1, §3.4 · p. 5Results
SCO improves visual quality, cross-modal alignment, and cross-image consistency across modular and unified generators, and improves temporal consistency in handcrafting and robot-manipulation video scenarios.
Source: §5 · p. 10Takeaways & Limitations
The findings support explicitly balancing event realization and state preservation in long-horizon multimodal generation.
Source: §5 · p. 10
Abstract
from arXiv · showhide
Multimodal large language models (MLLMs) have made significant progress in visual understanding and generation. However, generating interleaved image--text content remains challenging, as it requires tightly integrated multimodal understanding and generation capabilities. Although existing MLLMs provide promising solutions, most rely on additional training with augmented data, which is computationally expensive and remains limited in preserving visual subjects, temporal consistency, and physical plausibility. In this work, we propose self-correction optimization (SCO), an effective training-free method for consistent interleaved generation. SCO treats the classifier-free guidance update as a reference and performs minimal self-correction under two complementary constraints, including new-event and state-preserving constraints. Specifically, the new-event constraint promotes temporal consistency across image--text sequences, while the state-preserving constraint maintains the coherence of visual subjects throughout subsequent generation steps. Experiments on challenging interleaved multimodal generation benchmarks demonstrate significant improvements in temporal coherence and visual-subject preservation. Furthermore, SCO can be extended to video generation and improves the modeling of physically grounded processes, including robot manipulation and long-horizon handcrafting.
1 INTRODUCTION
Interleaved generation must coordinate new events with visual states carried forward across turns, but existing approaches add computationally expensive modeling, data, or architecture and may leave instruction conflicts uncorrected. SCO applies test-time self-correction to the CFG update using complementary constraints, with reported gains on interleaved benchmarks and physically grounded video tasks.
1 INTRODUCTION
- Interleaved image–text generation produces text and images in arbitrary order while conditioning on an evolving multimodal history, supporting applications such as storytelling, editing, planning, and embodied reasoning.The broader generation paradigm builds on progress in MLLM visual understanding and generation.Source: §1 · p. 1
- Existing systems use multimodal context modeling, additional training data, or architectural designs that are computationally expensive and may not correct conflicts between current instructions and inherited visual states.Insufficient event realization or subject drift and scene reconfiguration can accumulate across rounds and compromise procedural tasks.Source: §1 · p. 1
- SCO constrains the CFG update so the current image realizes the instructed event and its direct visual consequences while retaining prior subject identity, scene layout, and rendering style.The constraints are jointly optimized through a bounded two-dimensional projection, with independent correction budgets allowing controlled relaxation when they compete.Source: §1 · pp. 1–2, Figure 1 · p. 2
- SCO improves visual quality, cross-modal alignment, and cross-image consistency across modular and unified generators, and extends to video generation for handcrafting and robot-manipulation scenarios.The method jointly balances event realization and state preservation through a bounded minimal correction of the original CFG update.Source: §1 · p. 2
2 RELATED WORK
Prior work addresses interleaved generation through modular systems or unified multimodal models, while training-free guidance methods offer another route to controllability. These approaches leave constraints that motivate SCO's focus on cross-modal consistency and temporal coherence.
2.1 MODULAR INTERLEAVED GENERATION MODELS
- Modular systems connect autoregressive language models to pretrained visual generators, but their architectures restrict them to interleaved generation.Examples include GILL, MiniGPT-5, MM-Interleaved, and OpenLEAF; MM-Interleaved adds multi-scale and multi-image feature synchronization.Source: §2.1 · pp. 2–3
2.2 UNIFIED MULTIMODAL MODELS FOR INTERLEAVED GENERATION
- Unified multimodal models share an architecture for language and visual content, but may over-rely on recent visual states and accumulate deviations that cause subject drift or incorrect states.SCO is proposed to optimize complementary event and state-preserving constraints without task-specific training.Source: §2.2 · p. 3
2.3 TRAINING-FREE GENERATION GUIDANCE
- Classifier-free guidance combines conditional and unconditional predictions, but its single scalar scale offers limited control over multiple semantic objectives.SCO follows the training-free guidance paradigm while targeting cross-modal consistency and temporal coherence in interleaved generation.Source: §2.3 · p. 3
3 METHODOLOGY
The methodology frames interleaved generation as balancing a current event against visual information inherited from multimodal history. SCO minimally corrects the reference CFG update with complementary constraints, jointly scheduled and optimized.
3.1 PROBLEM FORMULATION AND MOTIVATION
- Each image is conditioned on the complete multimodal history and current instruction, but the original CFG update entangles the event with inherited visual state.This can underrealize the event or progressively change subject identity and scene layout.Source: §3.1 · p. 3
3.2 NEW-EVENT CONSTRAINT
- The new-event direction subtracts the same-history counterfactual update without the current event from the fully conditioned update.SCO activates correction when the reference CFG update provides insufficient directional support, promoting the requested event and its direct consequences.Source: §3.2 · p. 4
3.3 STATE-PRESERVING CONSTRAINT
- The state-preserving direction estimates inherited state by contrasting the fully conditioned update with an event-only update that omits history.It preserves subject identity, scene layout, and rendering style in generator-update space rather than requiring pixel-level correspondence, reducing drift and inconsistency across image slots.Source: §3.3 · p. 4
3.4 JOINT SELF-CORRECTION OPTIMIZATION
- SCO jointly corrects the reference CFG update through a bounded projection that accounts for interaction between the two constraint directions.Constraint strengths are scheduled over sampling; the two-coefficient convex optimization uses independent correction bounds. The procedure runs at test time and extends to frame-indexed video generation.Source: §3.4 · pp. 4–5
4 EXPERIMENTS
Across OpenING and ISG-Bench, SCO generally improves visual quality, cross-modal alignment, and cross-image consistency. Ablations show complementary constraints with a trade-off between event realization and subject consistency, while video experiments report gains across handcrafting and robot manipulation.
4.1 DATASETS
- OpenING has 5,400 annotated instances across 23 meta-topics and 56 tasks; ISG-Bench has 1,150 samples across 8 scenarios and 21 subtasks.ISG-Bench samples use interleaved scene graphs specifying entities, attributes, and relations.Source: §4.1 · pp. 5–6
4.3 EXPERIMENTAL RESULTS
- Qualitative examples report improvements on iterative image editing, novel-view synthesis, and embodied-AI scenarios when SCO is applied to SenseNova-U1.Source: Figure 2 · p. 6, §4.3 · p. 8
- Across ISG-Bench and OpenING, SCO generally improves visual quality, cross-modal alignment, and cross-image consistency for modular and unified models.The reported results cover both benchmarks and model types.Source: Table 1 · p. 7, Table 2 · p. 7, §4.3 · p. 7
4.2 IMPLEMENTATION DETAILS AND EVALUATION METRICS
- Evaluation covers image quality, cross-modality consistency, temporal coherence, factual consistency, and physical plausibility using image, embedding, and judge-based metrics.The metrics include FID, CLIP-I, CLIP-T, GPT-J1, Cs, Cb, and GPT-J2.Source: §4.2 · p. 7
4.4 COMPARISON WITH CFG-BASED METHODS
- 16.8% FID reduction; SCO also raises CLIP-I, CLIP-T, and GPT-J2 by 0.051, 0.036, and 0.028 versus SenseNova-U1 on OpenING.Source: Table 3 · p. 8, §4.4 · p. 8
4.5 HYPER-PARAMETER ANALYSIS
- Moderate constraint settings work best: α0 = 1.0, Mnew = 6, sstate approximately 0.7–0.9, Mstate around 10, and τend = 0.5.Weak constraints risk visual drift; overly strong or prolonged constraints can suppress valid changes or image details.Source: §4.5 · p. 8, Figure 3 · p. 8, Figure 4 · p. 9
4.6 ABLATION STUDY ON KEY COMPONENTS
- Jointly optimizing both constraints gives the best overall FID, CLIP-I, GPT-J1, and GPT-J2, while state-preserving-only yields slightly higher Cs.The results expose a trade-off between new-event realization and subject consistency.Source: Table 4 · p. 9, §4.6 · p. 9
4.7 GENERALIZATION ON VIDEO GENERATION
- Across handcrafting and robot-manipulation videos, SCO reduces FVD and LPIPS while improving Cs, Cb, and GPT-J2 across three generators.The reported video evaluation uses VideoWorld2, Cosmos-2.5, and Cosmos-3.Source: Table 5 · p. 9, §4.7 · p. 10
5 CONCLUSION
SCO is a training-free test-time method that improves interleaved image–text generation and extends to physically grounded video tasks. The conclusion reports broad improvements while emphasizing a balance between event realization and state preservation.
5 CONCLUSION
- SCO improves visual quality, cross-modal alignment, and cross-image consistency across modular and unified generators, and extends to video generation for handcrafting and robot manipulation.The conclusion identifies balancing event realization and state preservation as important in long-horizon multimodal generation.Source: §5 · p. 10
AI USE STATEMENT
- The authors report using generative AI only for manuscript readability edits and LaTeX formatting, not for data, experiments, implementation, or result interpretation.They state that all technical claims, results, figures, and tables were produced and verified by the authors.Source: AI USE STATEMENT · p. 10
REPRODUCIBILITY STATEMENT
- The reproducibility statement says the theory is presented, prompt-rewriting details are in the appendix, and hyperparameters are listed in the manuscript.The statement says these operations ensure reproducibility.Source: REPRODUCIBILITY STATEMENT · p. 10
A.1 EXTENSION TO VIDEO GENERATION
The video extension applies SCO to frame-indexed events and temporal contexts. A planner assigns events to frames, after which frame-dependent updates preserve both event-specific motion and persistent context.
A.1 EXTENSION TO VIDEO GENERATION
- A GPT-5-based planner decomposes the prompt into temporally ordered events and allocates the target frames among them before SCO is applied.Each frame is assigned to the event stage whose allocated interval contains it.Source: §A.1 · p. 14
- For each frame, SCO uses the event assignment and inherited temporal context to construct frame-aligned conditions and frame-indexed outputs.The temporal state is defined relative to the event assigned to that frame.Source: §A.1 · p. 14
- Although the CFG scale is shared, updates remain frame-dependent because event assignments and temporal contexts vary across frames.For jointly generated video, the update is a temporal component of a shared spatiotemporal update, not an independently sampled frame.Source: §A.1 · p. 14
- The new-event direction captures motion or physical transitions, while the state-preserving direction retains information from preceding temporal context.The bounded joint optimization also applies to diffusion- and flow-matching-based video generators while preserving cross-frame interactions.Source: §A.1 · p. 14
A.2 DATASET DETAILS
The evaluation spans open-ended and structured interleaved image–text benchmarks, long-horizon video data, and simulated language-guided manipulation. The supplied figure caption identifies qualitative results for embodiment and handcrafting.
A.2 DATASET DETAILS
- OpenING contains 5,400 human-annotated instances across 23 meta-topics and 56 fine-grained tasks, evaluating complete open-ended interleaved image–text responses.Its evaluation includes helpfulness, text and visual quality, text–image alignment, and coherence across generated images.Source: §A.2 · p. 14
- ISG-Bench contains 1,150 samples across 8 scenarios and 21 subtasks, with scene graphs supporting fine-grained checks of entities, attributes, relations, and requirements.It evaluates structural, block-level, image-level, and holistic response consistency.Source: §A.2 · p. 14
- Figure 6 presents qualitative results for SCO in embodiment and handcrafting fields.The caption identifies the application areas but does not state a specific comparison outcome.Source: Figure 6 · p. 15
- Video-CraftBench’s handcraft-making portion contains five tasks and approximately 9.5K first-person video clips collected from about seven hours of demonstrations, suitable for evaluating event realization and state preservation.The experiments use a subset of Interleaved X-Embodiment; the full collection contains approximately 210K robot-manipulation trajectories, and video experiments report event-oriented and preservation-oriented metrics.Source: §A.2 · pp. 15–16
- Language-Table evaluates simulated language-guided tabletop manipulation by generating future video frames from an initial observation and a short robot-action sequence.Its requested event is grounded in a physical state transition that must remain consistent with the robot trajectory.Source: §A.2 · p. 16
A.3 DETAILS OF EVALUATION METRICS
Evaluation combines reference-based metrics and GPT-based judgments across image quality, cross-modal consistency, temporal coherence, and physical plausibility. Absolute scores are comparable only within the same GPT-Judge run.
A.3 DETAILS OF EVALUATION METRICS
- The evaluation covers four dimensions with reference-based metrics and GPT-based judgments; absolute scores should be compared only within the same GPT-Judge run.GPT-based judgments use GPT-5.6-sol with fixed sampling settings.Source: §A.3 · pp. 16–17
- FID measures the feature-space distribution gap between generated and real images, with lower values indicating closer distributions; CLIP-I measures prediction-to-ground-truth-frame image similarity.CLIP-I is computed as cosine similarity between CLIP image embeddings.Source: §A.3 · p. 16
- CLIP-T measures prompt-to-generated-image semantic alignment, while GPT-J1 judges whether each frame reflects its text description and is factually and logically plausible.GPT-J1 scores each frame from 0 to 1 along those two axes.Source: §A.3 · p. 16
- Subject consistency uses DINOv2 feature similarity across consecutive frames, while background consistency uses CLIP image-embedding similarity across frame backgrounds.The subject metric tracks identity-preserving features; the background metric assesses low-level scene stability.Source: §A.3 · p. 16
- GPT-J2 gives a holistic 0–1 rating of fact consistency and physical plausibility, including object identities, causal relations, and common-sense dynamics.It uses the same GPT-5.6-sol backbone as GPT-J1.Source: §A.3 · p. 17
A.4 PROMPT CONSTRUCTION FOR SCO
SCO prompt construction combines persistent visual history with the newly requested event, then uses complementary rewrites to isolate event-related changes and identify state to preserve.
A.4 PROMPT CONSTRUCTION FOR SCO
- The full-task prompt combines persistent visual state from prior rounds with the newly requested event.A counterfactual prompt removes only the current event to isolate its visual change; an event-only prompt removes historical details to identify persistent state.Source: §A.4 · p. 17
- The event-only rewrite retains the current action while removing historical details such as subject identity and appearance.The comparison with the full condition identifies persistent state to preserve.Source: §A.4 · pp. 17–18
- In the illustrated example, the rewrites distinguish cutting an apple from the previously established woman, clothing, kitchen, table, and object states.Source: §A.4 · p. 18
A.5 ADDITIONAL QUALITATIVE RESULTS
Qualitative results show SCO balancing new motion with visual identity in storytelling, preserving scene content through iterative edits, and maintaining remaining objects across sequential removals.
A.5 ADDITIONAL QUALITATIVE RESULTS
- Combined constraints preserve the turtle’s visual identity while producing richer forward and upward motion, scale variation, and scene evolution.State preservation alone yields relatively conservative motion, while the new-event constraint produces more pronounced motion and viewpoint changes but gradually alters appearance.Source: §A.5 · p. 18
- Compared with the baseline, SCO better preserves window structure, spatial layout, and scene content through successive brightness and saturation edits.Source: §A.5 · p. 18, Figure 9 · p. 19
- Across sequential object-removal examples, SCO preserves the remaining objects’ identity, appearance, and spatial configuration while maintaining background and composition.It also reconstructs regions previously occluded by removed objects and supports cumulative removal instructions over multiple rounds.Source: §A.5 · pp. 18–19, Figure 10 · p. 20