Source-linked AI summary
MMCORE: MultiModal COnnection with Representation Aligned Latent Embeddings
Zijie Li, Yichun Shi, Jingxiang Sun, Ye Wang, Yixuan Huang, Zhiyao Guo, Xiaochen Lian, Peihao Zhu, Yu Tian, Zhonghua Zhai, Peng Wang
TL;DR
Multimodal generation must combine strong semantic understanding with high-fidelity visual synthesis without sacrificing training efficiency or tightly coupling incompatible modeling paradigms. MMCORE fine-tunes an MLLM to produce semantically aligned visual latent embeddings for a conditioned diffusion model, achieving improvements over state-of-the-art baselines while retaining a decoupled architecture. Its remaining limitations include a performance gap with leading models and incomplete parity with specialized understanding systems.
Problem
Combining autoregressive semantic modeling with diffusion-based image generation creates training-efficiency challenges, while queried embeddings alone have critical limitations.
Method
MMCORE jointly fine-tunes an MLLM to predict compact visual embeddings aligned with vision-language representations, then trains a flow-matching diffusion model conditioned on text and those embeddings.
Results
MMCORE significantly improves image-generation fidelity and instruction adherence over baseline counterparts, particularly in complex multi-turn interleaved generation scenarios.
Takeaways & Limitations
Combining autoregressive semantic modeling with diffusion-based visual synthesis enables high-quality multimodal generation without tightly coupling both paradigms within one network.
Takeaways & Limitations
MMCORE still trails Nano-Banana-pro and GPT image 1.5, and specialized understanding models remain stronger on capabilities such as VQA and OCR.
Abstract
from arXiv · showhide
We present MMCORE, a unified framework designed for multimodal image generation and editing. MMCORE leverages a pre-trained Vision-Language Model (VLM) to predict semantic visual embeddings via learnable query tokens, which subsequently serve as conditioning signals for a diffusion model. This streamlined design effectively transfers the rich understanding and reasoning capabilities of VLMs into the visual generation process. By obviating the need for deep fusion between autoregressive and diffusion models or training from scratch, MMCORE significantly reduces computational overhead while maintaining high-fidelity synthesis. MMCORE seamlessly integrates text-to-image synthesis with interleaved image generation, demonstrating robust multimodal comprehension in complex scenarios such as spatial reasoning and visual grounding. Comprehensive evaluations indicate that MMCORE consistently outperforms state-of-the-art baselines across a broad spectrum of text-to-image and single/multi-image editing benchmarks.
1 Introduction
MMCORE addresses the efficiency challenge of combining autoregressive semantic modeling with diffusion-based visual generation by using aligned visual latent embeddings. It reports consistent improvements over state-of-the-art baselines while keeping the two paradigms decoupled.
- Diffusion-based visual modeling reduces training efficiency because understanding uses clean features while generation uses noisy features.
- Autoregressive visual generators remain behind diffusion models in generation quality, while prior connector-based methods decouple semantic modeling from visual synthesis.
- MMCORE fine-tunes an MLLM to produce compact visual embeddings aligned with vision-language representations, then conditions a flow-matching diffusion model on text and those embeddings.The two-stage design removes explicit connector modules and supports later supervised fine-tuning and reinforcement learning.
- Across loss configurations, batch sizes, and image resolutions, latent visual embeddings consistently improve performance over state-of-the-art baselines.The reported gains are attributed to richer visual understanding than text-only conditioning.
- MMCORE combines autoregressive semantic modeling with diffusion-based synthesis without tightly coupling both paradigms inside one network.
- The current architecture uses separate visual encoders for image understanding and generation, leaving unified sequentialized image-latent processing as future work.The proposed future direction aims to let both tasks share a single forward pass.
2 Related Work
Related work combines multimodal understanding with visual generation through unified backbones, separate branches, or frozen multimodal models connected to diffusion decoders. MMCORE builds on representation transfer and lightweight alignment to connect pretrained multimodal representations with diffusion generation.
- Unified multimodal models support visual understanding, generation, editing, and downstream tasks through either shared backbones or separate understanding and generation branches.
- BLIP3-o, MetaQueries, and UniWorld couple frozen multimodal LLMs with trainable diffusion decoders to construct efficient unified multimodal models.
- MMCORE uses an MLLM to infer semantic visual latent embeddings from multimodal and query inputs, which condition a causal diffusion network for pixel-level understanding and generation.
- Some unified models duplicate LLM weights and require orders of magnitude more training tokens and extensive private data, unlike approaches trained from public checkpoints and data.
- Cross-modal alignment maps heterogeneous visual and textual representations into a shared semantic space for understanding and generation.Contrastive dual encoders and later multimodal transformers are major directions in this area.
- Recent work reduces computational costs through representation transfer and lightweight alignment for pretrained models, including transferring multimodal representations to diffusion decoders.
3 Approach
MMCORE aligns a fine-tuned multimodal language model with a diffusion generator through semantic visual tokens, combining full-sequence text conditioning with learned visual queries. Its staged design improves generation and editing while reducing training cost, though current visual pathways remain separate.
- Architecture: MMCORE uses an MLLM to infer semantic visual latent embeddings from multimodal and query inputs, while a causal diffusion network handles pixel-level understanding and generation.Target images provide both pixel-level diffusion supervision and semantic-level supervision for visual-token learning.
- Architecture: MetaQueries distills multimodal information into fixed-size query representations, but fixed budgets can miss complex context and add redundancy or compute.Small query budgets lose prompt nuances; large budgets introduce redundancy and computational overhead.
- Training modifications: MMCORE jointly fine-tunes the MLLM backbone on understanding and generation data instead of freezing it behind lightweight adapters.The authors note a minor regression in general MLLM understanding capabilities and attribute it to curriculum scheduling rather than an intrinsic architectural limitation.
- Training modifications: Semantic visual alignment distills frozen ViT or SigLIP features into generated query tokens, providing denser and more stable supervision than diffusion loss alone.The visual tokens are regressed toward semantically dense visual embeddings from a frozen vision encoder.
- Training modifications: Dual-pathway conditioning retains full-sequence text embeddings alongside visual query tokens, assigning global semantics and cross-modal grounding to the visual pathway.This design addresses the information bottleneck imposed by a fixed number of query tokens.
- Training strategy: The two-stage pipeline trains the MLLM independently before diffusion-head training, while brief SFT improves text faithfulness and controllability.Adding flow matching to MLLM training yields only marginal improvements while significantly increasing training costs; approximately 2K SFT steps are reported as beneficial.
- Efficiency and results: ∼30% of the computational budget achieves performance parity with native unified architectures such as Transfusion and BAGEL.The approach uses meta-query alignment and continued pre-training rather than training a native unified architecture from scratch.
- Limitations: Current architecture separates image understanding and generation through distinct visual encoders, leaving unified sequentialized image-latent tokens as future work.The proposed unification aims to let both processes share a single forward pass and improve compactness and efficiency.
4 Results
MMCORE demonstrates strong text-to-image generation and image-editing performance, including complex reasoning, spatial control, and multi-image composition. Ablations show that high-capacity alignment, larger batches, and targeted SFT improve alignment, while adding visual latents to conditional VAE features can hurt editing performance.
- Text-to-Image Generation: MMCORE significantly outperforms Seedream 4.0 and other state-of-the-art baselines on automatic prompt-image coherence evaluation.The benchmark evaluates text-to-image generation and image editing using an automated pipeline.
- Image Editing: MMCORE improves instruction alignment and consistency in single- and multi-image editing through dual-pathway conditioning.The additional visual representation complements text and metadata conditioning used by prior diffusion-based methods.
- Human Evaluation: MMCORE delivers consistent human-evaluation improvements in prompt-image alignment, structural and visual fidelity, and editing consistency.Editing consistency includes preserving identity and background, and the results support the fidelity of the automated metrics.
- Qualitative Analysis: MMCORE better handles counterfactual prompts, asymmetric relations, literalized non-literal phrases, localized details, and dense structured specifications than Seedream 4.0.These examples indicate stronger compositional role binding, attribute grounding, text fidelity, spatial reasoning, and constraint satisfaction.
- Qualitative Analysis: MMCORE performs precise spatial and geometric edits while preserving non-target content, and remains controllable with more than 10 input images.The model covers single-image semantic changes and complex multi-image compositions.
- Ablation Study: 0.8585 GPT-4o and 0.8915 Doubao alignment scores follow a short 2k-step SFT phase, compared with performance plateauing around 0.82 after extended pre-training.A 6-layer connector also improves performance by +10.5% over a shallow connector, while increasing batch size 5× raises the GPT-4o score from 0.8114 to 0.8199.
5 Conclusion and Limitations
MMCORE offers a resource-efficient framework that aligns MLLM reasoning with latent visual representations, improving generation fidelity and instruction adherence while retaining important efficiency benefits. The authors identify remaining gaps in MLLM quality, understanding-task parity, and visual-latent completeness.
- Conclusion: MMCORE aligns MLLM long-context understanding and reasoning with latent visual representations through a resource-efficient framework.The approach uses trainable queries and dual-pathway conditioning rather than prohibitive end-to-end retraining.
- Conclusion: MMCORE significantly improves image-generation fidelity and instruction adherence over baseline counterparts, particularly in complex multi-turn interleaved generation.
- Limitations: MMCORE still trails Nano-Banana-pro and GPT image 1.5, with the authors attributing this gap potentially to differences in the underlying MLLM.
- The Understanding-Generation Trade-off: Generative alignment can degrade pure understanding capabilities such as VQA and OCR, although joint training mitigates catastrophic forgetting without reaching specialized-model parity.
- Redundancy in Visual Latents: Learned visual tokens remain semantically redundant because they cannot independently support high-fidelity generation without the auxiliary ViT encoder and diffusion decoder.
- Redundancy in Visual Latents: Unifying understanding and generation at the representation level through an Omni-Tokenizer is identified as a primary direction for future work.
A Ethical Claims
The paper states that its images come from licensed sources and public license-free websites, and that the technique is intended for common industry tasks rather than unwanted violent or sexual content.
- Ethical Claims: The paper uses licensed images and images from public license-free websites including Unsplash and Pixabay.
- Ethical Claims: The authors caution that the technique should not be applied to unwanted violent or sexual content and may inherit biases and limitations from text-to-image models.