Source-linked AI summary
Transfer between Modalities with MetaQueries
Xichen Pan, Satya Narayan Shukla, Aashu Singh, Zhuokai Zhao, Shlok Kumar Mishra, Jialiang Wang, Zhiyang Xu, Jiuhai Chen, Kunpeng Li, Felix Juefei-Xu, Ji Hou, Saining Xie
TL;DR
Unified multimodal models must connect textual understanding with pixel generation, but doing so within one architecture can require complex training. MetaQueries use learnable queries to connect autoregressive MLLMs with diffusion decoders, and the approach remains effective with frozen MLLMs while supporting strong understanding, generation, and advanced generation tasks.
Problem
Connecting autoregressive MLLM understanding with image generation remains challenging, particularly for transferring latent world knowledge into image generators.
Method
MetaQueries feed learnable queries into a frozen MLLM, transform its extracted conditions through a connector, and train a conditional diffusion model with paired data and the original denoising objective.
Results
The framework preserves SOTA multimodal understanding while achieving SOTA-level multimodal generation and transferring MLLM knowledge and reasoning into image generation.
Takeaways & Limitations
The interface provides a simpler way to transfer capabilities between pretrained understanding and generation models, with instruction-tuning support for image editing and subject-driven generation.
Abstract
from arXiv · showhide
Unified multimodal models aim to integrate understanding (text output) and generation (pixel output), but aligning these different modalities within a single architecture often demands complex training recipes and careful data balancing. We introduce MetaQueries, a set of learnable queries that act as an efficient interface between autoregressive multimodal LLMs (MLLMs) and diffusion models. MetaQueries connects the MLLM's latents to the diffusion decoder, enabling knowledge-augmented image generation by leveraging the MLLM's deep understanding and reasoning capabilities. Our method simplifies training, requiring only paired image-caption data and standard diffusion objectives. Notably, this transfer is effective even when the MLLM backbone remains frozen, thereby preserving its state-of-the-art multimodal understanding capabilities while achieving strong generative performance. Additionally, our method is flexible and can be easily instruction-tuned for advanced applications such as image editing and subject-driven generation.
1 Introduction
MetaQueries bridge frozen multimodal LLMs and diffusion models, transferring understanding, knowledge, and reasoning into image generation without requiring MLLM fine-tuning. The framework uses learnable queries and paired image-caption data, while retaining strong understanding and generation performance.
- Motivation and approach: MetaQueries connect frozen MLLM backbones to diffusion models, assigning understanding to the MLLM and generation to the diffusion decoder.The design transfers capabilities between pretrained models specialized for different output modalities.
- Motivation and approach: Learnable queries extract multimodal conditions from frozen MLLMs and pass them through a connector into conditional diffusion models.Training uses the original denoising objective on paired image-caption data, with causal masking retained throughout the sequence.
- Results: 25M publicly available image-caption pairs support models that preserve SOTA image understanding and achieve SOTA-level text-to-image generation across multiple benchmarks.The framework also matches fully tuned MLLM performance while being significantly more efficient.
- Results: MetaQuery transfers MLLM reasoning and knowledge to image generation, achieving SOTA visual-commonsense generation and reasoning performance on CommonsenseT2I and WISE.The method supports complex prompts requiring the MLLM to process and answer questions before generating images.
- Capabilities: MetaQuery extracts detailed visual conditions beyond semantic similarity, enabling image reconstruction, editing, subject-driven generation, visual association, and logo design.The generative diffusion model can remain frozen or be instruction-tuned for advanced generation tasks.
2 Related Work
Related work explores extending autoregressive multimodal understanding models toward generation and using frozen LLMs in multimodal systems. These approaches differ in how deeply they fuse conditions and how flexibly they support architectures and training.
- Unified understanding and generation models: Unified multimodal models extend next-token prediction from language and multimodal understanding toward jointly generating text and images.The cited work frames this as training LLM backbones to generate images alongside multimodal content.
- Unified models with frozen LLMs: LMFusion trains image-generation FFNs and QKV modules alongside a frozen LLM to fuse input conditions and denoise visual outputs.The approach is described as having limited flexibility because it shares the specific LLM backbone architecture and requires additional training.
3 MetaQuery
MetaQuery uses learnable queries and a connector to transfer conditions from a frozen MLLM into a diffusion model. This design preserves understanding capabilities while providing efficient multimodal generation.
- 3.2 Design Choices: Learnable queries with N = 64 tokens achieve image-generation quality comparable to using all hidden states and can surpass them with more tokens.This approach also remains compatible with unified-model tasks such as in-context learning and interleaved multimodal output.
- 3.1 Architecture: MetaQuery uses randomly initialized learnable queries to extract conditions from a frozen MLLM for multimodal generation.A trainable connector aligns these conditions with a diffusion model’s conditional input space.
- 3.2 Design Choices: Frozen MLLMs preserve multimodal understanding without the degradation associated with jointly training text and pixel generation.Compared with full MLLM tuning, frozen backbones achieve comparable generation, with slightly lower prompt alignment but slightly improved visual quality.
- 3.3 Training Recipe: The Enc-Proj connector aligns conditions in the MLLM hidden-state dimension before projection, making it more effective and parameter-efficient.Proj-Enc instead projects conditions before transformer alignment.
- 3.2 Design Choices: As token counts increase, visual quality converges after 64 tokens while prompt alignment and image-reconstruction performance continue improving.The prompt-alignment gains are more pronounced for long captions.
4 Model Training
MetaQuery training uses frozen MLLM backbones together with trainable queries, connectors, and diffusion models in pre-training and instruction tuning. Its instruction data are curated from naturally occurring web image pairs rather than generated targets.
- 4 Model Training: MetaQuery is trained in pre-training and instruction-tuning stages while keeping the MLLM backbones frozen.The Base, Large, and X-Large variants use different frozen MLLMs, with N = 256 queries and a 24-layer Enc-Proj connector.
- 4 Model Training: Pre-training uses 25M publicly available image-caption pairs for 8 epochs with cosine-decayed learning rate and 4,000-step warmup.The initial learning rate is 1e-4, decaying to 1e-5.
- 4 Model Training: The reported benchmark table combines COCO FID measured with Stable Diffusion v1.5 and other metrics measured with Sana.Rewritten prompts and results tested under identical settings are marked separately.
- 4 Model Training: Instruction-tuning data are built from naturally occurring image pairs in web corpora instead of expert-generated target images.The pipeline groups images by caption similarity and constructs instruction data from the resulting pairs using an MLLM.
5 Experiments
Experiments show that MetaQuery supports strong understanding and generation, reconstruction, editing, subject-driven generation, and knowledge-augmented image generation. Frozen MLLMs provide useful reasoning and in-context capabilities through the MetaQuery interface.
- 5.1 Image Understanding and Generation: MetaQuery achieves competitive understanding across model sizes and SOTA visual quality on MJHQ-30K.It also beats diffusion-based approaches on GenEval and DPG-Bench, while differing from autoregressive models in failure modes.
- 5.2 Image Reconstruction: MetaQuery-B matches Emu2 and achieves comparable quality to GPT-4o on image reconstruction.This result is obtained after fine-tuning with a frozen MLLM.
- 5.3 Image Editing: Image reconstruction transfers to image editing after fine-tuning the pretrained Base model for only 1,000 steps.The MLLM backbone remains frozen during this transfer.
- 5.4 Subject-Driven Generation: Instruction-tuned MetaQuery-B achieves zero-shot subject-driven generation with multiple customized subjects and unlocks visual association and logo design.The instruction-tuning dataset contains 2.4M examples, and the model achieves SOTA performance on DreamBench.
- 5.5 Knowledge-Augmented Generation: MetaQuery achieves SOTA performance on WISE and CommonsenseT2I by using frozen MLLM knowledge, reasoning, and in-context learning.Compared with last-layer embeddings, native integration with the LLM supports processing questions before generating images.
- 5.6 Discussion: Instruction tuning improves multimodal understanding, but those improvements are orthogonal to image-generation performance when used to provide generation conditions.This comparison is reported across pre-trained LLM, instruction-tuned LLM, and instruction-tuned MLLM backbones.
6 Conclusion
MetaQueries provide a simple interface between MLLMs and diffusion decoders that remains effective with frozen MLLMs. The authors position the method as an accessible baseline for unified multimodal model development while hypothesizing that further data scaling may narrow the gap to leading proprietary systems.
- 6 Conclusion: MetaQueries connect MLLMs for understanding with diffusion decoders for generation, transferring MLLM knowledge and reasoning into multimodal generation.The interface remains effective when the MLLM is frozen and yields state-of-the-art understanding and generation performance.
- 6 Conclusion: The authors hypothesize that further data scaling may primarily help bridge the remaining gap to leading proprietary systems.They present MetaQueries as a powerful, accessible baseline for future unified multimodal model development.
A Data Curation Details
The data curation process uses an MLLM to create concise prompts from source images, a target image, and their captions. These prompts preserve shared similarities and target-only differences without independently specifying the target.
- Qwen/Qwen2-VL-7B-Instruct2 serves as the MLLM for data curation.
- The curation input includes one or more source images, one target image, and their captions.
- Prompts include a general similarity shared with the sources and all differences unique to the target.
- Prompts exclude target-specific details that would permit independent generation and are combined with source images during generation.
B Qualitative Comparison with SOTA Open-Source Model on Text-to-Image Generation
The paper qualitatively compares MetaQuery-XL with Janus-Pro-7B on MJHQ-30K. MetaQuery-XL follows prompts better and produces more visually appealing images in the reported comparison.
- The comparison is presented qualitatively in Figure 10.
- MetaQuery-XL follows the prompt better and generates more visually appealing images than Janus-Pro-7B.
C Training Objectives
The paper studies text-to-image and image-reconstruction training objectives for aligning an MLLM with image generation. Text-to-image training performs better, while mixing objectives adds reconstruction capability without generally harming text-to-image performance.
- The system uses an MLLM for multimodal perception alongside a standard text-to-image objective.
- Training with the text-to-image objective achieves much better performance than training with the image-reconstruction objective.
- Mixing image-reconstruction and text-to-image objectives enables image-reconstruction capabilities without generally harming text-to-image performance.