Source-linked AI summary
Harmonizing Visual Representations for Unified Multimodal Understanding and Generation
Size Wu, Wenwei Zhang, Lumin Xu, Sheng Jin, Zhonghua Wu, Qingyi Tao, Wentao Liu, Wei Li, Chen Change Loy
TL;DR
Unified visual understanding and generation require representations at different granularities, and VQ/VAE-based unified approaches prioritize intrinsic image features over semantics. Harmon uses a shared MAR encoder with three-stage training to harmonize both tasks, achieving strong generation benchmarks and competitive understanding performance, while its MAR pre-training data scale limits further understanding gains.
Problem
Visual understanding and generation require different representation granularities, while VQ/VAE-based unified encoders prioritize intrinsic image features over visual semantics and have limited understanding capability.
Method
Harmon uses a shared MAR-based visual encoder with three-stage training to support masked autoregressive image generation and next-token text prediction.
Results
Harmon surpasses similarly scaled unified methods on GenEval and WISE, achieves state-of-the-art performance on MJHQ30K, and competes with Janus and Janus-Pro on image understanding benchmarks.
Takeaways & Limitations
A shared MIM-trained MAR representation can support both visual generation and understanding, with co-training also boosting generation performance.
Takeaways & Limitations
Harmon is limited to 1.5B parameters, and its MAR encoder was pre-trained on 1.2M ImageNet1K samples, hindering further understanding improvements.
Abstract
from arXiv · showhide
Unifying visual understanding and generation within a single multimodal framework remains a significant challenge, as the two inherently heterogeneous tasks require representations at different levels of granularity. Current approaches that utilize vector quantization (VQ) or variational autoencoders (VAE) for unified visual representation prioritize intrinsic imagery features over semantics, compromising understanding performance. In this work, we take inspiration from masked image modelling (MIM) that learns rich semantics via a mask-and-reconstruct pre-training and its successful extension to masked autoregressive (MAR) image generation. A preliminary study on the MAR encoder's representation reveals exceptional linear probing accuracy and precise feature response to visual concepts, which indicates MAR's potential for visual understanding tasks beyond its original generation role. Based on these insights, we present \emph{Harmon}, a unified autoregressive framework that harmonizes understanding and generation tasks with a shared MAR encoder. Through a three-stage training procedure that progressively optimizes understanding and generation capabilities, Harmon achieves state-of-the-art image generation results on the GenEval, MJHQ30K and WISE benchmarks while matching the performance of methods with dedicated semantic encoders (e.g., Janus) on image understanding benchmarks. Our code and models will be available at https://github.com/wusize/Harmon.
1. Introduction
Harmon addresses the mismatch between visual understanding and generation by using a shared MAR-based representation that captures semantics and fine-grained image features. Its three-stage training framework achieves strong performance across both task types.
- Motivation: Unified multimodal models must reconcile coarse-grained representations for understanding with fine-grained features needed for generation.VQ and VAE encoders primarily preserve intrinsic image features for pixel-level reconstruction, limiting their semantic understanding capabilities.
- Motivation: MIM and MAR offer a potential solution because mask-and-reconstruct training can develop richer semantics while MAR supports autoregressive image generation.MAR also aligns with the next-token prediction paradigm used in multimodal language models.
- Preliminary study: MAR encoders show competitive linear probing accuracy and precise feature activations for high-level visual concepts, unlike VQGAN and VAE encoders.The preliminary study uses linear probing and Grad-CAM++ feature visualization to assess representation quality.
- Proposed framework: Harmon uses a shared MAR-based visual encoder to capture both coarse semantics and fine-grained features for image generation and understanding.Images are generated with masked autoregression, texts with next-token prediction, and a three-stage pipeline aligns the visual encoder with the LLM language space.
- Results: Harmon surpasses similarly scaled unified methods on GenEval and WISE, reaches state-of-the-art visual quality on MJHQ30K, and competes with Janus and Janus-Pro for understanding.The experiments also observe improved image generation when image understanding is included in co-training.
2. Related Work
Prior unified multimodal approaches combine generation and understanding through weakly coupled diffusion-LLM systems or shared VQGAN/VAE representations. These designs face integration, instruction-generation, or semantic-capacity limitations, while other methods separate pathways with additional vision-specific weights.
- Visual representations: Next-token and diffusion-based unified models commonly encode images with VQGAN or VAE compression models, reducing visual understanding capability.These encoders are designed primarily for visual compression rather than semantic representation.
- Alternative designs: ViLA-U combines vector quantization with visual foundation features, contrastive text alignment, and image reconstruction, but struggles to balance semantic alignment and pixel-level fidelity.Other approaches, such as LlamaFusion, disentangle pathways by adding weights activated only for vision signals.
3. Method
Harmon combines an MAR encoder, LLM, and MAR decoder in a unified autoregressive framework for image generation and understanding. Generation uses masked autoregression conditioned on language, while understanding encodes complete images and generates answers autoregressively.
- Framework: Harmon comprises an MAR encoder, an LLM, and an MAR decoder that jointly support multimodal generation and understanding.The framework omits connection layers between the LLM and the MAR encoder/decoder for brevity.
- Shared representation: The MAR encoder is used for both masked-autoregressive image generation and image-conditioned text autoregression.Figure 4 presents the shared encoder across the two task pathways.
- Text-to-image generation: For generation, the MAR encoder extracts features from seen patches and buffer embeddings, and the decoder predicts masked patches after LLM conditioning.Text prompts are mapped to embeddings, integrated by the LLM with encoder outputs, and passed to the MAR decoder with mask embeddings.
- Text-to-image generation: Generation begins with all patches masked and progressively reduces the mask ratio over K forward passes according to a cosine schedule.The number of predicted patches at step k is m_k − m_{k−1}; classifier-free guidance is also used.
- Image understanding: For image understanding, all image patches are encoded without masking, then the LLM generates an answer conditioned on the image representation and question.Answer-token prediction is supervised with a cross-entropy loss.
- Training: Harmon uses three training stages that progressively enhance generation and understanding, including knowledge-rich MAR training and multimodal data after LLM alignment.Stage I uses 22M densely captioned images and 1.2M ImageNet1K samples; Stage II unlocks the LLM with question-answering and text-to-image data.
4. Experiments
Harmon is evaluated across multimodal understanding and text-to-image generation, with ablations examining its visual encoder, training stages, resolution, and shared encoding. It performs competitively on understanding benchmarks and strongly on generation benchmarks, while qualitative results show improved prompt-image consistency and visual quality.
- Image Understanding: Harmon achieves performance on par with understanding-only models and Janus models on multimodal question-answering benchmarks.These comparisons include models using separate semantic encoders.
- Text-to-Image Generation: Harmon outperforms all unified models on MJHQ-30K and all unified and generation-only models on GenEval.MJHQ-30K measures aesthetic image quality, while GenEval evaluates controllability of attributes such as counting, position, and color.
- Text-to-Image Generation: Harmon also achieves state-of-the-art performance on MJHQ30K and higher visual quality on the WISE benchmark requiring world-knowledge comprehension.The reported evaluation spans instruction alignment, world knowledge, and visual quality.
- Qualitative Comparison: Harmon produces better prompt-image consistency and higher visual quality than Show-o and Janus-Pro in qualitative comparisons.Examples include accurately generating a pink stop sign and avoiding distorted paper-artwork and dog images.
- Analysis & Ablation: VQGAN and VAE lag behind MAR and SigLIP on understanding evaluation because they prioritize intrinsic image details and textures.The encoder study compares generative encoders with a semantic contrastive encoder.
- Analysis & Ablation: All three training stages contribute to overall performance, with stage III improving generation most and higher final-stage resolutions increasing both task performance.Higher resolutions particularly affect MSCOCO-30K and MJHQ-30K, which demand visual quality.
- Analysis & Ablation: The shared encoding design improves image generation while maintaining understanding capability in the mutual-supervision ablation.The comparison examines the interaction between understanding and generation supervision.
5. Conclusion
The paper concludes that MAR has potential for visual understanding beyond its generative role. Harmon’s shared visual encoding supports co-evolving understanding and generation, while co-training with understanding loss boosts generation performance.
- 5. Conclusion: The preliminary linear-probing and feature-visualization study identifies MAR as promising for visual understanding beyond its generative role.The conclusion frames this as the basis for investigating visual encoders in unified frameworks.
- 5. Conclusion: The study’s encoder comparisons include top-5 linear probing accuracy on ImageNet as an understanding measure.Table 6 uses Acc for this metric.
- 5. Conclusion: The ablations evaluate the effects of the three training stages and image resolution in stage III.These studies connect training design choices with unified-model performance.
- 5. Conclusion: Co-training with understanding loss boosts generation performance in the experiments.This result supports a mutual effect between the two capabilities under shared encoding.
A1.1. MAR
The MAR models used in Harmon follow an MAE-style encoder-decoder architecture and were originally trained on ImageNet1K for image generation.
- Model Details: MAR follows the encoder-decoder architecture of MAE and is trained on ImageNet1K for image generation.Class embeddings support class-conditional generation, with a null embedding for unconditional generation.
- Model Details: Harmon discards class embeddings and uses only null embeddings during MAR’s forward pass.The null embeddings are referred to as buffer embeddings in Harmon.
A1.2. Training Data
This appendix section states that the paper provides details about its training data, including data sources and re-captioning processes.
- Training Data: The paper documents its training-data sources and re-captioning processes.
A1.2.1. Image Understanding
Stage I uses 22M images with dense captions collected from several image-caption datasets and pre-training sources.
- Stage I: 22M images with dense captions are sourced from LLaVA-ReCap-CC3M, PixelProse, DenseFusion, MiniGemini, and ShareGPT4V pre-training data.The sources include re-captioned images and images labeled by a trained caption engine.
- Stage I: LLaVA-ReCap-CC3M captions are generated by LLaVA-NeXT-34B, while PixelProse captions are produced by Gemini-1.0-Pro-Vision.
- Stage I: DenseFusion labels 1M LAION images using a trained caption engine.
A1.2.2. Image Generation
Image-generation training combines rewritten captions, short captions, aesthetic filtering, and synthetic data across multiple stages.
- Stage II: Stage II rewrites dense captions into shorter descriptions with Qwen2.5-7B-Instruct for text-to-image generation.
- Stage II: Megalith10M uses provided short captions, while PD12M descriptions are rewritten with Qwen2.5-7B-Instruct.
- Stage II: 50M data samples are collected for Stage II text-to-image generation from dense-caption and image-generation datasets.
- Stage III: Stage III retains 10M images with aesthetic scores beyond 6.5 after filtering extreme height-width ratios.An additional 6M synthetic images are obtained from JourneyDB and Text-to-Image-2M.
A1.3. Visualization
The visualizations compare Harmon with unified and expert generation models and illustrate its generation, understanding, and shared-representation behavior, alongside stated scope limitations.
- Qualitative comparison: Harmon produces text-to-image results comparable to SDXL in visual quality and exhibits better prompt-image consistency in the reported qualitative comparison.The example contrasts positional instruction following involving a dog and a cat.
- Feature visualization: Figure A1 visualizes MAR feature-map activations and their mapping back to image pixels.The caption associates the activations with precise visual-concept responses and low-level image intrinsics.
- Limitations: Harmon’s model scale is limited to 1.5B, and its MAR pre-training uses 1.2M ImageNet1K samples, hindering further understanding-task improvement.
- Qualitative comparison: Figures A2 and A3 compare Harmon-1.5B with VILA-U, Show-o, Janus-Pro, and SDXL on text-to-image generation.Key prompt terms are highlighted in the qualitative comparisons.
- Additional examples: Figure A4 presents Harmon-1.5B text-to-image examples described as precise and diverse, while Figure A5 presents visual question-answering examples.