Harmonizing Visual Representations for Unified Multimodal Understanding and Generation

• VQ and VAE encoders prioritize intrinsic features for pixel-level reconstruction over visual semantics, limiting their image-understanding capability despite the coarse representations understanding requires.The paper frames understanding as requiring coarse-grained, high-level representations and generation as preferring fine-grained imagery features.Source: §1 · p. 2
• MIM [2] [34] uses mask-and-reconstruct pretraining to learn richer visual semantics, and MAR [28] [46] extends it to autoregressive generation, aligning with multimodal LLMs’ next-token prediction paradigm.The paper presents this shared-encoding approach as a way to balance generation and understanding.Source: §1 · p. 2
• Linear probing and feature visualization show that VQGAN and VAE encoders struggle with high-level semantics, while the MIM-trained MAR encoder achieves competitive accuracy and precise feature activations.The preliminary study uses Grad-CAM++ [8] for feature visualization.Source: §1 · p. 2, Figure 2 · p. 2
• Harmon uses a shared MAR-based encoder, generates images through masked autoregression, regresses text with next-token prediction, and aligns the encoder with the LLM language space through three training stages.The encoder is designed to capture both coarse semantics and fine-grained features for consistent semantic grounding across tasks.Source: §1 · pp. 2–3, Figure 3 · p. 2
• Harmon surpasses similarly scaled unified methods on GenEval [32] and all unified models on WISE [61], achieves state-of-the-art MJHQ30K results [45], matches Janus [87] and Janus-Pro [15] on image understanding, and improves generation through co-training with understanding.The paper reports that Harmon also exceeds VQ- or VAE-based methods on image understanding benchmarks.Source: §1 · p. 3

2. Related Work

• MLLMs [16] [44] [52] [57] [85] [99] use semantic encoders such as CLIP [68] and SigLIP [96] for understanding, while diffusion, autoregressive, masked, and MAR models generate images.MLLMs are limited to image-conditioned question-answering, whereas diffusion models commonly operate in VAE latents and autoregressive or masked-generation frameworks often use VQ tokens.Source: §2 · p. 3
• Diffusion-plus-MLLM systems [77] [78] [81] [84] lack deep interaction between image generation and text modeling, limiting instruction-based generation [32].Source: §2 · p. 3
• Unified next-token systems [54] [79] [86] [89] and LLM diffusion backbones [90] [91] [97] [98] use VQGAN or VAE compression, diminishing visual understanding.Source: §2 · p. 3
• ViLA-U [89] combines quantization, contrastive alignment, and reconstruction but struggles to balance semantic alignment with pixel-level fidelity; LlamaFusion [74] adds vision-specific weights.Source: §2 · p. 3
• Janus [15] [87] uses separate VQGAN [26] and SigLIP [96] encoders; Harmon instead shares an MIM-trained MAR encoder for understanding and generation.The paper argues that Janus's separate-encoder design neglects the potential for a unified representation to foster deeper cross-modal synergy.Source: §2 · p. 3

3. Method

• Harmon shares an MAR encoder and LLM across image-conditioned text generation and understanding, using an MAR decoder to generate images.Generation is masked autoregressive; understanding is image-conditioned text autoregression.Source: §3 · p. 3, Figure 4 · p. 4

3.1. Preliminaries

• MAR’s encoder captures visual concepts and semantics during generative pre-training, making it a candidate shared encoder for image understanding and generation.The paper motivates this choice with preliminary results indicating strong semantic representations.Source: §3.1 · p. 3

3.2. Text-to-Image Generation

• Text-to-image generation sends MAR encoder features through the LLM with caption prompts, then uses the MAR decoder to predict masked patches via reverse diffusion.Training empties the caption for 10% of samples to enable classifier-free guidance; inference progressively reduces the mask ratio and uses cached LLM keys and values.Source: §3.2 · pp. 4–5

3.3. Image Understanding

• Image understanding feeds all unmasked image patches and buffer embeddings through the MAR encoder, then autoregressively generates an answer conditioned on the image and question.Answer tokens are supervised with cross-entropy loss.Source: §3.3 · p. 5

3.4. Training Recipe

• Harmon trains in three progressively capability-enhancing stages: Stage I aligns modalities using 22M dense-caption images and 1.2M ImageNet1K samples, with only the MAR encoder and decoder trainable.Stage II unlocks the LLM and uses 20M multimodal-understanding samples plus 5M Stage I dense-caption samples, and 50M text-to-image images. Stage III filters the 50M Stage II images by size and aesthetic score, reserves 20%, adds 6M synthetic images, and raises resolution from 256 to 512.Source: §3.4 · p. 5

4.1. Comparisons with Existing Models

• Harmon performs on par with understanding-only and Janus models, and outperforms prior VAE/VQGAN unified models on multimodal question-answering benchmarks.The comparisons focus mainly on multimodal models with fewer than 2B LLM parameters; Janus uses a separate SigLIP semantic encoder.Source: Table 2 · p. 6, §4.1 · p. 6
• Harmon outperforms all unified models on MJHQ-30K and WISE, and all unified and generation-only models on GenEval; Harmon-0.5B is comparable to Janus-Pro-1.5B.The evaluations cover image quality on MJHQ-30K, instruction controllability on GenEval, and complex semantic and world knowledge on WISE.Source: Table 3 · p. 6, §4.1 · p. 6, Table 4 · p. 7

4.2. Analysis & Ablation

• VQGAN and VAE encoders lag behind MAR and SigLIP on understanding, while MAR performs comparably to the SigLIP semantic encoder.The comparison uses MAR-B and SigLIP-Base-Patch16; the findings align with prior masked-image-modelling results and the paper’s preliminary linear-probing study.Source: §4.2 · pp. 7–8

4.1. Comparisons with Existing Models

• Harmon-1.5B better matches prompts than Show-o-1.3B-512 and Janus-Pro-1.5B, including text-bearing stop signs and sinks, and produces less-distorted artwork and dog images.The qualitative comparison reports Harmon generating a pink stop sign with the requested text and a sink missed by Show-o.Source: §4.1 · pp. 6–7, Figure 5 · p. 8

4.2. Analysis & Ablation

• All three training stages contribute: stages I-II chiefly improve understanding, stage III chiefly improves generation and MJHQ-30K, while higher final-stage resolution lowers COCO and MJHQ-30K FID more than it improves understanding or GenEval.Resolution gains are more pronounced on the visual-quality benchmarks than on general understanding and instruction-alignment tasks.Source: §4.2 · p. 8
• Unified training maintains understanding scores versus single-task training, and understanding loss boosts generation, especially on GenEval; decoupling SigLIP and MAR removes this synergy.The ablation attributes the co-evolution of understanding and generation capabilities to a shared representation.Source: Figure 6 · p. 8, §4.2 · p. 8

5. Conclusion

• A preliminary study using linear probing and feature visualization identifies MAR’s potential for visual understanding beyond its generative role.Source: §5 · p. 8
• Harmon achieves state-of-the-art results on text-to-image generation benchmarks while performing competitively on multimodal question-answering benchmarks.Source: §5 · p. 9
• Co-training with understanding loss boosts generation performance, revealing synergy between the two tasks.Source: §5 · p. 9
• Harmon unifies image understanding and generation in MAR’s encoding space through a progressive three-stage training procedure.Source: §5 · p. 9

A1.1. MAR

• MAR [46] uses an MAE-style encoder-decoder trained on ImageNet1K for image generation.Its architecture follows MAE [34].Source: A1.1. MAR · p. 14
• For class-conditional generation, MAR feeds class embeddings to its encoder and includes a null embedding for unconditional generation.Source: A1.1. MAR · p. 14
• Harmon discards MAR’s class embeddings and uses only null embeddings, called buffer embeddings in the main text.Source: A1.1. MAR · p. 14
• MAR compresses images into VAE latents before encoding, following common practice [63] [64] [71].The appendix illustration omits the VAE component for brevity.Source: A1.1. MAR · p. 14
• MAR encoder feature activations indicate that generative training captures essential visual concepts.The appendix provides additional visualizations in Figure A1.Source: A1.1. MAR · p. 14
• Zero-shot mapping of MAR encoder features through its decoder yields image contents despite training to predict unseen patches rather than recover pixels.Source: A1.1. MAR · p. 14
• The decoder visualizations suggest MAR’s encoder representations also contain intrinsic imagery features needed for visual generation.Source: A1.1. MAR · p. 14

Training Data

• The section covers training-data sources and re-captioning processes.The passage does not specify the sources or describe the re-captioning procedures.Source: A1.2. Training Data · p. 14

A1.2.1. Image Understanding

• Training progresses from 22M dense-captioned images in Stage I to instruction tuning and high-quality fine-tuning in Stages II and III.Stage I draws on LLaVA-ReCap-CC3M [44], PixelProse [75], DenseFusion [47], MiniGemini [49] and ShareGPT4V [13]. Stage II uses 20M Infinity-MM-Stage3 [33] examples plus 5M Stage I images; Stage III uses LLaVA-One-Vision [44] instruction-tuning data.Source: A1.2.1. Image Understanding · p. 14

Image Generation

• Stage II collects 50M text-to-image samples from rewritten dense captions and generation datasets, with LAION-Aesthetics images labeled using Qwen2-VL-72B.Stage I uses 1.2M ImageNet1K samples with class names as captions; Stage II rewrites dense captions with Qwen2.5-7B-Instruct, including the original 22M captions and PD12M descriptions. Megalith10M uses provided short captions, while 6M LAION-Aesthetics images are crawled and labeled with precise prompts.Source: A1.2.2. Image Generation · p. 14
• For high-quality text-to-image generation, Stage III retains images scoring above 6.5 on an aesthetic model and discards extreme aspect ratios.The passage says 10M images are selected for Stage II, and 6M synthetic images from JourneyDB and Text-to-Image-2M are added to enhance visual quality.Source: A1.2.2. Image Generation · p. 14

A1.3. Visualization

• MAR feature maps respond precisely to visual concepts and can be mapped back to image pixels, revealing low-level image intrinsics.Source: Figure A1 · p. 15

A1.4. Limitations

• Harmon is limited to 1.5B parameters, and the authors plan to scale it up in future work.Source: A1.4. Limitations · p. 15
• MAR’s ImageNet1K pretraining uses 1.2M samples, far fewer than the billion-scale data used to train semantic encoders such as CLIP and SigLIP, limiting understanding-task improvement.Source: A1.4. Limitations · p. 15

A1.3. Visualization

• Harmon-1.5B matches SDXL [64] in visual quality and has better prompt-image consistency, while VILA-U [89], Show-o [91], and Janus-Pro [15] are also compared.In Figure A2, SDXL fails to follow the prompt’s left-right relation between a dog and a cat.Source: A1.3. Visualization · p. 14, Figure A2 · p. 16, Figure A3 · p. 17
• Harmon-1.5B generates precise and diverse text-prompted images in the additional examples.Source: A1.3. Visualization · p. 14, Figure A4 · p. 18
• Additional multimodal understanding examples show Harmon-1.5B answering questions about images in visual question-answering format.Source: A1.3. Visualization · p. 14, Figure A5 · p. 19