Source-linked AI summary
Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation
Chengyue Wu, Xiaokang Chen, Zhiyu Wu, Yiyang Ma, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, Chong Ruan, Ping Luo
TL;DR
Existing unified multimodal models often use one visual encoder despite understanding and generation requiring different information granularity. Janus decouples those visual pathways within one transformer, and experiments report state-of-the-art performance across both tasks with extensibility to additional modalities.
Problem
Unified multimodal models commonly use a single visual encoder for understanding and generation, although the tasks require different levels of information.
Method
Janus uses independent visual encoding pathways for multimodal understanding and generation while processing them with one unified transformer.
Results
Janus achieves state-of-the-art results across multimodal understanding and generation benchmarks, including 69.4 on MMBench, 63.7 on SEED-Bench, 87.0 on POPE, 8.53 FID on MSCOCO-30K, and 61% accuracy on GenEval.
Takeaways & Limitations
Janus combines strong performance with a flexible, extensible design that can incorporate additional input modalities.
Takeaways & Limitations
The semantic tokenizer used as a stronger ablation baseline is not used in the main experiment, where Janus uses an ordinary VQ tokenizer.
Abstract
from arXiv · showhide
In this paper, we introduce Janus, an autoregressive framework that unifies multimodal understanding and generation. Prior research often relies on a single visual encoder for both tasks, such as Chameleon. However, due to the differing levels of information granularity required by multimodal understanding and generation, this approach can lead to suboptimal performance, particularly in multimodal understanding. To address this issue, we decouple visual encoding into separate pathways, while still leveraging a single, unified transformer architecture for processing. The decoupling not only alleviates the conflict between the visual encoder's roles in understanding and generation, but also enhances the framework's flexibility. For instance, both the multimodal understanding and generation components can independently select their most suitable encoding methods. Experiments show that Janus surpasses previous unified model and matches or exceeds the performance of task-specific models. The simplicity, high flexibility, and effectiveness of Janus make it a strong candidate for next-generation unified multimodal models.
1. Introduction
Janus addresses the mismatch between the visual representations needed for multimodal understanding and generation by decoupling visual encoding while retaining a unified transformer. It achieves strong results across both task types and supports flexible extension.
- Motivation: Unified multimodal models often use one vision encoder for understanding and generation, although the tasks require significantly different representation granularity.Understanding emphasizes high-level semantic information and reasoning, whereas generation requires concrete, detailed information.
- Method: Janus introduces independent visual encoding pathways for multimodal understanding and generation, unified by the same transformer architecture.The design is intended to alleviate conflicts between the tasks and avoid encoder-selection trade-offs.
- Flexibility: Decoupling lets each task adopt domain-specific state-of-the-art encoding techniques and permits future encoders for point clouds, EEG signals, or audio data.Independent encoders extract features that are then processed by a unified transformer.
- Results: Janus (1.3B) achieved 69.4 on MMBench, 63.7 on SEED-Bench, and 87.0 on POPE, outperforming LLaVA-v1.5 (7B) and Qwen-VL-Chat (7B).These are multimodal understanding benchmark scores reported for Janus.
- Results: Janus achieved an FID score of 8.53 on MSCOCO-30K and an accuracy of 61% on GenEval, surpassing DALL-E 2 and SDXL.The paper reports these results alongside strong visual generation capabilities.
2. Related Work
Unified multimodal models seek seamless understanding and generation, but prior systems commonly reuse one visual representation despite differing task requirements. Janus instead decouples visual representations for the two tasks.
- Visual Generation: Visual generation includes autoregressive models that predict sequences of discrete visual tokens, masked prediction models, and continuous diffusion models.These approaches draw on transformer-based sequence prediction, masking, or probabilistic generation.
- Multimodal Language Models: Some multimodal language models use pretrained diffusion models as tools, generating images from conditions produced by the language model rather than directly generating images themselves.In these systems, the MLLM does not directly perform visual generation.
- Unified Models: Unified multimodal models aim to support seamless reasoning and generation across modalities, traditionally using a single visual representation for both tasks.This pattern appears in both autoregressive and diffusion-based unified approaches.
- Janus: Janus decouples visual representations for understanding and generation because the two tasks may require different information levels.This contrasts with approaches such as Chameleon that use one VQ tokenizer for both tasks.
3. Janus: A Simple, Unified and Flexible Multimodal Framework
Janus uses independent encoders and adaptors for text, multimodal understanding, and visual generation, while a unified autoregressive transformer processes their features. Its staged training and modular design support both capabilities and future extensions.
- Architecture: Independent encoding methods convert text, understanding images, and generation images into features processed by one unified autoregressive transformer.SigLIP supplies high-dimensional semantic features for understanding, while a VQ tokenizer is used for visual generation.
- Training Procedure: Janus uses three training stages: adaptor and image-head training, unified pretraining, and supervised fine-tuning.The stages progressively connect visual and linguistic representations, train multimodal capabilities, and improve instruction following.
- Training Objective: During training, Janus applies cross-entropy loss to text sequences for understanding tasks and image sequences for visual generation tasks.The model does not assign different loss weights to different tasks.
- Inference: At inference, Janus uses sequential next-token prediction for text and multimodal understanding, and classifier-free guidance for image generation.The guidance logit combines conditional and unconditional logits using a scale factor.
- Flexibility: Separate encoders let understanding use stronger semantic encoders and generation use finer-grained encoders that preserve more image detail.The framework can also incorporate dynamic high-resolution processing, generation-specific losses, or combined autoregressive and parallel generation methods.
- Additional Modalities: Janus can integrate additional encoders for modalities such as 3D point clouds, tactile signals, and EEG.Its straightforward architecture supports extending the unified model beyond visual inputs.
4. Experiments
Janus is evaluated across multimodal understanding and visual generation benchmarks, with ablations testing decoupled visual encoding and unified versus task-specific training. The experiments report strong benchmark performance and qualitative instruction following, while analyzing how encoder choices affect the two tasks.
- Multimodal Understanding Performance: Janus achieves 41% and 30% improvements over Show-o on MME and GQA, respectively.MME improves from 949 to 1338, while GQA improves from 48.7 to 59.1.
- Visual Generation Performance: 61% overall accuracy on GenEval surpasses Show-o, SDXL, and DALL-E 2.Janus scores 61%, compared with 53% for Show-o, 55% for SDXL, and 52% for DALL-E 2.
- Impact of Decoupling Visual Encoding: Decoupled visual encoding improves multimodal understanding while preserving strong visual generation performance.Exp-A reaches 8.72 FID on COCO, but has a substantial understanding gap; Exp-C’s understanding performance exceeds the single-encoder Exp-B, supporting the importance of decoupling.
- Unified Model vs. Pure Understanding & Pure Generation: Unified training incorporates strong generative abilities while minimally affecting multimodal understanding performance.The ablation compares unified training with pure understanding and pure generation training across the corresponding task data.
- Qualitative Results: Qualitative evaluations show Janus follows prompts closely and interprets meme text, emotions, objects, and attributes more accurately than compared models.Generation examples emphasize prompt adherence, while meme comparisons report recognition problems for Chameleon and Show-o.
5. Conclusion
Janus is presented as a simple, unified, extensible model that decouples visual encoding for understanding and generation. Experiments demonstrate its effectiveness and leading performance, with potential extension to additional input modalities.
- Janus decouples visual encoding for multimodal understanding and generation to address their differing demands on visual encoders.
- Janus is described as a simple, unified, and extensible multimodal understanding and generation model.
- Extensive experiments demonstrate Janus’s effectiveness and leading performance.
- Janus can incorporate additional input modalities and may inspire next-generation multimodal general-purpose models.
A.1. Architecture of Semantic Tokenizer
The semantic tokenizer adds a semantic decoder branch to a tokenizer architecture while retaining the original CNN pixel decoder. Its semantic decoder is a causal-attention ViT used for integration with an LLM.
- The semantic tokenizer adds a semantic decoder branch after vector quantization alongside the original CNN pixel decoder.
- The semantic decoder is a 12-layer ViT with 12 attention heads and a hidden dimension of 768.
- A causal attention mask supports next-token prediction when the semantic decoder is integrated with an LLM.
- The tokenizer architecture uses a downsample rate of 16.
A.2. Training
The semantic tokenizer is trained from scratch in two stages, first on ImageNet-1k and then through fine-tuning on 50 million images. Its loss combines RGB reconstruction with semantic feature reconstruction supervised by SigLIP-Large-Patch16-384.
- The semantic tokenizer is trained from scratch in two stages: 40 epochs on ImageNet-1k followed by 1 epoch of fine-tuning on 50 million images.
- The training data for the second stage comes from visual-generation data used during Janus pretraining.
- The training loss combines RGB reconstruction with semantic feature reconstruction.
- SigLIP-Large-Patch16-384 supervises semantic reconstruction by maximizing cosine similarity between predicted semantic features and SigLIP outputs.
A.3. Integrating with LLM
The integrated pipeline transforms an image into continuous features, lets the LLM predict image IDs, and converts those discrete IDs into RGB values through a pixel decoder.
- The image is transformed into continuous features through a CNN encoder, vector quantization, and a semantic decoder.
- The LLM processes these features and generates predictions for image IDs.
- A pixel decoder converts the predicted discrete image IDs into RGB values.
B. Additional Qualitative Results
Janus produces high-quality text-to-image outputs that closely follow prompts, including prompts in languages absent from its training data. It also demonstrates multimodal understanding across diverse inputs such as charts, artwork, and formula images.
- Text-to-image generation: Janus generates high-quality images that adhere closely to the given prompts.The paper presents additional text-to-image results in Figure 7.
- Text-to-image generation: Despite training exclusively on English text-to-image data, Janus can perform text-to-image generation from prompts in Chinese, French, Japanese, and English with emoji.The authors describe this multilingual capability as emergent and attribute it to the original LLM translating languages into a unified semantic space.
- Multimodal understanding: Janus demonstrates multimodal understanding across scientific charts, artwork images, LaTeX formula images, and other contexts.Figure 9 provides additional qualitative multimodal understanding results.