Source-linked AI summary
Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scaling
Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, Chong Ruan
TL;DR
Janus-Pro addresses limitations in Janus's training data, model capacity, and training strategy for unified multimodal understanding and generation. It combines decoupled visual encoding with revised training, expanded data, and scaling to larger models. Janus-Pro achieves stronger multimodal understanding and text-to-image instruction-following performance, while its limitations include restricted resolution and resulting detail loss.
Problem
Janus was limited by training data and model capacity, with suboptimal short-prompt generation and unstable text-to-image quality.
Method
Janus-Pro combines decoupled visual encoding with optimized training, expanded multimodal and synthetic aesthetic data, and scaling to a 7B model.
Results
Janus-Pro achieves the overall best multimodal understanding results and leads GenEval and DPG-Bench, including 84.19 on DPG-Bench.
Takeaways & Limitations
The unified approach improves multimodal understanding and text-to-image instruction-following performance across the reported benchmarks.
Takeaways & Limitations
Input resolution is limited to 384 × 384, constraining fine-grained understanding and leaving generated images lacking fine details.
Abstract
from arXiv · showhide
In this work, we introduce Janus-Pro, an advanced version of the previous work Janus. Specifically, Janus-Pro incorporates (1) an optimized training strategy, (2) expanded training data, and (3) scaling to larger model size. With these improvements, Janus-Pro achieves significant advancements in both multimodal understanding and text-to-image instruction-following capabilities, while also enhancing the stability of text-to-image generation. We hope this work will inspire further exploration in the field. Code and models are publicly available.
2. Method
Janus-Pro improves Janus through optimized training, expanded data, and larger model capacity while retaining decoupled visual encoding for understanding and generation.
- 2.1. Architecture: Janus-Pro decouples visual encoding for multimodal understanding and generation, mapping separate image representations into a shared autoregressive transformer.SigLIP extracts semantic features for understanding, while a VQ tokenizer produces discrete image IDs for generation; separate adaptors map both into the LLM input space.
- 2.2. Optimized Training Strategy: The revised training strategy lengthens Stage I and removes ImageNet data from Stage II, focusing later training directly on dense text-to-image descriptions.The authors report that fixed LLM parameters can still model pixel dependence in Stage I, while the redesigned Stage II uses text-to-image data more efficiently.
- 2.2. Optimized Training Strategy: Stage III changes the multimodal, pure-text, and text-to-image data ratio from 7:3:10 to 5:1:4, maintaining strong visual generation while improving multimodal understanding.This adjustment slightly reduces the text-to-image proportion during supervised fine-tuning.
- 2.3. Data Scaling: Janus-Pro scales multimodal understanding data by approximately 90 million samples and adds approximately 72 million synthetic aesthetic samples for visual generation.The synthetic data bring the real-to-synthetic ratio to 1:1 during unified pretraining, addressing noise in the earlier real-world generation data.
- 2.4. Model Scaling: The model scales from 1.5B to 7B parameters, and the larger language model substantially improves loss convergence for both multimodal understanding and visual generation.This scaling further validates the scalability of the visual encoding and decoding approach.
3. Experiments
Janus-Pro is evaluated on multimodal understanding and text-to-image generation benchmarks, where it achieves strong benchmark performance and produces detailed, semantically aligned images.
- Evaluation Setup: Janus-Pro is evaluated on multiple multimodal understanding benchmarks, including GQA, POPE, MME, SEED, MMB, MM-Vet, and MMMU.
- Evaluation Setup: GenEval measures compositional text-to-image capabilities, while DPG-Bench assesses semantic alignment on 1,065 dense prompts.
- Multimodal Understanding Performance: Janus-Pro achieves the overall best multimodal understanding results and outperforms the larger TokenFlow-XL 13B on every benchmark except GQA.
- Visual Generation Performance: 80% overall accuracy on GenEval and 84.19 on DPG-Bench place Janus-Pro-7B above the other reported methods on both text-to-image benchmarks.
- Qualitative Results: Qualitative results show realistic, detailed 384 × 384 images that capture prompt semantics and produce coherent imaginative scenes.
4. Conclusion
Janus-Pro improves multimodal understanding and text-to-image instruction following, but its 384 × 384 resolution limits fine-grained understanding and image detail.
- 384 × 384 input resolution limits Janus-Pro’s multimodal understanding performance on fine-grained tasks such as OCR.
- Low-resolution generation and vision-tokenizer reconstruction losses leave images semantically rich but lacking fine details.
- Small facial regions may appear under-detailed, while increasing image resolution could mitigate these generation issues.