Source-linked AI summary
JanusFlow: Harmonizing Autoregression and Rectified Flow for Unified Multimodal Understanding and Generation
Yiyang Ma, Xingchao Liu, Xiaokang Chen, Wen Liu, Chengyue Wu, Zhiyu Wu, Zizheng Pan, Zhenda Xie, Haowei Zhang, Xingkai yu, Liang Zhao, Yisong Wang, Jiaying Liu, Chong Ruan
TL;DR
Unified multimodal models must support both visual understanding and image generation without relying on unnecessarily complex architectures. JanusFlow integrates rectified flow with an autoregressive LLM, adds decoupled encoders and representation alignment, and achieves strong performance across both tasks, including results comparable to or better than specialized models.
Problem
Existing unified systems face architectural complexity or capability constraints when combining multimodal understanding with image generation.
Method
JanusFlow integrates rectified flow into an autoregressive LLM with lightweight adaptation, decoupled understanding and generation encoders, and representation alignment during training.
Results
JanusFlow achieves state-of-the-art performance across multimodal comprehension and text-to-image generation, outperforming existing unified approaches and several specialized models.
Takeaways & Limitations
The results support a unified model that harmonizes autoregressive and rectified-flow architectures while retaining strong performance in both understanding and generation.
Takeaways & Limitations
The reported training-data setting uses an expanded text corpus relative to Janus, with preliminary experiments indicating minimal impact on visual-generation quality.
Abstract
from arXiv · showhide
We present JanusFlow, a powerful framework that unifies image understanding and generation in a single model. JanusFlow introduces a minimalist architecture that integrates autoregressive language models with rectified flow, a state-of-the-art method in generative modeling. Our key finding demonstrates that rectified flow can be straightforwardly trained within the large language model framework, eliminating the need for complex architectural modifications. To further improve the performance of our unified model, we adopt two key strategies: (i) decoupling the understanding and generation encoders, and (ii) aligning their representations during unified training. Extensive experiments show that JanusFlow achieves comparable or superior performance to specialized models in their respective domains, while significantly outperforming existing unified approaches across standard benchmarks. This work represents a step toward more efficient and versatile vision-language models.
1. Introduction
JanusFlow unifies multimodal understanding and image generation by integrating rectified flow with an LLM through a minimalist architecture. It uses separate vision encoders and representation alignment to improve unified performance, achieving strong results across both domains.
- Motivation: Researchers are pursuing unified systems because specialized models separately address image comprehension and text-to-image generation.Existing approaches either combine pretrained generative components with LLMs or train a single LLM for both tasks, introducing architectural complexity or capability constraints.
- Results: JanusFlow surpasses state-of-the-art unified multimodal models and several task-specific understanding models on visual understanding benchmarks.The framework also generates high-quality 384 × 384 images.
- Architecture: JanusFlow integrates rectified flow with an LLM using only a lightweight encoder and decoder for generation.The design follows a minimalist architectural principle.
- Architecture: Separate vision encoders for understanding and generation prevent task interference and enhance comprehension capabilities.The model also aligns intermediate representations during training to strengthen semantic coherence in generation.
- Results: 9.51, 0.63, and 80.09% are JanusFlow’s scores on MJHQ FID-30k, GenEval, and DPG-Bench, respectively.It also scores 74.9, 70.5, and 60.3 on MMBench, SeedBench, and GQA, respectively, using a 1.3B-parameter LLM.
2. Related Work
Related work has progressed from diffusion-based multimodal extensions toward flow-based generative models and unified architectures. JanusFlow differentiates itself through rectified flow, decoupled vision encoders, and representation alignment.
- Flow-based generation: Flow-based generative models emerged as a simplified alternative to diffusion and have achieved strong performance with faster sampling.JanusFlow integrates rectified flow into an LLM for unified understanding and generation.
- Unified models: Multimodal research increasingly seeks architectures that simultaneously support visual understanding and generation.One major approach extends multimodal LLMs with pretrained diffusion models.
- JanusFlow: JanusFlow combines autoregressive capabilities with flow or diffusion models while addressing limitations of similar unified approaches.Its stated advantages are a simple rectified-flow generation process, decoupled vision encoders, and representation alignment regularization.
3. JanusFlow
JanusFlow unifies multimodal understanding and image generation in one LLM architecture by combining autoregressive prediction with rectified flow. Its design separates task-specific visual encoders, aligns representations, and trains the model through staged adaptation, unified pre-training, and instruction tuning.
- Unified architecture: JanusFlow uses autoregressive next-token prediction for visual understanding and rectified flow for image generation within one LLM.Generation starts from Gaussian noise and iteratively predicts velocity vectors until the final latent is decoded into an image.
- Unified architecture: Multimodal understanding encodes images into projected visual embeddings that are processed alongside tokenized text by the LLM.The understanding encoder produces a feature map, which is flattened and projected into the LLM embedding dimension.
- Unified architecture: Image generation operates in the SDXL-VAE latent space, where a generation encoder combines noisy latents with time embeddings before iterative Euler updates.The process replaces the current latent at each step, reaches z_1, and decodes it with the VAE decoder; classifier-free guidance improves semantic alignment.
- Decoupled encoders: JanusFlow decouples understanding and generation encoders, using SigLIP for semantic understanding features and separate ConvNeXt modules for generation.The design addresses the reported suboptimality of shared encoders and includes a long skip connection between the generation encoder and decoder.
- Training scheme: Training proceeds in three stages: adapting randomly initialized modules, unified pre-training on multimodal, generation, and text-only data, then supervised fine-tuning.The final stage uses instruction-tuning data and unfreezes the SigLIP encoder so the model can respond to both understanding and generation instructions.
- Training objective: Representation alignment trains a projection of intermediate LLM features to match the understanding encoder’s semantic feature space, improving generation quality.The alignment loss uses cosine similarity and does not back-propagate through the understanding encoder.
4. Experiments
JanusFlow is evaluated on multimodal understanding and image generation benchmarks, with ablations testing representation alignment and decoupled visual encoders. It achieves strong results across both tasks while retaining a unified architecture.
- Experimental overview: JanusFlow is evaluated on standard multimodal understanding and image generation benchmarks, followed by ablation studies of its key design choices.The experiments assess both capabilities and the effects of representation alignment and visual-encoder decoupling.
- Implementation: The 1.3B framework uses a 24-block LLM with 4,096-token sequences, SigLIP for understanding, and SDXL-VAE latent representations for generation.Its generation pathway uses lightweight ConvNeXt-based encoder and decoder modules.
- Training data: JanusFlow combines multimodal understanding, image-generation, and text-only data across its training stages.Stage 1 and Stage 2 use the three data types, while Stage 3 uses multimodal instruction, image-generation, and text-only data.
- Ablation studies: Representation alignment improves both image quality and semantic alignment, while decoupled visual encoders improve the unified model’s capabilities.The ablations report lower MJHQ FID and higher CLIP scores with alignment, and validate separately trained understanding and generation encoders.
- Qualitative results: Qualitative results show high visual quality, faithful instruction following, and visual reasoning across natural-language dialogues.The paper presents image-generation examples and multimodal conversations across varied scenarios.
5. Conclusion
JanusFlow unifies autoregressive and rectified-flow models for multimodal understanding and generation. Experiments show comparable performance to task-specific models and motivate further research on unified models.
- Contribution: JanusFlow harmonizes autoregressive and rectified-flow models within one framework for multimodal understanding and generation.The conclusion frames this integration as addressing current challenges in multimodal learning.
- Conclusion: Extensive experiments show that the unified framework achieves comparable performance to task-specific models.The conclusion presents this result as evidence that the two model paradigms can be successfully integrated.
- Implications: The successful integration opens new possibilities for future research in training unified models.This consequence is stated within the paper’s conclusion rather than as a broader field-wide prescription.
A. Performance Analysis of 256 Resolution Model
The 256×256 model is evaluated for visual understanding and text-to-image generation, with lower-resolution understanding but stronger semantic-generation performance than the 384×384 model.
- Evaluation scope: The 256×256 model is evaluated on visual understanding, GenEval, DPG-Benchmark, and MJHQ FID-30k.The main results use the 384 × 384 model, while the 256 × 256 evaluation reports these task-specific results.
- Visual understanding: The 256×256 model shows slightly lower visual understanding performance than the 384×384 model because of reduced resolution.
- Generation performance: The 256×256 model outperforms the 384×384 model on GenEval and DPG-Bench, which assess instruction following and semantic accuracy.The passage attributes this to better control over lower-resolution images and reduced visual complexity.
B. Details of the Datasets
JanusFlow uses separate pre-training datasets for understanding and generation, combining captioned, multimodal, scientific, and image-generation sources.
- Understanding datasets: Understanding pre-training uses DetailedCaption, SAM, arXivQA, DenseFusion-1M, MMSci, PixelProse, and several re-captioned or multimodal datasets.
- Generation datasets: Generation pre-training uses re-captioned LAION-Aesthetics, DALL-E 3 1M, SAM, Open Images V4, Megalith-10M, YFCC-15M, PixelProse, and JourneyDB.
- Dataset organization: The dataset design separates the sources used for understanding pre-training from those used for generation pre-training.
C. Analysis of CFG Factor and Sampling Steps
The analysis varies classifier-free guidance and sampling steps around the main setting of w=2 and 30 steps, revealing different effects on FID and CLIP similarity.
- Experimental settings: The main results use CFG factor w=2 and 30 sampling steps to calculate FID.
- CFG factor: Varying CFG factors while holding sampling steps at 30 reveals an optimal CFG value for FID scores.
- CFG factor: CLIP similarity continues to improve as CFG increases, unlike FID, which has an optimal CFG value.
D. Details of REPA Ablation
The ablation compares FID and CLIP similarity during the first 50,000 pre-training iterations with and without representation alignment regularization.
- Evaluation: The ablation tracks FID and CLIP similarity during the first 50,000 iterations of pre-training.
- Ablation setup: The comparison includes models trained with and without representation alignment regularization.
- Ablation result: The gap between the two models demonstrates the benefits of representation alignment regularization.
E. Additional Qualitative Results
Additional qualitative examples show JanusFlow handling diverse multimodal understanding tasks and producing images with strong visual quality and semantic alignment. The examples include visual reasoning, person identification, character recognition, and code generation.
- JanusFlow demonstrates diverse understanding capabilities, including code generation, person identification, character recognition, and visual reasoning.
- For image generation, JanusFlow shows strong visual quality and semantic alignment with input prompts.
- A Mona Lisa rendition with a dog illustrates the model’s ability to explain humorous or artistic visual content.
- The response “George W. Bush” demonstrates person identification in an image.