Source-linked AI summary
Cheers: Decoupling Patch Details from Semantic Representations Enables Unified Multimodal Comprehension and Generation
Yichen Zhang, Da Peng, Zonghao Guo, Zijian Zhang, Xuesong Yang, Tong Sun, Shichu Sun, Yidan Zhang, Yanghao Li, Haiyan Zhao, Wang Xu, Qi Shi, Yangang Sun, Chi Chen, Shuo Wang, Yukun Yan, Xu Han, Qiang Ma, Wei Ke, Liang Wang, Zhiyuan Liu, Maosong Sun
TL;DR
Unified visual comprehension and generation remains challenging because the tasks require mismatched decoding mechanisms and representations. CHEERS decouples semantic representations from patch-level details using a unified tokenizer and hybrid decoding architecture, and it matches or exceeds advanced unified multimodal models while achieving 4× token compression and requiring only 20% of Tar’s training cost.
Problem
Visual comprehension and high-fidelity image generation require fundamentally different decoding mechanisms and visual representations, making their joint optimization challenging.
Method
CHEERS decouples patch-level details from semantic representations through a unified vision tokenizer, hybrid autoregressive-and-diffusion decoding, and cascaded flow matching with gated detail residuals.
Results
CHEERS matches or exceeds state-of-the-art unified multimodal models in visual comprehension and generation, achieves 4× token compression, and outperforms Tar on GenEval and MMBench using 20% of the training cost.
Takeaways & Limitations
Hierarchical generation from global semantic layout to localized detail refinement supports effective and efficient unified multimodal modeling.
Takeaways & Limitations
CHEERS uses a relatively small model, lacks initialization from large-scale pretrained VLMs, and is trained on single-image datasets, limiting current capacity and generalization.
Abstract
from arXiv · showhide
A recent cutting-edge topic in multimodal modeling is to unify visual comprehension and generation within a single model. However, the two tasks demand mismatched decoding regimes and visual representations, making it non-trivial to jointly optimize within a shared feature space. In this work, we present Cheers, a unified multimodal model that decouples patch-level details from semantic representations, thereby stabilizing semantics for multimodal understanding and improving fidelity for image generation via gated detail residuals. Cheers includes three key components: (i) a unified vision tokenizer that encodes and compresses image latent states into semantic tokens for efficient LLM conditioning, (ii) an LLM-based Transformer that unifies autoregressive decoding for text generation and diffusion decoding for image generation, and (iii) a cascaded flow matching head that decodes visual semantics first and then injects semantically gated detail residuals from the vision tokenizer to refine high-frequency content. Experiments on popular benchmarks demonstrate that Cheers matches or surpasses advanced UMMs in both visual understanding and generation. Cheers also achieves 4x token compression, enabling more efficient high-resolution image encoding and generation. Notably, Cheers outperforms the Tar-1.5B on the popular benchmarks GenEval and MMBench, while requiring only 20% of the training cost, indicating effective and efficient (i.e., 4x token compression) unified multimodal modeling. We will release all code and data for future research.
1 Introduction
Unifying visual comprehension and generation is difficult because the tasks require different decoding mechanisms and visual representations. CHEERS addresses this conflict by separating semantic representations from patch-level details within a unified multimodal architecture.
- Motivation: Visual comprehension and high-fidelity image generation rely on different decoding mechanisms and visual representations, complicating joint optimization.Autoregressive visual tokens can lose information through quantization and dimensional constraints, while diffusion modeling captures global image context.
- Approach: CHEERS decouples patch-level details from semantic representations to stabilize understanding and improve image-generation fidelity.Its unified vision tokenizer integrates structural and semantic features, while semantically gated high-frequency residuals refine generated images.
- Approach: CHEERS combines a unified vision tokenizer, an LLM-based Transformer with autoregressive and diffusion decoding, and a cascaded flow matching head.The design supports efficient LLM conditioning while allowing image generation to synthesize semantics before injecting high-frequency details.
- Results: 4× token compression enables more efficient high-resolution image understanding and generation.The compression is part of CHEERS’s unified vision-tokenizer design.
- Results: CHEERS matches or exceeds state-of-the-art unified multimodal models across visual comprehension and generation benchmarks.It also outperforms the Tar model on GenEval and MMBench while requiring only 20% of the training cost.
2 CHEERS
CHEERS unifies multimodal understanding and image generation through a vision tokenizer, shared LLM-based Transformer, and cascaded flow matching head. Its architecture compresses semantic visual tokens for LLM conditioning while injecting gated high-frequency details during image synthesis.
- Architecture: CHEERS comprises a unified vision tokenizer, an LLM-based Transformer backbone, and a cascaded flow matching head for image generation.Text tokenization and an LM head support language encoding and visual understanding tasks.
- Unified Vision Tokenizer: The unified vision tokenizer reconstructs VAE latents into pixel space, extracts semantic tokens with SigLIP2-ViT, and preserves fine-grained features relevant to OCR.Direct latent processing was found to discard fine-grained features and hinder OCR-centric understanding.
- Unified Vision Tokenizer: 2D token compression reduces semantic-token spatial resolution by Pixel-Unshuffle before projecting channels to the LLM hidden size.The compressed representation has spatial dimensions h/2 × w/2 and channel dimension c.
- Unified Transformer: The shared LLM processes semantic visual and text tokens, using autoregressive decoding for text and comprehension and flow matching for image generation.Visual tokens use bidirectional attention to capture global context, while text tokens use causal masking.
- Cascaded Flow Matching Head: The cascaded flow matching head first generates low-resolution semantic features, then injects gated high-frequency patch details before predicting the velocity field.The first stage upsamples semantic features, while the second stage adaptively controls fine-grained detail injection.
- Inference: High-frequency injection is dynamically coupled to the timestep and naturally intensifies as generation progresses without explicit supervision.Image generation repeatedly tokenizes the current latent, conditions the LLM, predicts velocity, and numerically integrates from Gaussian noise to the terminal latent.
3 Experiments
CHEERS is evaluated across multimodal understanding and image-generation benchmarks, with ablations examining staged training, joint objectives, and high-frequency injection. Results show competitive or superior performance, progressive generation gains, and improved detail quality without compromising understanding.
- Main Results: CHEERS achieves competitive understanding performance on nearly all evaluated multimodal benchmarks.The evaluation covers general, OCR, spatial, and knowledge-focused benchmarks.
- Main Results: CHEERS consistently matches or exceeds existing approaches on image-generation benchmarks under comparable parameter scales.The reported generation evaluation uses GenEval and DPG-Bench.
- Main Results: 83M training samples suffice for CHEERS to achieve strong image-generation results, indicating high data efficiency.The reported total includes visual, image, and text training samples.
- Training Analysis: Generation improves progressively during training, with the clearest surge occurring during Refined Pre-Training on synthetic, instruction-oriented data.Earlier stages use real-world image-caption pairs, while synthetic data provides clearer compositions and attribute bindings.
- Ablation Studies: Joint training with understanding and generation objectives preserves comparable or slightly superior understanding performance relative to understanding-only fine-tuning.The controlled comparison also gives the jointly trained model image-generation capability.
- Ablation Studies: High-frequency injection minimally affects understanding but substantially improves generation quality, texture fidelity, and structural sharpness.Without HFI, images remain semantically consistent but lack fine-grained visual details.
- Implementation: CHEERS leverages pretrained native ViT weights and jointly fine-tunes core modules, avoiding the overhead of training a unified vision encoder.The architecture does not freeze core parameters during joint fine-tuning.
4 Related Work
Related work frames unified multimodal modeling around the tension between semantic understanding and detail-preserving generation. Existing approaches differ in representation format, feature granularity, tokenizer design, and decoding paradigm, including autoregressive, diffusion, and hybrid frameworks.
- Visual Tokenizers: Image tokenizers trade off representation format, feature granularity, and architectural integration when bridging comprehension and generation.These choices determine how semantic and reconstructive information are modeled.
- Representation Formats: Discrete tokenizers support autoregressive modeling but can introduce information bottlenecks, whereas continuous latent representations preserve richer details for high-fidelity generation.The contrast reflects different fidelity and modeling-paradigm priorities.
- Feature Granularity: Semantic encoders capture reasoning-oriented features but may overlook textures, while generative VAEs preserve detail but lack global context.Late-fusion and cascaded designs attempt to reconcile these complementary weaknesses.
- Unified Visual Tokenizers: Unified visual-tokenizer research includes discrete shared-codebook methods and continuous frameworks that combine semantic encoders with generative spaces.Prior continuous approaches include frozen semantic features, joint modeling, and related unified designs.
- Unified Multimodal Models: Unified multimodal models are commonly organized as pure autoregressive, pure diffusion, or hybrid frameworks within shared transformer architectures.These paradigms differ in their modeling objectives for multimodal perception and synthesis.
- Cross-Task Synergy: Recent work also studies bidirectional benefits between comprehension and generation through feature alignment and latent-space coordination.Examples align diffusion features with pretrained visual encoders or address reconstruction–generation optimization conflicts.
5 Conclusion
CHEERS unifies visual comprehension and high-fidelity generation by separating semantic representations from patch-level details. Its reported strengths include competitive performance, 4× token compression, and zero-shot image editing, while several scale and data constraints remain.
- 5 Conclusion: CHEERS decouples patch-level details from semantic representations to support visual understanding and high-fidelity image generation within one framework.A unified vision tokenizer extracts stable semantics, while a cascaded flow matching head injects semantically gated high-frequency residuals.
- 5 Conclusion: 4× token compression enables efficient high-resolution image understanding and generation.
- 5 Conclusion: CHEERS demonstrates competitive performance across ten understanding benchmarks and multiple generation benchmarks.
- 5 Conclusion: CHEERS shows zero-shot image editing capabilities despite training only on a relatively modest dataset of 83M samples.
- 5 Conclusion: The study is constrained by CHEERS’s relatively small parameter scale, lack of large-scale pretrained VLM initialization, and reliance on single-image datasets.
A Why Reconstruct Pixels Before Semantic Encoding?
Directly encoding VAE latents into a vision transformer severely harms OCR-centric understanding, and increasing alignment data does not resolve the degradation. Reconstructing pixels before semantic encoding substantially improves OCR performance.
- A Why Reconstruct Pixels Before Semantic Encoding?: Directly projecting VAE latents into SigLIP2-ViT causes severe degradation in OCR-centric understanding.
- A Why Reconstruct Pixels Before Semantic Encoding?: At 256 × 256 resolution, direct latent processing produces extremely poor OCR-related benchmark performance, indicating loss of fine-grained textual information.
- A Why Reconstruct Pixels Before Semantic Encoding?: Scaling the alignment dataset to 4.6M samples yields negligible OCR improvement while retaining the same 858K fine-tuning set.
- A Why Reconstruct Pixels Before Semantic Encoding?: Pixel reconstruction decodes the latent representation into pixel space before SigLIP2-ViT semantic encoding, substantially improving OCR-centric benchmarks.
B Emergent Abilities
After Stage 3 training, CHEERS exhibits emergent capabilities on image editing and multi-image tasks despite training generation only on text-to-image data. The unified visual tokenizer is presented as supporting shared representations across tasks.
- B Emergent Abilities: Generation training through Stage 3 uses only text-to-image tasks, without image-editing or multi-image data.
- B Emergent Abilities: Despite this training scope, CHEERS demonstrates potential on image-editing and multi-image tasks.
- B Emergent Abilities: The unified visual tokenizer represents different visual tasks in a shared feature space, facilitating capability transfer across tasks.
C Examples of Multimodal Understanding
Qualitative examples illustrate CHEERS’s multimodal comprehension and reasoning, including image-editing and multi-image composition prompts. The examples are presented as emergent-capability demonstrations after refined pre-training.
- C Examples of Multimodal Understanding: Qualitative examples demonstrate CHEERS’s ability to comprehend and reason over multimodal inputs.For multimodal understanding, images are resized with the longer side at 512 pixels while preserving aspect ratio and then padded.
- C Examples of Multimodal Understanding: Tables 7 and 8 present emergent capabilities of CHEERS after refined pre-training.
- C Examples of Multimodal Understanding: The examples include changing an image background to blue and transforming an apple into a watermelon.
- C Examples of Multimodal Understanding: The examples also include combining a red bell pepper from one figure with green broccoli from another into one image.