Source-linked AI summary
UniCom: Unified Multimodal Modeling via Compressed Continuous Semantic Representations
Yaqi Zhao, Wang Lin, Zijian Zhang, Miles Yang, Jingyuan Chen, Wentao Zhang, Zhao Zhong, Liefeng Bo
TL;DR
Unified multimodal models must bridge discrete visual tokenization, which can lose fine-grained semantics, and high-dimensional continuous representations, which are difficult to model generatively. UniCom compresses continuous visual embeddings into a compact unified representation and compares unified prediction pathways. It reports strong generation and editing performance, while acknowledging fidelity and scalability trade-offs in compressing and training large continuous models.
Problem
Existing approaches either lose fine-grained semantic information through discrete visual tokenization or face difficult generative modeling of high-dimensional continuous visual representations.
Method
UniCom uses a continuous semantic compressor to map high-dimensional visual embeddings into a compact latent space and predicts those latents through unified generative pathways.
Results
UniCom achieves state-of-the-art or competitive performance across generation, reconstruction, and editing, with the Transfusion pathway converging faster and producing more consistent editing than the query-based pathway.
Takeaways & Limitations
Compressed semantic representations can support unified visual understanding and generation while preserving high-level semantics and fine-grained visual details without relying on VAE features.
Takeaways & Limitations
Compression may lose very fine-grained details in complex images, and the resource demands of training large models can limit accessibility and sustainability.
Abstract
from arXiv · showhide
Current unified multimodal models typically rely on discrete visual tokenizers to bridge the modality gap. However, discretization inevitably discards fine-grained semantic information, leading to suboptimal performance in visual understanding tasks. Conversely, directly modeling continuous semantic representations (e.g., CLIP, SigLIP) poses significant challenges in high-dimensional generative modeling, resulting in slow convergence and training instability. To resolve this dilemma, we introduce UniCom, a unified framework that harmonizes multimodal understanding and generation via compressed continuous representation. We empirically demonstrate that reducing channel dimension is significantly more effective than spatial downsampling for both reconstruction and generation. Accordingly, we design an attention-based semantic compressor to distill dense features into a compact unified representation. Furthermore, we validate that the transfusion architecture surpasses query-based designs in convergence and consistency. Experiments demonstrate that UniCom achieves state-of-the-art generation performance among unified models. Notably, by preserving rich semantic priors, it delivers exceptional controllability in image editing and maintains image consistency even without relying on VAE.
1 Introduction
UniCom addresses the tension between preserving rich visual semantics and making continuous representations tractable for generation by compressing visual embeddings into a compact unified space. Its design combines channel-wise compression, attention-based projection, and unified prediction to support understanding, reconstruction, generation, and editing.
- Motivation: Unified multimodal models face a representational divide between discrete language reasoning and continuous visual synthesis.Hybrid VAE–ViT designs use different feature spaces for understanding and generation, while continuous ViT representations remain difficult to model generatively.
- Semantic compression: UniCom compresses high-dimensional visual semantics into a compact continuous latent space that preserves semantic, structural, and textural information.The compressor is evaluated for reconstruction fidelity and is reported to achieve near-lossless visual-embedding compression.
- Compression design: Compressed representations preserve both understanding and pixel-level reconstruction capabilities while improving training stability and computational efficiency.These findings are described as preliminary experiments in the introduction.
- Unified modeling: The framework compares a Transfusion-inspired end-to-end pathway with a query-based pathway for predicting compressed visual latents from textual conditions.The Transfusion pathway uses a unified flow-matching objective over mixed-modality sequences.
- Unified modeling: UniCom generates directly over compressed visual embeddings and achieves state-of-the-art or competitive performance across reconstruction, text-to-image generation, and image editing.The framework is designed to preserve high-level semantics and fine-grained visual details without relying on VAE features for identity preservation.
- Compression design: Channel-dimension compression preserves information better than sequence reduction, while attention-based projection maintains semantic structure.The paper identifies channel compression as superior for preserving information and the attention-based projector as crucial for semantic preservation.
2 Related Work
Prior unified multimodal approaches integrate image generation into autoregressive language-model frameworks by serializing visual features as token sequences, often with diffusion decoders. These designs target the challenge of image tokenization while combining autoregressive understanding with diffusion-based generation.
- Autoregressive multimodal generation: Autoregressive language-model frameworks integrate image generation by addressing the challenge of image tokenization.Early methods serialize CLIP or SigLIP vision-language features into text-like token sequences.
- Autoregressive multimodal generation: Serialized visual-language encoders are typically paired with diffusion decoders to combine autoregressive understanding with diffusion-based generation.The cited examples include CLIP- and SigLIP-based encoders.
3 UniCom
UniCom constructs a compressed continuous semantic space that supports unified visual understanding and generation. It compares transfusion and query-guided prediction pathways over shared compressed representations, with reconstruction and flow-matching training.
- Problem Formulation and Unified Framework: UniCom decomposes conditional image generation through a compressed semantic latent ˜z, retaining reconstruction fidelity while making P(˜z | c) smoother to model.The compressed latent has lower channel dimension than the native visual representation.
- Semantic Compression and Reconstruction: The framework jointly pre-trains a semantic compressor and diffusion decoder, then freezes them while training a generative prior in the fixed latent space.This separates representation learning from conditional sampling in the compressed space.
- Semantic Compression and Reconstruction: The compressor Cϕ maps high-dimensional visual features into ˜Z ⊂ R^N×d with d ≪ D, balancing semantic and visual fidelity against generative tractability.Its reconstruction objective combines flow-matching and perceptual losses.
- Semantic Compression and Reconstruction: An attention-based compressor preserves long-range relationships between image patches that token-independent MLP compression may disregard.The design uses a shallow transformer-style module to retain structured semantic information.
- Two Ways of Predicting Representations: The transfusion pathway interleaves discrete text tokens with continuous image latents in one transformer, using causal text attention and bidirectional attention within each image.Visual understanding uses uncompressed features, while image editing uses compressed latents to reduce context length.
- Two Ways of Predicting Representations: The query-guided pathway feeds text and learnable MetaQueries through a frozen multimodal model, then projects query states into the flow-matching decoder.Both prediction pathways share the same compressor and diffusion decoder for controlled comparison.
4 Experiments
UniCom experiments evaluate reconstruction, text-to-image generation, image editing, feature-shape choices, projector architectures, and prediction pathways. Results favor channel compression with attention-based projection and Transfusion-style full-sequence modeling.
- Main Results: UniCom achieves competitive or state-of-the-art performance across image reconstruction, text-to-image generation, and image editing tasks.The evaluation covers reconstruction, GenEval, DPG-Bench, WISE, ImgEdit-Bench, and GEdit-Bench.
- Main Results: An 18× reduction from d1152 to d64 incurs negligible reconstruction-fidelity loss on ImageNet.The d64 variant also recovers high-frequency details and maintains facial consistency comparably to the Flux VAE.
- Optimal Feature Shape: Reducing feature dimension preserves reconstruction and generation quality better than reducing sequence length.Sequence reduction causes noticeable blurring and drastically undermines image quality, whereas dimension reduction to 256 or 64 dimensions causes relatively minimal loss.
- Generative Modeling Dynamics: Compressing d from 1152 to 64 accelerates convergence by approximately 5× and improves final generative quality.Directly modeling uncompressed features produces slow convergence and suboptimal performance.
- Projector Architecture: The MHA projector preserves semantic structure better than the MLP projector despite comparable reconstruction fidelity.MHA embeddings remain tightly clustered and structured, and MHA consistently outperforms MLP on downstream visual-understanding benchmarks.
- Prediction Pathways: Transfusion outperforms query-based prediction because full-sequence modeling preserves spatial correspondence and yields better editing fidelity and structural consistency.The query-based pathway converges significantly more slowly and its bottleneck discards spatial details.
5 Conclusion
UniCom unifies visual understanding and generation by compressing high-dimensional visual embeddings into a compact latent space. The framework preserves semantic and fine-grained visual information while supporting generation and understanding without a VAE.
- Conclusion: UniCom compresses high-dimensional visual embeddings into a compact latent space for unified multimodal understanding and generation.The framework explores Transfusion and MetaQuery prediction pathways over compressed representations.
- Conclusion: UniCom improves text-to-image generation and image editing while preserving high-level semantics and fine-grained visual details.The paper reports that compressed semantic representations support unified generation and understanding without relying on VAEs.
A Implementation details
The implementation combines frozen SigLIP2 vision features, a multi-head self-attention compressor, and a FLUX.1-dev diffusion backbone. Training uses multi-resolution aspect-ratio buckets and jointly optimizes compression and diffusion components.
- Decoder Setup: UniCom uses SigLIP2 as the vision encoder, FLUX.1-dev as the diffusion backbone, and multi-head self-attention for semantic feature compression.The compressor bridges dense semantic features and the generative latent space.
- Decoder Setup: Training uses 33 aspect-ratio buckets spanning 1:4 to 4:1 around a 1024 × 1024 base resolution.The global batch size is 256, and training continues to 50K steps after high-quality reconstruction emerges within 10K steps.
- Understanding Setup: The language component uses Qwen2.5-7B-Instruct, while visual features enter the LLM through a two-layer MLP projector.The vision encoder remains frozen across the multi-stage training pipeline.
A.3 Ablation Setup.
The ablation setup uses reduced-scale data and simplified configurations to study reconstruction, generative dynamics, and multimodal understanding. Related work motivates continuous representations as alternatives to quantized visual tokens.
- Ablation Setup: Ablations train compression and diffusion modules on subsets of internal images and use filtered text-to-image data to measure convergence and generation quality.Understanding capabilities are evaluated on multiple multimodal benchmarks.
- Related Work: VAE–ViT hybrid encoders facilitate generation but create divergent representation spaces for semantic understanding and image synthesis.This motivates unified visual representations rather than separate semantic and generative features.
- Related Work: Quantizing continuous ViT features into discrete visual tokens inevitably loses information and struggles to express nuanced intra-image complexity.Continuous visual representations are presented as an alternative for sharing one representational space across understanding and generation.
C Impact of Dimensionality Compression on Training
Channel-dimensional compression preserves reconstruction fidelity while offering favorable training dynamics compared with sequence reduction and uncompressed features.
- Training dynamics: Reducing dimensions maintains higher reconstruction fidelity than reducing the sequence length.Figure 9 compares reconstruction loss and PSNR across compressed representations.
- Training dynamics: As data and compute scale, the performance gap between dimension compression and sequence reduction becomes negligible.Dimension compression has slower initial convergence before rapidly matching the uncompressed baseline.
D More Reconstruction Results
UniCom’s channel-compressed reconstructions preserve fine-grained image structures that prior unified tokenizers often lose, including high-frequency details.
- Reconstruction quality: Prior semantic-encoder tokenizers often produce blurred text and distorted small objects, whereas UniCom preserves these fine-grained structures.The comparison covers qualitative reconstruction results in Figure 10.
E More Results of Image Editing
UniCom supports diverse image editing and generation scenarios, including instruction-based edits, multi-element composition, and knowledge-driven transformations while preserving semantic information and subject identity.
- Evaluation scope: UniCom evaluates across single-image editing, multi-element composition, and knowledge-driven intelligent editing scenarios.The qualitative evaluation spans Figures 13–15.
- Single-image editing: UniCom covers add, remove, extract, replace, style transfer, background, subject-driven, and controllable generation tasks.These capabilities are illustrated in the single-image editing results.
- Single-image editing: Its compressed continuous representation retains fine-grained information for precise, instruction-following edits.The supported controls include spatial or structural guidance.
- Multi-element composition: UniCom composes multiple reference subjects into coherent scenes while preserving each element’s identity.The representation preserves rich semantic priors and does not rely on VAE-based reconstruction.
- Intelligent image editing: UniCom achieves an overall score of 4.35/5 on WorldEdit-Test.The score covers diverse cause categories in the image-editing benchmark.
- Chain-of-thought generation: Chain-of-thought training enables the model to reason about visual transformations before generating the output image.During inference, the model can identify changed attributes, spatial relationships, and instruction sub-steps before generation.
G Limitation
UniCom’s main limitations concern the compression–fidelity trade-off and the substantial computational resources required for large-scale or high-resolution training.
- Compression and fidelity: Compression can still lose information, particularly fine-grained details in complex images.The authors identify the trade-off between compression and fidelity as an ongoing challenge.
- Scalability: UniCom requires substantial computational resources for large-scale datasets and high-resolution image tasks.The associated energy consumption may limit accessibility and sustainability in resource-constrained environments.