Source-linked AI summary
Pseudo-Unification: Entropy Probing Reveals Divergent Information Patterns in Unified Multimodal Models
Songlin Yang, Xianghao Kong, Anyi Rao
TL;DR
UMMs aim to combine language reasoning and image generation, but divergent modality behaviors and limited existing probes leave the causes of pseudo-unification unclear. The paper introduces information-theoretic probing of encoding and prompt–response dynamics, finding dual divergence across ten UMMs and stronger unification when both modalities share contextual prediction.
Problem
UMMs exhibit divergent response behaviors, while existing probing methods lack model-internal insight or overlook prompt–response dependencies.
Method
The paper proposes an information-theoretic framework that measures modality-specific prompt entropy and prompt–response conditional entropy in implicit Transformer representations.
Results
Across ten representative UMMs, pseudo-unification reflects modality-asymmetric encoding and pattern-split responses, with Harmon providing evidence of shared contextual-prediction logic.
Takeaways & Limitations
Genuine multimodal synergy requires consistency in information flow and aligned generative logic, not merely shared parameters.
Takeaways & Limitations
The UMM formulation assumes an implicit joint probability distribution P(X, Y) over visual and textual inputs.
Abstract
from arXiv · showhide
Unified multimodal models (UMMs) were designed to combine the reasoning ability of large language models (LLMs) with the generation capability of vision models. In practice, however, this synergy remains elusive: UMMs fail to transfer LLM-like reasoning to image synthesis and exhibit divergent response behaviors. We term this phenomenon pseudo-unification. Diagnosing its internal causes is important, but existing probing methods either lack model-internal insight or ignore prompt-response dependencies. To address these limitations, we propose an information-theoretic probing framework that jointly analyzes how UMMs encode inputs and generate outputs. Applied to ten representative UMMs, our framework reveals that pseudo-unification stems from a dual divergence: (i) Modality-Asymmetric Encoding, where vision and language follow different entropy trajectories, and (ii) Pattern-Split Response, where text generation exhibits high-entropy creativity while image synthesis enforces low-entropy fidelity. Only models that unify both sides (e.g., via contextual prediction) achieve more genuine unification, enabling stronger reasoning-based text-to-image generation even with fewer parameters. Our work provides the first model-internal probing of unification, demonstrating that real multimodal synergy requires consistency in information flow, not just shared parameters.
1. Introduction
UMMs were intended to combine language reasoning with image generation, but their modalities often remain behaviorally divergent, a phenomenon termed pseudo-unification. The paper addresses this gap with information-theoretic probing of internal encoding and prompt–response information flow.
- Motivation: UMMs were expected to unite LLM reasoning with vision-model image synthesis, but reasoning-based generation has not reliably transferred to images.Text generation emphasizes contextual plausibility, whereas text-to-image models favor fidelity to the prompt.
- Motivation: Pseudo-unification describes UMMs whose shared representation space does not prevent divergent response patterns across text and image generation.The paper illustrates this with text correctly retrieving “American flag” while image generation fails to produce it.
- Research gap: Existing probing approaches either lack model-internal insight or overlook dependencies between prompts and responses.Task-specific datasets and case-study benchmarks cannot explain why models internally unify differently, while prior model-internal analyses mainly examine prompt representations.
- Approach: The proposed framework probes prompt encoding through embedding and layer-wise hidden-state entropy across visual and linguistic modalities.This analysis reveals modality-specific differences in information preservation, compression, and possible representational bottlenecks.
- Approach: The framework also estimates prompt–response conditional entropy to characterize response behavior across layers.Classical entropy estimation is infeasible for high-dimensional, variable-length Transformer representations without explicit joint densities, motivating the RKHS reformulation.
- Findings: Across ten representative UMMs, pseudo-unification reflects modality-asymmetric encoding and pattern-split responses.Vision and language follow divergent entropy trajectories, while text remains high-entropy and creative and image synthesis remains low-entropy and fidelity-oriented.
2. Related Work
Related work evaluates UMM capabilities and studies representation geometry, but existing approaches do not fully probe multimodal unification. This paper extends analysis to both input representations and prompt–response dynamics.
- UMM evaluation: UMM evaluation includes joint comprehension, generation, and mixed-modality benchmarks, alongside narrower text-to-image benchmarks.Examples include MME-Unify, UniEval, MMMG, T2I-CoReBench, and WISE.
- UMM architectures: Architectural UMM approaches include native unified models and pipelines pairing a multimodal LLM with a diffusion model.Native models jointly produce text and image tokens, whereas pipeline models delegate image synthesis to a separate diffusion model.
- UMM evaluation: Bidirectionally coupled tasks and stepwise protocols evaluate whether understanding enhances generation and generation enhances understanding.These benchmarks show that co-locating capabilities in one model does not ensure effective cross-capability reinforcement.
- Representation analysis: Representation studies use probes and similarity measures to examine semantic or linguistic structure across neural-network layers.Recent work often identifies mid-depth Transformer layers as favorable for abstraction and links pre-training objectives to representational structure.
- Research gap: Prior representation research is largely confined to language models and prompt representations, omitting prompt–response dependencies and multimodal joint reasoning.The paper addresses this gap by probing both input representations and prompt–response dynamics in UMMs.
3. An Entropy-Probing Formulation for Unification Analysis in UMMs
The paper formalizes UMM unification as coherence in an implicit vision-language joint distribution and probes it through representation entropy and prompt-conditioned response uncertainty. Because Transformer embeddings lack explicit densities, it uses kernel-based matrix entropy and a non-parametric conditional-entropy proxy.
- 3.5. A Proxy for Conditional Entropy: The framework estimates prompt entropy, joint prompt-response entropy, and their difference to probe how representations encode information and how responses depend on prompts.The two levels respectively compare modality-specific encoding and layer-wise prompt-response dependencies.
- 3.1. Modeling UMMs via an Implicit Joint Distribution: UMMs are modeled as learning an implicit joint distribution over visual and textual inputs, with unification assessed through entropy-based measures of internal coherence.The formulation covers conditional multimodal operations such as image captioning and text-to-image generation.
- 3.2. Entropy and Conditional Entropy: Prompt entropy measures representational uncertainty and effective dimensionality, while conditional entropy measures residual output uncertainty given the prompt.Low conditional entropy corresponds to fidelity-driven generation; high conditional entropy corresponds to creative responses.
- 3.3. Challenges in Classic Entropy Estimation: Classical density-based entropy estimation is infeasible because UMMs expose neither explicit probability densities nor fixed-length, low-dimensional representations.The resulting gap motivates a non-parametric method based solely on representational geometry.
- 3.4. Matrix-Based Rényi Entropy: Matrix-based Rényi entropy quantifies information content from kernel similarity structure, enabling comparisons across heterogeneous modalities and variable-length embedding sequences.The method constructs a Gaussian-kernel Gram matrix, normalizes it by its trace, and computes entropy from the resulting probability matrix.
- 3.4. Matrix-Based Rényi Entropy: Entropy increases with the number of independent information clusters, validating the kernel-based measure as an indicator of representational diversity.The validation uses synthetic embedding sequences with progressively different cluster structures.
- 3.5. A Proxy for Conditional Entropy: The conditional-entropy proxy interprets additional structural complexity in the joint representation beyond the prompt representation as residual response uncertainty.It is computed from matrix-based Rényi entropies of joint and prompt kernel matrices.
4. Probing Framework and Setting
The probing framework analyzes prompt representations and prompt-response dependencies across layers, then evaluates ten UMMs on established text and image multimodal benchmarks.
- Probing Framework: Prompt-level probing compares text and image prompt entropy across layers to expose modality-specific encoding and representational asymmetries.Response-level probing estimates conditional entropy across layers to identify divergent response patterns.
- Model Setting: The study evaluates ten state-of-the-art UMMs spanning MoT-based, lightweight all-in-one, VQ-VAE-tokenized, and Flow Matching architectures.The evaluated models include BAGEL, Harmon, Janus-Pro, JanusFlow, Show-o, and their listed variants.
- Benchmark Setting: Text prompts come from T2I-CoReBench, while image prompts come from MMBench, covering reasoning and perception tasks across varied input formats.T2I-CoReBench contains 1,080 prompts; MMBench contains 3,217 images.
5. Prompt Representation
UMMs encode text and images according to architecture- and scale-dependent entropy dynamics rather than semantic structure, producing systematic cross-modal asymmetries. These asymmetric trajectories include modality-specific compression and convergence patterns that leave the nominally shared representation space heterogeneous.
- 5.1.1. Effect of Prompt Length on Embedding Entropy: Entropy increases with text prompt length, while absolute embedding entropy varies by architecture and stronger LLMs retain higher isotropy after fine-tuning.Models sharing an LLM backbone show similar entropy baselines, indicating a dominant backbone prior.
- 5.1.2. Effect of Prompt Length on Layer Entropy: Text layer entropy shows model-dependent early compression: large models often collapse entropy early, whereas smaller models preserve information through smoother oscillatory growth.Show-o2 (7B) instead reaches high entropy quickly on short prompts but delays entropy growth for longer prompts.
- 5.1.3. Effect of Prompt Type on Layer Entropy: Across text prompt types, embedding levels and layer-wise trajectories are nearly identical, indicating structure-agnostic encoding rather than preservation of reasoning cues.Large models show early entropy collapse, while smaller models maintain higher deep-layer entropy and more semantic diversity.
- 5.2. Image Prompt: Across image prompt types, layer-wise entropy trajectories are nearly identical despite differences in semantic density and structural complexity.The finding indicates that visual encoding is governed by architectural priors rather than semantic or cognitive demands.
- 5.3. Pseudo-Unification in Prompt Encoding: Across modalities, UMMs exhibit systematic asymmetries in initial entropy and convergence dynamics, revealing a heterogeneous representation space.These cross-modal differences persist despite architectural unification and are associated with divergent generative behaviors.
- 5.3. Pseudo-Unification in Prompt Encoding: BAGEL compresses text early while maintaining high, stable visual entropy; Harmon rapidly raises linguistic entropy while vision converges more gradually.BAGEL text later rebounds to a moderate 5∼6 plateau from an early collapse, while images begin near ≈9; Harmon stabilizes text around 7∼8.
- 5.3. Pseudo-Unification in Prompt Encoding: Show-o and Janus both produce early entropy surges and plateaus for text and images, despite using different image-generation objectives.Their cross-modal behavior is consistent even though their image-generation objectives differ.
- 5.3. Pseudo-Unification in Prompt Encoding: Within modalities, entropy dynamics are structure-agnostic, but across modalities, distinct geometric trajectories in initial levels, convergence speed, and stable states reinforce pseudo-unification.The reported misalignment likely underlies divergent downstream response patterns because vision and language do not readily conform to shared reasoning logic.
6. Response Pattern
Nearly all UMMs show divergent response patterns: text generation has higher conditional entropy than image generation, whereas Harmon exhibits cross-modal convergence. This contrast reflects creative text generation versus fidelity-oriented image synthesis and points to shared contextual prediction as a route toward unification.
- Nearly all UMMs exhibit higher conditional entropy for text generation than image generation, revealing significant cross-modal inconsistency.The analysis compares layer-wise conditional entropy, defined as response uncertainty given the prompt.
- Higher text entropy reflects creative generation from a broad semantic distribution, while lower image entropy reflects deterministic prompt fidelity.These patterns preserve modality-specific optimization objectives: LLM-like open-endedness for text and fidelity constraints for images.
- The creative-text versus fidelity-image split indicates that current UMMs retain modality-specific generative logic rather than achieving genuine unification.The paper identifies this dual-track response mechanism as a hallmark of pseudo-unification.
- 6.2. More Discussion on Harmon: Harmon is the exception: image entropy exceeds text entropy in early layers, but rising text entropy surpasses image entropy in final layers.This layer-wise convergence distinguishes Harmon from the other models.
- 6.2. More Discussion on Harmon: Harmon’s masked autoencoder for images parallels next-token prediction in text, giving both modalities a shared contextual-prediction inductive bias.Masked patches use visible visual context, while future tokens use prior linguistic context.
7. Conclusion and Future Direction
The paper concludes that shared parameters alone do not unify multimodal models because representation and response information flows remain divergent. It argues that future progress should prioritize information consistency and reconsider generative objectives rather than relying only on scaling or additional benchmarks.
- Current UMMs exhibit pseudo-unification: shared parameters coexist with divergent representations and response patterns across modalities.The disconnect is attributed to misaligned generative inductive biases, with creative text and fidelity-oriented vision.
- The paper argues that scaling architectures or curating more benchmarks will not suffice while underlying information flow remains fragmented.The conclusion redirects attention from multi-task performance toward multimodal synergy.
- Harmon demonstrates that more genuine unification is possible when both modalities share a contextual-prediction paradigm.This architecture-level exception supports aligning generative inductive biases across modalities.
- Future Direction: Future work should probe and design for information consistency, including reconsidering pre-training objectives.The paper also reports difficulty changing information patterns already learned from prompt inputs.