LatentUMM: Dual Latent Alignment for Unified Multimodal Models

Yinyi Luo; Wenwen Wang; Hayes Bai; Marios Savvides; Jindong Wang

Source-linked AI summary

LatentUMM: Dual Latent Alignment for Unified Multimodal Models

Yinyi Luo, Wenwen Wang, Hayes Bai, Marios Savvides, Jindong Wang

arXiv:2605.17766v1May 18, 2026cs.CV

TL;DR

Unified multimodal models can share representations yet remain inconsistent when reinterpreting their generated outputs. LatentUMM explicitly aligns modality and capability transformations, with experiments showing consistent improvements across multimodal benchmarks and architectures.

Problem
Unified multimodal models may generate high-quality outputs yet fail to preserve semantics when reprocessing them, because joint training does not align understanding and generation behaviors.
Method
LatentUMM constructs a refined shared latent space with dual modality-capacity alignment and rollout-based preference optimization for semantically consistent transformations.
Results
LatentUMM consistently outperforms strong baselines across generation, understanding, editing, and unified evaluation benchmarks and generalizes across diverse architectures.
Takeaways & Limitations
The results suggest that multimodal unification requires structured coordination of shared representations across capabilities, not shared representations alone.
Takeaways & Limitations
Performance may vary with embedding-model and hyperparameter choices, while the study primarily evaluates consistency rather than other aspects.

Abstract

from arXiv · show

Unified multimodal models (UMMs) achieve strong performance in both understanding and generation by learning a shared latent space, yet they often exhibit functional inconsistency between these two capabilities. We observe that this issue does not stem from a lack of shared representations, but from the absence of explicit alignment between the transformations that map into and out of the latent space. As a result, generation and re-encoding can follow inconsistent trajectories, leading to semantic drift under modality transitions. In this work, we propose LatentUMM, a framework that constructs an enhanced shared latent space to explicitly align these transformations and improve cross-modal consistency. LatentUMM consists of two stages. First, dual latent alignment enforces consistency at both the modality and capacity levels: cross-modal alignment uses a stronger embedding model to impose structured cross-modal semantics, while dual capacity alignment enforces bidirectional consistency under generation and re-encoding. Second, latent dynamics stabilization improves robustness via stochastic latent rollouts and preference optimization, favoring trajectories that better preserve semantic consistency. Experiments show that LatentUMM consistently improves multimodal consistency across diverse architectures. Code is available at: https://github.com/AIFrontierLab/TorchUMM/tree/main/src/umm/post_training/LatentUMM.

1 Introduction

UMMs share a latent space for multimodal understanding and generation but can process their own outputs inconsistently, causing semantic drift across modality transitions. LatentUMM addresses this by explicitly aligning modalities and bidirectional capabilities, then stabilizing latent trajectories through rollout-based preference optimization.

Problem: UMMs jointly learn a shared latent space for understanding and generation, yet often fail to preserve semantics when reinterpreting their own generated outputs.A model may generate an image matching a text prompt but produce mismatched or incomplete semantics when processing that image again.
Problem: Implicitly learned mappings into and out of the latent space are not explicitly coordinated, allowing repeated cross-modal transitions to produce semantic drift.The paper attributes this issue to the insufficiency of merely sharing a latent space and to the difficulty of interpreting, constraining, or verifying its learned structure.
Method: LatentUMM explicitly aligns modalities and bidirectional understanding-generation capacities within an enhanced shared latent space.Its dual latent alignment is presented as a two-step process that couples the transformations of understanding and generation rather than relying only on joint-training representations.
Method: Rollout and preference optimization explores multiple latent transformation trajectories and selects semantically consistent ones to improve robustness and stability.The strategy is designed to stabilize latent dynamics by favoring trajectories that better preserve semantic consistency.
Experiments: LatentUMM is model-agnostic and consistently improves multimodal consistency across diverse architectures.This claim is listed among the paper’s contributions under extensive experiments.

2 Related Work

Related work develops unified multimodal models that combine understanding and generation, while recent studies identify persistent inconsistency between these capabilities despite shared architectures and latent spaces.

Unified Multimodal Models: Unified multimodal models integrate multimodal understanding and generation within a single architecture.A dominant approach uses decoder-only autoregressive transformers trained on interleaved multimodal tokens.
Unified Multimodal Models: Hybrid generative frameworks combine autoregressive modeling with diffusion or flow-based components to improve visual generation and cross-modal alignment.
Inconsistency in UMMs: Despite shared architectures and latent spaces, UMMs show persistent inconsistency between understanding and generation.Models producing high-quality outputs may fail to preserve semantic consistency when processing their own generations, indicating weak bidirectional coherence.

3 Method

LatentUMM refines a pretrained UMM’s shared latent space to address functional inconsistency between understanding and generation. It combines dual latent alignment with stochastic rollout-based preference optimization to preserve semantic consistency across modalities and latent trajectories.

Motivation: LatentUMM explicitly refines the shared latent space because joint training alone does not guarantee consistency between understanding and generation.The method targets cases where understanding representations fail to induce faithful generation and generative trajectories deviate from consistent semantics.
Dual Latent Alignment: The stronger embedding model E∗ maps inputs into the same dimensionality d as the UMM latent space, enabling direct geometric comparison.This shared dimensionality supports structured alignment between the refined embedding space and the original latent representation.
Dual Latent Alignment: Dual latent alignment combines cross-modal alignment with bidirectional capacity alignment for understanding and generation.A stronger embedding model provides shared semantic geometry, while latent generation and re-encoding are constrained to preserve semantic identity.
Latent Dynamics Stabilization: Latent dynamics stabilization samples Gaussian perturbations, rolls each latent through generation and re-encoding, and scores trajectories by cosine similarity.Each rollout follows z(k) → ˆx(k) = G(z(k)) → ˆz(k) = ϕ(ˆx(k)), with s(k) = Sim(z, ˆz(k)).

4 Experiments

Across understanding, generation, and editing benchmarks, LatentUMM consistently improves performance and multimodal consistency over post-training baselines. Ablations and analyses attribute these gains to shared latent alignment, embedding quality, stabilized rollouts and decoding, with improvements transferring across architectures.

Main Results: LatentUMM consistently improves understanding across all evaluated benchmarks, with especially strong gains on comprehensive suites such as MME and MM-Vet.The results suggest latent consistency enhances global multimodal alignment rather than overfitting to specific task formats.
Main Results: LatentUMM outperforms the baseline on every generation benchmark, improving fine-grained entity and attribute generation and robustness across diverse scenarios.The gains indicate better fidelity for object details and properties, alongside more reliable multimodal outputs.
Main Results: LatentUMM consistently improves editing, with the largest overall gains in geometric mean and better Semantic Correctness and Perceptual Quality.These improvements reflect stronger adherence to intended semantic changes while preserving visual realism.
Alignment Analysis: Shared latent alignment improves unified performance beyond direct SFT, helping models preserve stronger semantic alignment when re-interpreting generated outputs.SFT can remain competitive on individual tasks but underperforms in the unified setting because generation and understanding are optimized independently.
Ablation Studies: Gemini embeddings achieve the strongest overall alignment performance, while rollout depth, noise, and decoding expose trade-offs between stability, exploration, and long-horizon aggregation.Shorter and longer rollouts benefit different benchmarks; noise 0.1 balances exploration and stability, whereas self-consistency provides the most stable decoding gains.
Analysis and Generalization: LatentUMM improves text-image alignment in the shared latent space and transfers consistently across Bagel, Janus-Pro, and Harmon.The cross-architecture results support latent-space consistency as a general principle, with larger gains on weaker baseline backbones.

5 Conclusion · Appendix

LatentUMM addresses inconsistency between understanding and generation in unified multimodal models by enforcing dual alignment within a refined shared latent space. Experiments show consistent gains across multiple multimodal benchmarks, while performance remains sensitive to design choices and broader objectives remain unexplored.

5 Conclusion: LatentUMM targets inconsistency between understanding and generation despite a shared common latent space.The paper identifies this inconsistency as a key limitation of unified multimodal models.
5 Conclusion: LatentUMM explicitly enforces dual alignment within a refined shared latent space.This is the proposed approach for addressing the identified limitation.
5 Conclusion: Extensive experiments evaluate LatentUMM across generation, understanding, editing, and unified evaluation benchmarks.The evaluation spans four multimodal capability and assessment categories.
5 Conclusion: LatentUMM consistently outperforms strong baselines and post-training methods across these evaluations.The conclusion reports consistent improvements relative to both strong baselines and post-training methods.
5 Conclusion: LatentUMM’s performance may vary with design choices such as embedding models and hyperparameters.This sensitivity is identified as a limitation despite the method’s consistency improvements.
5 Conclusion: The work primarily focuses on consistency, leaving extension to other aspects for future research.The limitation explicitly identifies broader objectives as an open direction.

A Consistency Analysis of Latent Transformations

The section introduces a cosine-based diagnostic for measuring semantic drift during repeated cross-modal transformations. Experiments show that LatentUMM reduces consistency error relative to the baseline, especially over longer transformation chains.

Consistency Diagnostic: The diagnostic measures semantic drift under repeated cross-modal transformations to test whether encoding and generation transformations are misaligned.It directly targets consistency beyond task-level benchmark performance.
Consistency Diagnostic: Each transformation generates an output from the current latent representation and re-encodes it with a shared embedding model.The process tracks latent representations across multiple transformation steps.
Consistency Diagnostic: Consistency error is defined using cosine similarity as a normalized semantic-drift measure, with lower values indicating higher consistency.An error of 0 corresponds to identical representations, so comparisons are interpreted relatively across models.
Experimental Setup: The evaluation compares Bagel with and without LatentUMM on 1,000 Text-to-Image-2M samples across text-to-image-to-text-to-image transformations and T ∈{1, 2, 3, 4}.Image-to-text transitions use the fixed prompt “Please describe the image in detail.”, and all representations use the same embedding model.
Results and Analysis: The baseline’s consistency error increases with transformation steps, whereas LatentUMM remains lower at every step and widens the gap as T increases.The results indicate accumulated semantic drift in the baseline and greater stability over longer transformation chains with LatentUMM.
Interpretation: The findings show that shared representations alone do not ensure consistent behavior, while explicitly aligned latent transformations reduce drift and stabilize latent trajectories.This provides direct empirical evidence from repeated cross-modal transformations in baseline UMMs.

B Efficiency and Complexity · C More Details for Experiments · C.1 Architecture and Training Modifications

LatentUMM adds modest, controllable training overhead while preserving the backbone’s dominant scaling behavior. Experiments use a pretrained UMM with frozen backbone components, a frozen semantic reference embedding model, and LoRA adapters on selected projection layers.

B Efficiency and Complexity: LatentUMM preserves the backbone’s dominant scaling behavior with only modest computational overhead.The additional latent-alignment and rollout terms contribute only a minor constant-factor increase relative to Cbase.
B Efficiency and Complexity: Latent projection and alignment cost scales as O(B · d), which is negligible compared with Cbase in practice.Transformer computation is dominated by attention and large hidden dimensions.
B Efficiency and Complexity: Sparse rollout scheduling makes its amortized training cost small when rollout is invoked infrequently.Rollout is controlled by trigger interval r and activation probability p.
B Efficiency and Complexity: ∼20 GPU hours is the additional single-H100 cost under a dense every-10-steps rollout schedule.Distributed execution reduces this cost to approximately 6 hours using 4 H100 GPUs, nearly linearly.
C More Details for Experiments: The implementation details cover architecture, training configuration, rollout design, and hyperparameter sensitivity on a pretrained UMM backbone.All experiments use the pretrained unified multimodal model unless otherwise specified.
C.1 Architecture and Training Modifications: The backbone combines modality-specific encoders Et and Ei, fusion module F, and decoder G to produce reconstructed outputs ˆx = G(z).Its computation is zt = Et(xt), zi = Ei(xi), and z = F(zt, zi).
C.1 Architecture and Training Modifications: Training updates only LoRA adapters on q_proj_moe_gen, v_proj_moe_gen, gate_proj, up_proj, and down_proj.All other parameters, including Et, Ei, F, G, and the frozen external embedding model E∗, remain frozen; E∗ provides fixed geometric supervision without inference participation.

C.2 Training Configuration · C.3 Latent Rollout Design · C.4 Hyperparameters

The method uses a two-stage training pipeline, followed by stochastic latent rollouts and hyperparameter selection to stabilize latent dynamics. Training separates dual-capability alignment from latent dynamics stabilization while using perturbed trajectories and stable parameter ranges.

C.2 Training Configuration: Stage I performs dual-capability alignment within the two-stage training pipeline.
C.2 Training Configuration: Stage II performs latent dynamics stabilization as the second training stage.
C.2 Training Configuration: Both stages use the same optimizer settings as Stage I.
C.2 Training Configuration: Gradient clipping with norm 1.0 is applied in both training stages.
C.3 Latent Rollout Design: Latent rollout design samples K perturbed latent trajectories using z^(k) = z + ϵ^(k), with ϵ^(k) ∼ N(0, σ^2I).The trajectories are indexed by k = 1, . . . , K.
C.3 Latent Rollout Design: Rollouts are performed every 10 training steps.
C.4 Hyperparameters: Stable performance is observed for λ1 ∈ [0.05, 0.1], λ2 ∈ [0.05, 0.1], and σ ∈ [0.03, 0.07].These values are reported within the hyperparameter sweep.

C.5 Sensitivity Analysis … D Failure Case Study

Sensitivity analysis identifies moderate settings as preferable for consistency strength, trajectory ranking, and exploration, while excessive values destabilize or over-constrain training. Inference matches the original UMM without added latency, and rollout computation is training-only on NVIDIA H100 GPUs.

C.5 Sensitivity Analysis: Moderate λ1 offers the best semantic stability–diversity trade-off, whereas small values weaken cross-capability coupling and large values over-constrain the latent space.λ1 controls consistency strength.
C.5 Sensitivity Analysis: Moderate λ2 improves local smoothness; λ2 = 0 reduces alignment to pointwise behavior, while large values destabilize training through noisy preference signals.λ2 controls trajectory-level ranking.
C.5 Sensitivity Analysis: σ = 0.05 provides the best stability, while small exploration gives insufficient local coverage and large exploration exits the semantic manifold.σ controls the exploration radius.
C.6 Inference-time behavior: At inference, no rollout, preference sampling, or external embedding computation is used, and the model is identical to the original UMM.The additional alignment machinery is not activated during inference.
C.7 Compute and Hardware: Rollout computation runs only during training, scales linearly with rollout frequency and sample count, and introduces no additional inference-time latency.All experiments use NVIDIA H100 GPUs.

D.1 Failure Modes of Stochastic Latent Exploration … E.1 Image generation

LatentUMM’s latent-space alignment has two recurring failure modes: noisy rollouts can cause semantic degradation, while excessive consistency weighting can collapse output diversity. Quantitative analyses show bounded rollout benefits and a consistency–diversity trade-off, while qualitative image generation improves structural fidelity and realism over Harmon-1.5B.

D.1 Failure Modes of Stochastic Latent Exploration: LatentUMM exhibits instability under stochastic exploration and representational collapse under over-constrained latent dynamics across datasets and backbones.These reflect trade-offs among stability, expressiveness, and robustness.
D.1 Failure Modes of Stochastic Latent Exploration: Large σ or insufficient K can make stochastic rollouts leave the semantic manifold, destabilizing object counts, spatial arrangements, and environmental consistency.Coarse structure may remain intact while fine-grained grounding is partially lost.
D.1 Failure Modes of Stochastic Latent Exploration: Excessive λ1 prioritizes consistency over diversity, producing conservative embeddings and overly similar or near-identical outputs across sampling attempts.The resulting latent representation has reduced effective entropy and favors reconstruction fidelity.
D.2 Quantitative Analysis of Failure Modes: The quantitative analysis evaluates rollout-induced degradation and alignment collapse using relative changes against each metric’s baseline.It covers perturbation scale σ and consistency weight λ1.
D.2.1 (1) Effect of rollout perturbation scale: Moderate perturbation σ = 0.1 improves consistency and generation quality by approximately 4–5%, whereas excessive noise causes up to 19% loss.This establishes a non-monotonic effect with benefits only within a bounded perturbation regime.
D.2.2 (2) Effect of consistency weight: Increasing λ1 improves reconstruction consistency by +5.8% to +8.6% while reducing output diversity by up to -7.7%.The comparison uses λ1 = 0.1 as the reference setting.
D.2.2 (2) Effect of consistency weight: Rollout has non-monotonic performance effects, while consistency regularization monotonically increases reconstruction consistency at the cost of reduced diversity.Together, the results expose an inherent trade-off in latent-space optimization.
E Qualitative Results / E.1 Image generation: Across six text prompts, LatentUMM outperforms Harmon-1.5B in multi-object handling, complex attributes, spatial layouts, structural fidelity, and textural realism.It correctly arranges three wheelchairs in a tidy row, addressing baseline composition errors.

E.2 Image Understanding · F AI Assistants Usage

LatentUMM’s qualitative image-understanding examples span diverse technical domains and show more accurate, better-grounded interpretations than the baseline. AI assistants supported manuscript language refinement, while authors retained responsibility for experimental design and methodology.

E.2 Image Understanding: LatentUMM is evaluated qualitatively across chemistry, mechanical engineering, music theory, public health, energy systems, and geography.These examples illustrate multimodal understanding across diverse domains.
E.2 Image Understanding: LatentUMM consistently produces more accurate responses than the baseline across the illustrated domains.The examples emphasize improved visual grounding and interpretation of structured visual information.
E.2 Image Understanding: The strongest gains appear in domain-specific reasoning over charts, diagrams, and scientific illustrations.LatentUMM improves visual grounding and the interpretation of structured visual information.
E.2 Image Understanding: The baseline often fails through incomplete visual comprehension or incorrect alignment between visual elements and domain knowledge.These errors limit its ability to answer the illustrated questions correctly.
E.2 Image Understanding: LatentUMM mitigates these failures by integrating visual cues more faithfully with contextual understanding, yielding more correct answers.This mechanism addresses the baseline’s visual-comprehension and alignment errors.
F AI Assistants Usage: AI assistants were used mainly for language refinement, clarity, and organization when preparing the manuscript.The authors made the experimental design and methodological choices.

G Broader Impact

LatentUMM may improve the reliability, accessibility, and usability of unified multimodal systems, while also increasing over-trust and misuse risks. Its benefits and risks reflect existing multimodal-model concerns, potentially amplified by greater consistency and fluency.

Positive impacts: Improved consistency can make multimodal systems more reliable in assistive technologies, education, and content creation, reducing contradictory outputs, confusion, and potentially improving user trust.These benefits arise from better alignment between understanding and generation in interactive settings.
Positive impacts: Training-time alignment can reduce deployment costs compared with expensive inference-time orchestration, improving access to advanced multimodal reasoning in resource-constrained environments.The accessibility benefit follows from shifting alignment costs from inference to training.
Risks and limitations: More coherent and convincing outputs may increase user over-trust when models are incorrect, especially in high-stakes domains requiring verification.Consistency can make erroneous outputs appear more credible without guaranteeing correctness.
Risks and limitations: Scaled generation of synthetic multimodal content can support creativity and productivity but may also spread misleading or manipulated content when misused.This risk is shared with other generative models.
Risks and limitations: LatentUMM introduces no fundamentally new risks beyond existing multimodal models, but improved consistency and fluency may amplify both benefits and misuse, motivating factual grounding, transparency, and safeguards.The paper encourages combining consistency with these measures to support responsible deployment.