PaperlayerView exact revision on arXiv

Source-linked AI summary

LatentUM: Unleashing the Potential of Interleaved Cross-Modal Reasoning via a Latent-Space Unified Model

Jiachun Jin, Zetong Zhou, Xiao Yang, Hao Zhang, Pengfei Liu, Jun Zhu, Zhijie Deng

arXiv:2604.02097v1cs.CVcs.LG

TL;DR

Unified models have been limited by separate visual representations for understanding and generation, requiring pixel-space mediation for interleaved reasoning. LatentUM places modalities in a shared semantic latent space, achieving state-of-the-art performance among unified models while supporting self-reflective generation, cross-modal planning, and action-conditioned world modeling. Its current limitations include fixed-resolution generation, modest pretraining scale, a pixel-space rollout interface, and representation generality tied to one VLM.

  • Problem

    Existing unified models use disjoint visual representations that require pixel decoding between understanding and generation, limiting efficient interleaved cross-modal reasoning.

  • Method

    LatentUM represents modalities as shared semantic latent tokens, using quantized visual features to enable direct latent-space cross-modal reasoning and generation.

  • Results

    LatentUM achieves state-of-the-art performance among unified models on visual generation and cross-modal reasoning benchmarks, while supporting semantically coherent action-conditioned future-state prediction.

  • Takeaways & Limitations

    The shared semantic latent-space formulation supports flexible interleaved reasoning, self-reflective visual generation, and temporally grounded future-state prediction.

  • Takeaways & Limitations

    The model remains limited to fixed-resolution generation and modest pretraining scale, uses a pixel-space recurrent rollout interface, and has representation generality underexplored beyond one VLM.

Abstract

from arXiv · show

Unified models (UMs) hold promise for their ability to understand and generate content across heterogeneous modalities. Compared to merely generating visual content, the use of UMs for interleaved cross-modal reasoning is more promising and valuable, e.g., for solving understanding problems that require dense visual thinking, improving visual generation through self-reflection, or modeling visual dynamics of the physical world guided by stepwise action interventions. However, existing UMs necessitate pixel decoding as a bridge due to their disjoint visual representations for understanding and generation, which is both ineffective and inefficient. In this paper, we introduce LatentUM, a novel unified model that represents all modalities within a shared semantic latent space, eliminating the need for pixel-space mediation between visual understanding and generation. This design naturally enables flexible interleaved cross-modal reasoning and generation. Beyond improved computational efficiency, the shared representation substantially alleviates codec bias and strengthens cross-modal alignment, allowing LatentUM to achieve state-of-the-art performance on the Visual Spatial Planning benchmark, push the limits of visual generation through self-reflection, and support world modeling by predicting future visual states within the shared semantic latent space.

1 Introduction

LatentUM reframes unified models as systems for interleaved cross-modal reasoning rather than only visual generation. It uses shared semantic visual tokens and latent-space interactions to support planning, self-reflective generation, and world modeling.

  • Existing unified models focus mainly on visual generation and underperform task-specialized models on those tasks.
  • Interleaved reasoning tasks favor semantic correctness, but existing models mediate between understanding and generation through pixel decoding.This mediation is described as ineffective and inefficient and introduces codec bias and cross-modal misalignment.
  • LatentUM embeds modalities in a shared semantic latent space and represents visual information as discrete semantic tokens alongside language.The design targets semantic correctness rather than pixel-level fidelity.
  • Model behavior aligned quantization discretizes CLIP features while preserving vision-language prediction ability instead of pixel fidelity.
  • An autoregressive Transformer with Mixture-of-Modal Experts uses next-token prediction, modality-specific parameters, and self-attention for cross-modal interaction.The architecture also inherits visual understanding capability from off-the-shelf vision-language models.
  • LatentUM achieves state-of-the-art performance among unified models on Visual Spatial Planning, GenEval, and GenEval2, while also supporting semantically coherent action-conditioned future-state prediction.The model is evaluated after post-training on visually grounded planning and self-reflective visual generation.

2 Related Work

Prior unified models commonly separate visual understanding and generation representations, while newer work explores interleaved reasoning and world modeling. LatentUM builds on these directions by predicting future visual states in a shared latent space.

  • Early unified models extend pretrained language models with discrete visual tokens for multimodal generation.Examples include unified transformers using next-token prediction and hybrid autoregressive or diffusion approaches.
  • Prior work distinguishes semantic features for visual understanding from pixel-detail features for generation, motivating separate encoders or dual vocabularies.
  • Interleaved multimodal reasoning includes explicit intermediate visual steps and implicit reasoning in latent spaces.ThinkMorph, Uni-CoT, and UniCorn are identified as especially relevant approaches.
  • World models predict future environmental states conditioned on past observations and actions, aligning with unified understanding-and-generation objectives.
  • Recent unified models demonstrate world-modeling capabilities through next-token prediction or action-conditioned video pretraining, and LatentUM adopts this perspective.

3 Method

LatentUM replaces pixel-space mediation with shared semantic visual tokens and combines modality-specific processing with cross-modal attention. Its tokenizer preserves understanding-relevant semantics, while an optional decoder renders pixels without making pixel fidelity the core training objective.

  • Shared semantic representation: LatentUM unifies visual understanding and generation in a shared semantic latent space, allowing generated visual content to be used for cross-modal reasoning without pixel-space mediation.This addresses the feature-space mismatch that otherwise requires decoding visual features into pixels and re-encoding them for understanding.
  • Visual tokenizer: CLIP features provide language-aligned visual representations, and MBAQ discretizes them by preserving VLM prediction behavior rather than reconstructing pixel details.The tokenizer minimizes the KL divergence between VLM output distributions for original and quantized features.
  • Visual tokenizer: The quantizer produces a discrete code matrix Z with C code indices per visual token from a codebook containing K × C entries.The resulting effective vocabulary is K^C, while the de-quantized features serve as a semantic approximation of the original CLIP features.
  • Mixture-of-Modal Experts: MoME uses parallel understanding and generation branches with separate feed-forward and projection parameters but shared self-attention for cross-modal information flow.Routing uses a special <BOI> token: following tokens use the generation branch, while other tokens use the understanding branch.
  • Decoupled pixel decoder: A separately trained diffusion decoder optionally maps quantized semantic features to pixels, keeping the core model focused on semantics rather than pixel fidelity.The decoder recovers pixel-level content despite the quantizer discarding fine-grained details less relevant to understanding.
  • Interleaved reasoning: Multi-frame interleaved reasoning processes visual tokens through both MoME branches with an attention mask, enabling all visual states to be trained in one forward pass.During pre-training, the model is restricted to text-only conditioning contexts because of data availability, although its architecture supports interleaved multimodal conditioning.

4 Experiments

LatentUM is evaluated across multimodal understanding, visual generation, interleaved reasoning, planning, and world modeling. Its shared semantic representation supports strong benchmark performance, self-reflective generation, spatial reasoning, and action-conditioned visual dynamics.

  • Visual Understanding: LatentUMBase preserves strong visual understanding under continuous features, while quantized features cause relatively small degradation across most multimodal benchmarks.On some benchmarks, quantized LatentUM even outperforms baselines using original continuous features, indicating semantically meaningful representations.
  • Visual Generation: LatentUMBase generates semantically coherent images across diverse prompts, with emergent legible in-image text enabled by shared visual-language tokens.The text-rendering capability is highlighted in the gallery’s last column.
  • Visual Generation: 0.85 overall score on GenEval: LatentUMBase outperforms all unified models on compositional instruction-following despite using the smallest training-data amount.The result is attributed to semantically aligned visual features induced by MBAQ rather than pixel-reconstruction-oriented features.
  • Interleaved Cross-Modal Reasoning: LatentUM’s fine-grained visual-textual planning yields a substantial performance advantage, supporting deeper interleaved reasoning through unified semantic representations rather than pixel-space bridging.After each textual action, the model generates an updated visual representation that conditions the next action; the training set contains 18,000 filtered samples.
  • World Modeling: LatentUM supports action-conditioned world modeling by predicting latent semantic states, producing temporally consistent future frames and achieving ATE 1.34 and RPE 0.34.The model supports zero-shot world simulation from free-form textual instructions and remains competitive with the specialized NWM baseline.
  • Ablations: MBAQ consistently outperforms reconstruction-based quantizers across five visual understanding benchmarks, showing the value of incorporating next-token prediction behavior during quantizer training.The alternatives directly reconstruct continuous semantic features using MSE or cosine-similarity objectives.

5 Conclusion and Future Work

LatentUM unifies modalities in a shared semantic latent space, making generated visual tokens directly interpretable and supporting interleaved reasoning, generation, and preliminary world modeling. The paper also identifies fixed resolution, limited scale, pixel-space rollout interfaces, and underexplored representation generality as remaining limitations.

  • Conclusion: LatentUM embeds all modalities in a shared semantic latent space, eliminating pixel-space mediation and supporting flexible interleaved cross-modal reasoning.Generated visual tokens are directly interpretable by the model itself.
  • Conclusion: LatentUM achieves state-of-the-art performance among unified models on visual generation and cross-modal reasoning benchmarks.
  • Conclusion: The shared latent formulation also supports action-conditioned world modeling by predicting future visual states as semantic tokens.This extends the modeling scope from static reasoning to temporally grounded future-state prediction.
  • Future Work: Current limitations include fixed-resolution generation, modest pretraining scale, pixel-space recurrent rollout, and MBAQ alignment to a single VLM.The latter leaves the generality of the learned semantic representation underexplored.
  • Future Work: Future work targets larger data and models, variable resolution, longer context, improved long-horizon temporal consistency, and fully latent rollout pipelines.

A.1 MBAQ Tokenizer Training

MBAQ converts continuous CLIP features into discrete visual semantic tokens while prioritizing vision-language prediction ability over pixel fidelity. LatentUM then models these tokens autoregressively with modality-specific parameters and supports both guided and CFG-free generation.

  • MBAQ Tokenizer: MBAQ projects CLIP features from d=4096 to de=256, partitions them into C=8 chunks, and independently quantizes each chunk with K=2048 codes.The chunked design yields an effective vocabulary of K^C = 2048^8 from 16,384 total codes.
  • MBAQ Tokenizer: The training objective combines answer-position KL divergence with codebook commitment and entropy regularization losses.The entropy term encourages uniform code usage, while the commitment cost is β=0.25 and the entropy weight is α=0.1.
  • MBAQ Tokenizer: MBAQ uses a straight-through estimator for quantization gradients and EMA-based codebook tracking to restart dead codes.Codes with EMA usage below 0.03/K are identified as dead, with at most 64 reset per step.
  • LatentUM Architecture: The MoME generation branch adds dedicated vision projections, normalization layers, and MLPs to each transformer layer while retaining cross-modal self-attention.
  • LatentUM Architecture: The autoregressive vision head predicts C=8 codebook indices per visual position using a causal Transformer decoder and cross-entropy objective.Pretraining uses 32M text-image pairs at fixed 448×448 resolution, represented by 256 tokens.
  • LatentUM Architecture: MBAQ-aligned semantic tokens enable CFG-free generation at guidance scale s=1.0 because they share a semantic space with language.

A.3 Pixel Decoder Training

The pixel decoder maps MBAQ de-quantized embeddings into an MMDiT-based decoder and is trained with flow matching. At inference, it reconstructs 448×448 images in 25 denoising steps using the semantic embeddings as its sole conditioning signal.

  • Decoder Architecture: The pixel decoder replaces Stable Diffusion 3.5 Medium’s text conditioning module with a projector from 256-dimensional MBAQ embeddings to the MMDiT context dimension.
  • Decoder Training: Decoder training uses a flow matching objective with weighted mean squared error between predicted and target VAE latents.The timestep distribution is logit-normal with μ=0.0, σ=1.0, and mode scale 1.29.
  • Decoder Training: The decoder is pretrained on the same 32M BLIP3o text-image pairs used for MoME pretraining.The per-GPU batch size is 45.
  • Decoder Inference: At inference, the decoder uses 25 Euler denoising steps at fixed 448×448 resolution, conditioned only on MBAQ de-quantized features with s=1.0.

B Additional Analysis of Visual Generation with Self‑Reflection

LatentUM’s self-reflection generation uses GRPO and a strict verification-based reward, while CFG-scale analysis shows coherent images even without classifier-free guidance. The reward evaluates object existence, counts, attributes, and spatial relations jointly.

  • Self-Reflection Training: GRPO self-reflection training samples G=16 generations per prompt, normalizes advantages within groups, and uses PPO clipping with coefficient 0.1 and KL weight β=0.005.
  • Self-Reflection Training: Rollouts use guidance scale 1.0, temperature 0.9, and top-p=0.95, without classifier-free guidance.
  • Self-Reflection Training: Post-training updates only the visual-generation pathway, preserving the pretrained understanding backbone that supplies the self-reward signal.
  • Verification Reward: The self-reward decomposes each prompt into atomic visual checks covering object existence, count, color, material, pattern, and pairwise spatial relations.
  • CFG Analysis: LatentUM produces coherent images at CFG = 1.0 under identical prompts across CFG scales from 1.0 to 5.0.
  • Verification Reward: The reward evaluates verification questions by extracting candidate-option logits and applying a softmax over the restricted option set.
  • Verification Reward: The multiplicative reward is intentionally strict, favoring generations that satisfy object counts, attributes, and relations simultaneously.

B.3 Qualitative Comparison Before and After RL

Self-reward GRPO improves LatentUM’s adherence to compositional visual constraints, with especially clear gains on counting, attribute binding, and spatial relations.

  • Self-reward GRPO makes LatentUM follow compositional constraints more accurately, especially for counting, attribute binding, and spatial relations.The reward directly encourages semantic correctness rather than low-level visual fidelity.
  • After GRPO, LatentUM follows counting prompts more accurately than LatentUMBase.

B.4 Failure Cases of Detection‑Based Metrics

Detection-based metrics can label semantically correct generated images as failures because their external recognition pipelines are brittle. GenEval2 therefore uses a VLM judge that is better aligned with prompt semantics and less sensitive to visual variation.

  • Detection-based metrics can incorrectly mark qualitatively correct generated images as failures.Their external recognition pipeline may fail even when the image content is correct.
  • Detection-based evaluation is especially vulnerable for dense layouts, small objects, and visually unusual but valid renderings.
  • GenEval2 introduces a VLM-as-a-judge evaluator to better align scoring with semantic prompt content.Compared with detection-based scoring, the VLM judge is less brittle to visual variation.

C Fine‑Grained VSP Data Construction

The fine-grained VSP dataset converts procedurally generated, solvable FrozenLake mazes into interleaved action-and-image trajectories. Each target sequence alternates action prediction with rendered next-state supervision.

  • The VSP training set is procedurally generated from deterministic FrozenLake mazes.Mazes span four difficulty levels from 3 × 3 to 6 × 6 grids.
  • Breadth-first search filters out unsolvable mazes, while shortest-path intervals remove trivial instances and control planning horizon.Duplicate layouts are removed using their flattened grid strings.
  • Each trajectory renders intermediate states with the agent position and previously visited cells, highlighting the goal upon arrival.
  • The target response alternates actions with next-state images, ending with Done!.The t-th image slot is supervised by the rendered state after executing action a_t.

D Improving Decoding Consistency in Long‑Horizon Prediction

Long-horizon visual prediction requires temporal consistency in addition to semantic correctness. A decoder conditioned on the first-frame VAE latent provides a stable visual anchor and substantially improves consistency across extended predictions.

  • Semantic correctness alone is insufficient for long-horizon prediction because decoded frames can drift in object identity, layout, and appearance.
  • Reference-conditioned decoding adds the first frame’s VAE latent alongside target-frame semantic tokens.Semantic tokens specify what to render, while the reference latent anchors how the scene should look.
  • Reference-conditioned decoding produces much better cross-frame consistency than independent keyframe decoding.
  • The reference-conditioned decoder substantially improves consistency in long-horizon settings.It better maintains stable appearance over extended prediction horizons, reducing visual drift in rendered pixels.
Loading 2604.02097v1…