PaperlayerView exact revision on arXiv

Source-linked AI summary

STARFlow2: Bridging Language Models and Normalizing Flows for Unified Multimodal Generation

Ying Shen, Tianrong Chen, Yuan Gao, Yizhe Zhang, Yuyang Wang, Miguel Ángel Bautista, Shuangfei Zhai, Joshua M. Susskind, Jiatao Gu

arXiv:2605.08029v1cs.CVcs.LG

TL;DR

Existing unified multimodal models either quantize images or combine causal text generation with iterative diffusion, leaving a mismatch between text and image generation mechanisms. STARFlow2 uses the Pretzel architecture to interleave a pretrained VLM and autoregressive flow under one causal Transformer structure, achieving strong results across multimodal understanding and image-generation benchmarks.

  • Problem

    Existing unified multimodal models either lose visual information through image quantization or combine causal text generation with structurally different diffusion-based image denoising.

  • Method

    STARFlow2 vertically interleaves a frozen pretrained VLM and TARFlow stream under the same causal mask, using a shared continuous latent space for multimodal modeling.

  • Results

    STARFlow2 achieves strong performance across multimodal understanding and image-generation benchmarks, including 0.82 on GenEval and 84.14 on DPG-Bench.

  • Takeaways & Limitations

    The results support autoregressive normalizing flows as a viable foundation for unified multimodal modeling with shared causal text-image generation.

  • Takeaways & Limitations

    The paper treats preserving pretrained VLM understanding and achieving high-fidelity continuous image generation as explicit design desiderata.

Abstract

from arXiv · show

Deep generative models have advanced rapidly across text and vision, motivating unified multimodal systems that can understand, reason over, and generate interleaved text-image sequences. Most existing approaches combine autoregressive language modeling with diffusion-based image generators, inheriting a structural mismatch between causal text generation and iterative visual denoising. We observe that autoregressive normalizing flows are autoregressive Transformers--sharing the same causal mask, KV-cache mechanism, and left-to-right structure as LLMs--making them the most natural paradigm for true unified multimodal generation. We present STARFlow2, built on the Pretzel architecture that vertically interleaves a pretrained VLM stream with a TarFlow stream via residual skip connections, both operating under the same causal mask. Combined with a deep-shallow flow design and a unified FAE latent space, STARFlow2 enables cache-friendly interleaved generation where both text and visual outputs directly enter the KV-cache without re-encoding. Experiments demonstrate strong performance across image generation and multimodal understanding benchmarks, validating autoregressive flows as a viable foundation for unified multimodal modeling.

1 Introduction

STARFlow2 addresses the lack of truly unified multimodal generation by combining continuous visual generation with language modeling under a shared autoregressive mechanism. Its Pretzel architecture interleaves a frozen pretrained VLM stream and a TARFlow stream through residual skip connections.

  • Motivation: Unified multimodal models aim to perceive, reason over, and generate interleaved text–image sequences for interactive editing and visual problem solving.
  • Limitations of Existing Approaches: Tokenizing images in a single language model sacrifices visual continuity because quantization introduces information loss and limits generation fidelity.
  • Design Requirements: A truly unified architecture must preserve pretrained VLM understanding, support high-fidelity continuous image generation, and use structurally unified causal generation.The causal mechanism should share the mask and KV-cache, use single-pass decoding, and avoid diffusion’s iterative denoising or re-encoding overhead.
  • Autoregressive Flows: Normalizing flows parameterized by causal Transformers generate continuous visual data left to right using the same causal mask, KV-cache mechanism, and autoregressive structure as LLMs.Their output head predicts flow quantities rather than discrete token logits.
  • STARFlow2: STARFlow2’s Pretzel architecture vertically interleaves a frozen pretrained VLM stream with a TARFlow stream through residual skip connections for multimodal understanding and continuous visual generation.The framework uses one autoregressive Transformer mechanism under a shared causal mask, enabling interleaved generation without quantization, iteration, or visual re-encoding.

2 Preliminaries

Unified multimodal generation targets a single model that understands and generates interleaved text–image sequences, while avoiding degradation of pretrained understanding and asymmetry between modalities. STARFlow2 addresses this with a shared FAE latent space and autoregressive normalizing flows parameterized by causal Transformers.

  • Unified Multimodal Generation: Unified multimodal models process interleaved sequences of discrete text tokens and continuous visual latents for both understanding and visual generation.They support image-conditioned text generation and text-conditioned image synthesis within one model.
  • Unified Multimodal Generation: The central integration challenge is adding visual generation without degrading pretrained VLM understanding or creating structural asymmetry between modalities.
  • Feature Auto-Encoder (FAE): STARFlow2 uses a Feature Auto-Encoder latent space as a compact continuous representation shared by multimodal understanding and normalizing-flow image generation.The FAE is trained on DINOv2-g/14 features, which the authors find better suited for generation than SIGLIP-based representations while retaining strong understanding performance.
  • Autoregressive Normalizing Flows: Normalizing flows learn invertible mappings between data distributions and simple priors, yielding generative models by sampling from the prior and inverting the mapping.Their likelihood objective includes a Jacobian term accounting for local volume changes.
  • Autoregressive Normalizing Flows: Autoregressive flows parameterize affine transformations with causal Transformers under a self-exclusive causal mask, making the process analogous to next-token prediction.STARFlow’s deep-shallow design uses a deep AF block for most capacity and shallow AF blocks to refine image generation; following left-to-right order gives it language-model-like causal structure.

3 STARFlow2

STARFlow2 combines a Pretzel architecture that interleaves VLM and TARFlow streams under one causal mask with a deep-shallow flow and progressive three-stage training. These components support unified multimodal understanding, generation, editing, and interleaved synthesis while preserving cache-friendly generation.

  • 3.1 Pretzel Architecture: Pretzel vertically interleaves a pretrained VLM and TARFlow stream through residual skip connections under a shared left-to-right causal mask.Both streams process the same sequence of text tokens and visual latents, while skip connections exchange information at every position.
  • 3.3 Multi-Stage Training: Training proceeds in three stages: text-to-image flow training, adapter alignment for multimodal understanding, and joint activation of skip connections across understanding, generation, editing, and interleaved tasks.Stage 1 trains TARFlow and shallow blocks with the VLM frozen; Stage 2 trains only the adapter; Stage 3 jointly optimizes the unified system.
  • 3.1 Pretzel Architecture: The VLM supplies semantic representations, while TARFlow predicts visual affine parameters autoregressively and performs causal sequence modeling for text.Zero-initialized projections preserve pretrained VLM and flow behaviors while enabling learned multimodal corrections.
  • 3.2 Deep-Shallow Flow: The deep-shallow flow factorizes visual generation so visual-only shallow blocks absorb local spatial complexity and the deep block models global structure and cross-modal dependencies.Because shallow blocks operate only on visual latents, they preserve the Pretzel architecture’s left-to-right causal structure and cache-friendly interleaved generation.
  • 3.3 Multi-Stage Training: During Stage 1, TARFlow performs Next Gaussian Prediction in visual-latent space, predicting each next latent’s Gaussian mean and scale from preceding multimodal context.Sampling uses the predicted mean and scale with Gaussian noise: u_n = µ_D(u_<n; c) + σ_D(u_<n; c) · z_n, z_n ∼ N(0, I).

4 Experimental Setup

STARFlow2 is trained in multiple stages on large-scale text-image data and evaluated across multimodal understanding and visual generation benchmarks. The setup uses frozen pretrained VLM and image-encoder components alongside flow-based visual streams.

  • Datasets: Stage 1 uses around 800M text–image pairs from an in-house dataset, CC12M, and JourneyDB to establish the text-to-image generation backbone.Stage 2 trains the visual adapter for multimodal understanding using a mixture of CC12M and Cambrian-7M.
  • Evaluation: Evaluation covers MME, SEED-Bench, MMBench, MMMU, GQA, and AI2D for multimodal perception, reasoning, visual reasoning, and diagram comprehension.Visual generation is evaluated on GenEval and another widely used benchmark named in the setup passage.
  • Model and Training Details: The model uses Qwen2.5-VL-7B-Instruct as the pretrained VLM and FAE trained on DINOv2-g/14 features as the image encoder.Both the pretrained VLM and FAE encoder remain frozen throughout all training stages.
  • Model and Training Details: The architecture follows STARFlow with a causal Deep TARFlow stream and two visual-only shallow TARFlow blocks.The setup also aligns flow-based visual latents with the VLM representation space.

5 Results

STARFlow2 preserves strong multimodal understanding while supporting text-to-image generation, editing, and cache-friendly interleaved generation. Joint multimodal training improves image-generation scores, while vertical skip connections provide substantial visual fusion and modest textual correction.

  • Multimodal understanding: STARFlow2 achieves strong performance on MME-P, GQA, SEED, MMBench, MMMU, and AI2D, preserving the pretrained VLM’s perception and reasoning capabilities.Evaluation uses 256 × 256 resolution.
  • Image generation: 0.82 on GenEval and 84.14 on DPG-Bench demonstrate STARFlow2’s text-to-image generation capability across instruction following and compositional alignment.GenEval evaluates object presence, counting, colors, attributes, and spatial relationships; DPG-Bench evaluates global, entity, attribute, and relation alignment.
  • Effect of joint training: 60.8% on GenEval and 3.6% on DPG-Bench are the relative gains from Stage 3 over Stage 1 after joint multimodal training.Stage 3 activates vertical skip connections and jointly optimizes multimodal data, without degrading the pretrained visual generation pathway.
  • Qualitative results: Qualitative results show that STARFlow2 follows editing instructions, modifies local attributes, adjusts visual content, and preserves overall scene structure during interleaved generation.The examples support its use as a unified multimodal generator with cache-friendly interleaved generation.
  • Vertical skip connections: 0.472 mean contribution ratio shows that the projected VLM stream supplies a substantial fraction of the fused visual representation, with near-zero cosine similarity indicating complementary information.This analysis uses 50 randomly sampled text prompts and measures the visual vertical skip connection.
  • Vertical skip connections: The textual skip connection has a much smaller contribution ratio, indicating that TARFlow lightly corrects the pretrained VLM representation while preserving language modeling capability.The result is consistent with using TARFlow to provide modest multimodal corrections.

6 Related Work

Related work contrasts autoregressive text generation with diffusion and flow-matching vision generation, motivating unified multimodal designs. STARFlow2 addresses this mismatch through Pretzel-based vertical interleaving and a shared continuous FAE latent space.

  • Generative Modeling Paradigms: Text generation is dominated by autoregressive LLMs, whereas visual generation is led by diffusion and flow-matching methods with iterative sampling.These paradigms are structurally distinct from single-pass autoregressive decoding.
  • Unified Multimodal Models: Autoregressive-diffusion multimodal models create structural asymmetry because text enters the KV-cache causally while images require iterative denoising and re-encoding.MoT instead routes modalities to separate feed-forward parameters, maintaining two subnetworks within one shell.
  • Unified Multimodal Models: Discrete unified approaches avoid hybrid autoregressive-diffusion designs but sacrifice continuous fidelity.STARFlow2 achieves true unification through the Pretzel architecture, which vertically interleaves modalities.
  • Visual Representations: Many unified models decouple understanding and generation representations, while recent work explores shared representations.STARFlow2 uses the FAE latent space for compact continuous latents serving both understanding and flow-based generation.

7 Conclusion

STARFlow2 bridges language models and normalizing flows through the Pretzel architecture’s shared causal Transformer mechanism. Its design preserves pretrained multimodal understanding, enables high-fidelity continuous image generation without quantization, and unifies text and vision without iterative diffusion denoising.

  • STARFlow2 is a unified multimodal model that bridges language models and normalizing flows.
  • Pretzel vertically interleaves a frozen pretrained VLM with a TARFlow stream through residual skip connections under one causal Transformer mechanism.
  • STARFlow2 preserves pretrained multimodal understanding, generates high-fidelity continuous images without quantization, and unifies modalities without iterative denoising.

A Limitations and Future Work · B Impact Statement

STARFlow2 shows promise as a unified multimodal foundation, but its staged training, reliance on FAE latents, and incomplete benchmark leadership leave substantial directions for improvement. The approach may enable more efficient and flexible interleaved multimodal systems with broader accessibility and communication benefits.

  • A Limitations and Future Work: The multi-stage pipeline integrating the pretrained VLM, FAE, adapter, and TARFlow adds complexity and may cause under-optimization.Future work should optimize all components end-to-end so visual representation and cross-modal fusion are jointly shaped by next-token prediction.
  • A Limitations and Future Work: Dependence on the pretrained FAE encoder limits image resolution, visual quality, fine-grained fidelity, and text rendering.Native pixel-level or patch-level embeddings could reduce reliance on an external visual tokenizer or autoencoder.
  • A Limitations and Future Work: Although STARFlow2 supports understanding, generation, editing, and interleaved text–image generation, it is not state-of-the-art on every benchmark.The paper identifies data scale, training stability, visual representation learning, and long-context interleaved generation as future priorities.
  • A Limitations and Future Work: Autoregressive normalizing flows remain a promising foundation for unified multimodal modeling through continuous visual generation, exact likelihood training, and cache-friendly causal decoding.This combines the paper’s proposed modeling direction with its stated architectural benefits.
  • B Impact Statement: The method generates text and visual latents under one causal framework for unified multimodal understanding and generation.The intended applications include interleaved text-image generation, image editing, and interactive visual reasoning.
  • B Impact Statement: Unified multimodal models could improve accessibility and communication by helping users express themselves.The impact statement frames the approach as contributing to more efficient and flexible multimodal systems.

C Implementation Details · C.1 Architecture Design

Table 5 specifies the STARFlow2 model and states that both the pretrained VLM and FAE encoder remain frozen.

  • C.1 Architecture Design: Table 5 provides the model specification for STARFlow2.The passage identifies this specification as part of the architecture design.
  • C.1 Architecture Design: The architecture includes a pretrained VLM.The pretrained VLM is explicitly identified in the model specification.
  • C.1 Architecture Design: The architecture includes an FAE encoder.The FAE encoder is explicitly identified in the model specification.
  • C.1 Architecture Design: The pretrained VLM is kept frozen during the model design.The passage explicitly states that the pretrained VLM remains frozen.
  • C.1 Architecture Design: The FAE encoder is kept frozen during the model design.The passage explicitly states that the FAE encoder remains frozen.
  • C.1 Architecture Design: The model specification treats the pretrained VLM and FAE encoder as frozen components.Both components are covered by the passage’s statement that they are kept frozen.

C.2 Training Details · D Qualitative Examples

STARFlow2 uses a shared training setup across experiments, with staged learning rates and training on 64 H100 GPUs. Qualitative results include text-to-image generation and image editing examples.

  • C.2 Training Details: STARFlow2 is trained on 64 H100 GPUs using a shared configuration across experiments.
  • C.2 Training Details: The training configuration uses batch_size=1024, AdamW, bf16 mixed precision, cosine scheduling, and weight decay of 1e-4.
  • C.2 Training Details: The learning rate is 1e-4 for Stage 1 and Stage 2 training, then 5e-5 for Stage 3.
  • D Qualitative Examples: Figures 8 and 9 present qualitative examples for text-to-image generation and image editing.
  • D Qualitative Examples: Figure 8 shows STARFlow2 text-to-image generation examples at 256 × 256 resolution.
  • D Qualitative Examples: Figure 9 shows image editing examples from STARFlow2.
Loading 2605.08029v1…