PaperlayerView exact revision on arXiv

Source-linked AI summary

Multimodal Flow: Unified Flow Modeling of Language and Vision in Embedding Spaces

Hongyuan Tao, Xinggang Wang, Lianghui Zhu, Yongkang Li, Yunchao Wei, Bin Feng, Shaoyu Chen, Qian Zhang, Chang Huang, Kai Yu

arXiv:2609.40362v1cs.CV

TL;DR

Unified multimodal modeling has yet to bring continuous language and vision representations under a shared generative process. Multimodal Flow models ordered text and image hyperchunks with a shared chunk-causal flow backbone. MF-1 achieves competitive multimodal results and outperforms discrete and hybrid architectures in controlled comparisons.

  • Problem

    Existing unified approaches either quantize images, limiting visual fidelity to tokenizer detail, or retain continuous images while using modality-dependent objectives and sampling.

    Source: §1 · pp. 1–2

  • Method

    Multimodal Flow represents text blocks and images as ordered hyperchunks and models them with a shared chunk-causal flow backbone, joint attention and modality-specific feed-forward networks.

    Source: §1 · pp. 2–3

  • Results

    MF-1 outperforms fully discrete and hybrid architectures on evaluated tasks; with 150B pretraining tokens, it scores 75.3 average across VQAv2, MMBench and POPE.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The results support continuous hyperchunk modeling as a flexible approach to multimodal pretraining that learns transferable representations across language and vision.

    Source: §4 · p. 10

    Extending the approach to longer interleaved sequences, video and other structured modalities remains future work.

    Source: §4 · p. 10

Abstract

from arXiv · show

We present Multimodal Flow, a fully continuous generative model of language and vision. Most unified multimodal models either model both language and quantized images as discrete tokens or combine discrete language prediction with continuous image generation. The former introduces a visual quantization bottleneck. The latter requires modality-dependent objectives and sampling procedures. Fully continuous modeling avoids these trade-offs and enables a shared generative process, but remains underexplored for multimodal pretraining. Multimodal Flow introduces a unified continuous architecture that integrates multimodal continuous representations with a shared chunk-causal flow backbone. It organizes text blocks and images as ordered continuous hyperchunks, preserving textual token order and visual spatial structure. The backbone learns a single vector field over these hyperchunks through Flow Matching. Joint attention enables cross-modal interaction, while modality-specific feed-forward networks process each modality. The model predicts multiple target chunks in parallel during training and generates hyperchunks sequentially at inference. We instantiate MF-1 and pretrain it on multimodal data. Across 0.6B, 1.2B, and 1.6B scales, continued pretraining consistently improves multimodal modeling. With only 150B pretraining tokens, MF-1 achieves an average score of 82.8 across GenEval and DPG-Bench and 75.3 across VQAv2, MMBench, and POPE, remaining competitive with unified models trained on substantially more data. Under matched data, optimization, and parameter budgets, Multimodal Flow further outperforms representative hybrid and discrete models. These results establish continuous chunk-based embedding flow modeling as a new fully continuous paradigm for unified multimodal modeling. The related code and model are publicly released at https://github.com/hustvl/Multimodal-Flow.

1 INTRODUCTION

Unified multimodal models have struggled to combine continuous visual representations with a shared generative process for language and vision. Multimodal Flow uses ordered hyperchunks and a shared chunk-causal flow backbone; MF-1 shows competitive multimodal performance and downstream transfer.

1 INTRODUCTION

  • Discrete models risk losing visual detail through quantization, while hybrid models retain continuous visual states but use modality-dependent objectives and sampling procedures.Neither paradigm combines continuous visual states with a common generative process across modalities.Source: §1 · p. 2, Figure 1 · p. 2
  • A causal framework for task-defined multimodal sequences trained on text-only, image-only and bidirectional paired data through one continuous objective remains underexplored.The introduction frames this as a gap beyond systems centered on joint denoising or predefined cross-modal routes.Source: §1 · p. 2
  • Ordered hyperchunks preserve text-token order and visual-grid structure, while a shared chunk-causal flow backbone combines joint attention with modality-specific feed-forward networks.Training predicts target chunks in parallel; inference generates hyperchunks sequentially.Source: §1 · p. 2, Figure 2 · p. 3
  • 0.821 on GenEval, 83.44 on DPG-Bench, and a 75.3 average across VQAv2, MMBench and POPE accompany wins over fully discrete and hybrid architectures on evaluated tasks.The 1.6B MF-1 was pretrained on 150B tokens.Source: §1 · p. 2

2 MULTIMODAL FLOW

Multimodal Flow maps text and images to continuous representations, models them as ordered chunks with a shared chunk-causal flow backbone, and applies the same interface and objective to mixed pretraining and downstream finetuning.

2.2 CONTINUOUS MULTIMODAL REPRESENTATIONS

  • Modality-specific encoders form continuous text and image chunks that preserve token order and visual spatial layout, then decode generated representations back into text or images.The implementation uses frozen SigLIP 2 and T5-small encoders, a pretrained image decoder, and a separately trained text decoder; the decoders remain frozen during flow pretraining.Source: §2.2 · pp. 3–4

2.3.1 CHUNK-CAUSAL FLOW MODELING

  • The chunk-causal backbone exposes preceding chunks, hides future chunks, and models positions within the current chunk jointly, with joint attention and modality-specific feed-forward networks.Task-specified chunk order and target modality let the same backbone operate across multimodal contexts.Source: §2.3.1 · p. 4

2.3.2 PARALLEL FLOW MATCHING TRAINING

  • Multiple target chunks are predicted in parallel during training, with each target attending to preceding clean chunks and its own perturbed state but not future chunks or its clean counterpart.Sequence packing places logical sequences into a fixed maximum length, while identifiers prevent information exchange across sequence boundaries.Source: §2.3.2 · pp. 4–5, Figure 3 · p. 5

2.3.3 SEQUENTIAL CHUNK INFERENCE

  • At inference, the model initializes each next chunk from Gaussian noise, integrates the learned vector field from t = 0 to t = 1, and appends the generated chunk to the prefix.Chunk-causal attention enables KV caching of completed chunks; classifier-free guidance and task-defined prefixes support unimodal and cross-modal generation.Source: §2.3.3 · p. 5

2.4.1 MIXED MULTIMODAL PRETRAINING

  • Mixed pretraining samples task-specific chunk sequences and target sets for one backbone, keeping the continuous interfaces and Flow Matching objective fixed while varying chunk content, order, and target modalities.Text-only, image-only, and paired image–text sequences cover unimodal modeling and bidirectional cross-modal conditionals.Source: §2.4.1 · p. 6

2.4.2 DOWNSTREAM FINETUNING

  • Downstream finetuning changes task data and chunk sequences but retains the model interface and objective, reusing frozen modality codecs and a backbone initialized from mixed pretraining.For visual question answering, images and questions prefix answer targets; for text-to-image generation, prompts prefix the visual target.Source: §2.4.2 · p. 6

3 EXPERIMENTS

Experiments compare MF-1 with unified models, test scaling and mixed-pretraining transfer, and analyze its visual representations, cross-modal computation, and guidance settings.

3.1 EXPERIMENTAL SETUP

  • The evaluation uses a 1.6B-parameter backbone trained from scratch with 150B tokens, compares model scales under a common protocol, and includes published-result and matched-budget comparisons.MF-1 results in Tables 1–3 use the same checkpoint after 5B tokens of joint finetuning.Source: §3.1 · p. 6, §3.2 · p. 6, §3.5 · p. 9

3.2 COMPARISON ACROSS MULTIMODAL MODELING PARADIGMS

  • MF-1 has the strongest compositional generation performance and best long-prompt alignment among unified models, while surpassing comparable from-scratch models on understanding and remaining competitive with similarly sized LLM-initialized models.The main model uses a 1.6B-parameter backbone trained from scratch with 150B tokens; reported results use a checkpoint after 5B tokens of joint finetuning.Source: §3.2 · p. 6, §3.1 · p. 6
  • With matched data, optimization schedules, and trainable-parameter budgets, MF-1 achieves stronger image generation and visual understanding than fully discrete and hybrid architectures.The Transfusion-style hybrid is a particularly close reference, sharing MF-1’s visual pathway and backbone design while using autoregressive text modeling.Source: §3.2 · p. 7, Table 4 · p. 8

3.3 MIXED MULTIMODAL PRETRAINING AND MODEL CAPACITY

  • As training proceeds, all three model sizes improve on Flow Matching objective, language PPL, GenEval, and image-captioning CLIPScore, with clearer language-modeling and captioning gains at larger capacity.The 1.6B model maintains the strongest performance later in training.Source: §3.3 · p. 8, Figure 4 · p. 8

3.4 DOWNSTREAM FINETUNING FROM MIXED PRETRAINING

  • Mixed-pretrained 1.6B models gain on GenEval, DPG-Bench, and all VQA benchmarks over randomly initialized models when downstream token exposure is matched.The comparison matches the randomly initialized baseline’s downstream training tokens to the pretrained model’s combined pretraining and finetuning budget.Source: Table 5 · p. 8, §3.4 · p. 9

3.5.1 CONTINUOUS VISUAL REPRESENTATION SPACE

  • At the evaluated training budget, DINOv2 yields the highest image-generation scores, while SigLIP2 better balances generation and captioning with higher CIDEr and CLIPScore but lower GenEval and DPG-Bench scores.Reconstruction-oriented VAE representations and raw pixels perform weakly on both tasks in this setting.Source: Figure 5 · p. 8, §3.5.1 · p. 9

3.5.2 SHARED INTERACTION AND MODALITY-SPECIFIC COMPUTATION

  • Sharing FFNs degrades performance, particularly for image-conditioned text generation, while shared and modality-specific attention projections perform similarly overall.The results motivate shared attention projections for cross-modal interaction and modality-specific FFNs for language and vision.Source: §3.5.2 · p. 9, Table 6 · p. 9

3.5.3 BIDIRECTIONAL CLASSIFIER-FREE GUIDANCE

  • Image captioning performs best at CFG scale 3, whereas text-to-image generation performs best at scale 5 and declines at higher scales.MF-1 applies the same CFG mechanism to both generation directions, with guidance strength adjusted by output modality.Source: Figure 5 · p. 8, §3.5.3 · p. 9

4 CONCLUSION

The conclusion presents Multimodal Flow as a fully continuous language–vision model using ordered hyperchunks and a shared chunk-causal flow backbone. It frames this approach against discrete and hybrid models and identifies longer interleaved sequences and video as future directions.

4 CONCLUSION

  • Multimodal Flow organizes text blocks and images as ordered hyperchunks in a shared chunk-causal flow backbone, expressing multimodal understanding and generation under one continuous objective.The conclusion reports gains with increased model scale and training tokens, competitive generation and understanding, and downstream benefits from mixed pretraining.Source: §4 · pp. 9–10
  • Longer interleaved sequences, video, and other structured modalities remain future-work directions for continuous hyperchunk modeling.The conclusion also characterizes the framework as a flexible multimodal pretraining paradigm that learns transferable representations across language and vision.Source: §4 · p. 10

A.1 DISCRETE AND HYBRID DISCRETE–CONTINUOUS MULTIMODAL MODELING

  • Fully discrete models depend on visual-tokenizer quality, while hybrid models retain discrete language modeling alongside continuous visual generation and must coordinate different objectives and sampling procedures.Quantization can discard image information that the downstream backbone cannot fully recover.Source: §A.1 · p. 15

A.2 CONTINUOUS GENERATIVE MODELING OF LANGUAGE AND VISION

Prior work demonstrates continuous generation in language and vision, but the paper identifies gaps in how multimodal structure is represented and pretrained. Multimodal Flow addresses these gaps with ordered hyperchunks and one chunk-causal Flow Matching objective.

A.2 CONTINUOUS GENERATIVE MODELING OF LANGUAGE AND VISION

  • Prior continuous generative work spans visual latents and semantic representations, as well as language generation from token embeddings to compressed sentence- and block-level latents.Earlier multimodal systems include composable diffusion, joint denoising, and cross-modal flows.Source: §A.2 · p. 15
  • Prior multimodal systems often use reconstruction-oriented VAE latents or compact visual tokens and do not explicitly model token-ordered text and spatially structured images as ordered conditional units.Mixed pretraining of a randomly initialized continuous backbone on multimodal data remains largely unexplored.Source: §A.2 · p. 15
  • Multimodal Flow organizes contextual text embeddings and spatial semantic visual embeddings into ordered hyperchunks modeled with one chunk-causal Flow Matching objective.The paper’s scaling, transfer, and controlled comparisons support this formulation as a foundation for multimodal pretraining beyond cross-modal generation.Source: §A.2 · p. 15

B IMPLEMENTATION DETAILS

The implementation details describe the main MF-1 configuration and its frozen text and vision codecs, which are reused during flow pretraining and downstream finetuning.

B.1 BACKBONE CONFIGURATION

  • Table 7 summarizes the backbone and representation settings of the main 1.6B MF-1 configuration.The supplied configuration passage identifies the table’s scope but does not state its settings.Source: §B.1 · p. 16, Table 7 · p. 16

B.2 REPRESENTATION ENCODERS AND DECODERS

  • A frozen T5-small encoder maps text to normalized 512-dimensional latents, and a six-layer bidirectional Transformer predicts tokens in parallel within each eight-token block.The decoder is trained separately, then frozen during multimodal pretraining; at inference it converts generated text latents into tokens block by block.Source: §B.2 · p. 16
  • The vision pathway uses a frozen SigLIP2-so400m encoder with patch size 14; each 224 × 224 image yields 256 spatially structured patch embeddings of dimension 1152.A pretrained representation decoder reconstructs images from generated visual embeddings, and both modality codecs remain frozen during flow pretraining and downstream finetuning.Source: §B.2 · p. 16

B.3 FLOW TRAINING AND SEQUENCE PACKING

Flow training samples modality-specific timesteps for target chunks and converts predicted clean states to velocities near the clean endpoint. Sequence packing keeps examples distinct through attention masks.

B.3 FLOW TRAINING AND SEQUENCE PACKING

  • Image targets use α = 8 and text targets α = 6 when sampling shifted-logitnormal timesteps independently for each target chunk.Timesteps are sampled once per 256-token image chunk and once per 8-token text block.Source: §B.3 · p. 16
  • The flow path interpolates between the clean normalized state and noise, while velocity conversion clamps its denominator to at least 0.05.The conversion is used for numerical stability near the clean endpoint.Source: §B.3 · p. 16
  • Independent examples are packed into sequences of at most 32,768 model positions, with cross-example attention blocked and padding inactive.Each example retains a distinct sequence identifier, while chunk-causal attention permits prescribed within-example interactions.Source: §B.3 · pp. 16–17

C.1 TRAINING CONFIGURATIONS

The training configurations specify the main model’s data mixture and token budget, the datasets used for pretraining and finetuning, and controlled comparison settings. Architecture comparisons match key training conditions and parameter budgets while retaining each architecture’s native objective.

C.1 TRAINING CONFIGURATIONS

  • The scaling study follows 0.6B, 1.2B, and 1.6B variants at matched checkpoints, while controlled studies use distinct pretraining and finetuning budgets.Architecture comparisons use 1.6B models with 50B pretraining and 5B finetuning tokens; transfer comparisons use 150B pretraining and 5B finetuning tokens, versus 155B downstream tokens for the randomly initialized counterpart.Source: §C.1 · p. 17
  • Pretraining uses GPIC for image generation, LLaVA-OneVision-1.5 for image understanding, and Ultra-FineWeb-L3 for text-only modeling; finetuning uses separate text-to-image and VQA datasets.The image-only task uses GPIC images without paired text; text-to-image finetuning uses BLIP3o-60k, the LAION DALL-E 3 Discord dataset, and ShareGPT-4o-Image, while VQA primarily uses LLaVA-v1.5 instruction-tuning data.Source: §C.1 · p. 17
  • The main 1.6B model is pretrained from scratch on 150B tokens with a 70% text-only, 20% image-understanding, 9% text-to-image, and 1% image-only task mixture.It receives 5B finetuning tokens for downstream evaluation; Table 8 records the main and controlled experiments’ pretraining and finetuning budgets.Source: §C.1 · p. 17, Table 8 · p. 17
  • Architecture comparisons match data, supervised-token exposure, optimization, and trainable-parameter budgets while preserving each architecture’s native objective.Trainable backbones share 32 layers, width 1600, 25 attention heads of dimension 64, RMSNorm, and positional encoding conventions; Table 9 summarizes the compared representations, objectives, and parameter-sharing choices.Source: §C.1 · p. 17

C.2 BENCHMARKS AND METRICS

The benchmarks cover language modeling and captioning, multimodal understanding, and image generation, using task-specific metrics. The evaluation also distinguishes the compared architectures’ training objectives.

C.2 BENCHMARKS AND METRICS

  • The controlled architecture comparison distinguishes autoregressive cross-entropy from flow velocity MSE; parenthesized values in Table 9’s last column are FFN hidden dimensions.Source: Table 9 · p. 18
  • Captioning on the COCO Karpathy test split uses CIDEr and CLIPScore, while generated-text fluency is measured with GPT-2-large perplexity.GPT-2-large perplexity is an external fluency metric rather than the flow model’s likelihood.Source: §C.2 · pp. 17–18
  • Multimodal understanding is evaluated on POPE, MMBench, SEED-Bench, VQAv2, GQA, and OK-VQA.Source: §C.2 · p. 18
  • Image generation is evaluated with GenEval for compositional generation and DPG-Bench for long-prompt adherence; higher values are better for these metrics.Lower values are better for perplexity, while higher values are better for the other reported metrics.Source: §C.2 · p. 18

C.3 EVALUATION SAMPLING

Evaluation settings cover visual question answering and text-to-image generation. Text sampling steps apply per generated chunk, and a separate sweep varies classifier-free guidance.

C.3 EVALUATION SAMPLING

  • For text outputs, sampling steps apply to each generated chunk.Source: §C.3 · p. 18
  • Figure 5(b) varies the guidance scale in a classifier-free guidance sweep instead of fixing it to the listed values.Source: §C.3 · p. 18
  • Table 10 presents inference settings for visual question answering and image generation.Source: Table 10 · p. 18

Related papers

Loading 2609.40362v1…