PaperlayerView exact revision on arXiv

Source-linked AI summary

FlowInOne:Unifying Multimodal Generation as Image-in, Image-out Flow Matching

Junchao Yi, Rui Zhao, Jiahao Tang, Weixian Lei, Linjie Li, Qisheng Su, Zhengyuan Yang, Lijuan Wang, Xiaofeng Zhu, Alex Jinpeng Wang

arXiv:2604.06757v3cs.CV

TL;DR

Existing multimodal generation separates linguistic control from visual reasoning, motivating a unified visual representation. FlowInOne converts conditions into visual prompts and learns image-in, image-out flow matching, with reported state-of-the-art performance across evaluated tasks. The authors also identify limits in unconstrained generalization, resolution, and multi-turn use.

  • Problem

    Text-dominant pipelines make language the central conditioning source while leaving multimodal understanding, editing, and generation fragmented across representation spaces.

  • Method

    FlowInOne renders textual and visual instructions onto image canvases and uses a unified flow matching model with dual-path spatially-adaptive modulation for generation and editing.

  • Results

    FlowInOne achieves state-of-the-art performance among open-source models across all evaluated tasks and remains competitive with leading commercial systems.

  • Takeaways & Limitations

    FlowInOne presents a shared visual-space foundation in which perception and generation coexist across diverse multimodal tasks.

  • Takeaways & Limitations

    Generalization remains limited in highly complex unconstrained scenarios, output is fixed at 256 × 256 pixels, and the approach is optimized for single-turn instructions.

Abstract

from arXiv · show

Multimodal generation has long been dominated by text-driven pipelines where language dictates vision but cannot reason or create within it. We challenge this paradigm by asking whether all modalities, including textual descriptions, spatial layouts, and editing instructions, can be unified into a single visual representation. We present FlowInOne, a framework that reformulates multimodal generation as a purely visual flow, converting all inputs into visual prompts and enabling a clean image-in, image-out pipeline governed by a single flow matching model. This vision-centric formulation naturally eliminates cross-modal alignment bottlenecks, noise scheduling, and task-specific architectural branches, unifying text-to-image generation, layout-guided editing, and visual instruction following under one coherent paradigm. To support this, we introduce VisPrompt-5M, a large-scale dataset of 5 million visual prompt pairs spanning diverse tasks including physics-aware force dynamics and trajectory prediction, alongside VP-Bench, a rigorously curated benchmark assessing instruction faithfulness, spatial precision, visual realism, and content consistency. Extensive experiments demonstrate that FlowInOne achieves state-of-the-art performance among open-source models across all unified generation tasks while remaining competitive with leading commercial systems, thereby establishing a new foundation for fully vision-centric generative modeling, in which perception and creation coexist within a unified continuous visual space. Our code and models are released on https://csu-jpg.github.io/FlowInOne.github.io/

1 Introduction

FlowInOne addresses the fragmentation of text-dominant multimodal generation by representing all conditions visually and using one image-in, image-out flow matching model. VisPrompt-5M and VP-Bench support unified training and evaluation across generation, editing, and instruction-following tasks.

  • Text-dominant pipelines let language control vision but leave vision unable to reason or generate independently, fragmenting representation spaces across tasks.
  • FlowInOne converts text and image conditions into visual inputs, replacing separate conditioning arrangements with a single image-in, image-out model.Traditional text-to-image uses a text encoder, while text-image-to-image requires both text and image encoders.
  • VisPrompt-5M pairs visual prompt canvases with target images across text-in-image generation, visual editing, and physics-aware instruction following.The dataset provides supervision as continuous visual evolution without task-specific modules or auxiliary channels.
  • VP-Bench evaluates instruction faithfulness, content consistency, visual realism, and spatial precision.
  • The authors report state-of-the-art performance across unified generation, precise editing, and physics-aware instruction following.

2 Related Works

Related work contrasts diffusion-based multimodal pipelines with flow matching and vision-centric conditioning. FlowInOne renders heterogeneous controls into a shared visual input to align semantics and geometry while avoiding specialized conditioning branches.

  • Flow matching learns a continuous transport map between distributions, reducing reliance on complex noise schedules while improving sampling efficiency and stability.
  • Conventional text- and image-conditioned systems use cross-attention, diffusion priors, adversarial frameworks, or external control channels for multimodal generation and editing.
  • These approaches commonly depend on explicit masks or task-specific interfaces for geometric and semantic control.
  • FlowInOne renders heterogeneous constraints such as text and arrows into one image input, aligning semantics and geometry within the visual domain.
  • The related-work discussion identifies modality gaps, tokenization artifacts, and Gaussian-noise dependence as limits on spatial precision.

3 Dataset and Benchmark

VisPrompt-5M implements the visual-prompt paradigm through image canvases paired with target images across diverse generation, editing, and physics-aware tasks. VP-Bench evaluates these capabilities with VLM, human, and tailored quantitative measures centered on instruction satisfaction and visual quality.

  • Dataset construction: VisPrompt-5M embeds textual and spatial instructions directly into input canvases paired with target images for unified image-in, image-out training.This removes auxiliary text channels while preserving geometric alignment.
  • Dataset construction: Its construction combines large-scale text and class generation data with approximately 1.6M text-in-image editing examples.
  • Dataset construction: Bounding-box editing retains 24K high-quality pairs after filtering from 45K examples, while visual-marker editing contributes 250K pairs.
  • Dataset construction: Doodle editing yields 1K manually inspected pairs, and force and trajectory understanding add physics-aware visual instruction data.Trajectory data contains 1.5K curated image pairs.
  • Benchmark and evaluation: VP-Bench uses VLM and human evaluation requiring simultaneous instruction faithfulness, content consistency, visual realism, and spatial precision.
  • Benchmark and evaluation: Tailored metrics include CLIP-IQA for visual realism, CLIP Score for semantic alignment, and DINOv3 Directional Similarity for fine-grained spatial and physical changes.

4 Method

FlowInOne represents multimodal generation as continuous visual evolution, converting instructions into visual prompts and transporting them to target images in a shared latent space. Its dual-path modulation adapts structural conditioning to distinguish text-to-image generation from source-image editing.

  • Flow Matching Formulation: Flow matching models visual instruction-to-image generation as continuous transport between source and target latent distributions.The source latent represents the unified visual instruction, while the target latent represents the image; inference deterministically solves an ODE.
  • Unified Visual Representation: Textual instructions and visual cues are rendered onto an image canvas to preserve spatial layouts and structural priors within one visual modality.A visual encoder extracts patch-level semantic features that jointly represent textual semantics and visual geometry.
  • Dual-Path Modulation: Dual-Path Spatially-Adaptive Modulation compensates for structural information lost during visual encoding and switches computation according to task type.The mechanism addresses missing fine-grained source-image structure while retaining a shared flow-matching formulation.
  • Task-Adaptive Conditioning: For text-to-image generation, the model bypasses cross-attention; for image editing, it injects source-image structure through cross-attention and adaptive token-level gating.The binary task indicator nullifies structural modulation for pure text inputs and activates spatial refinement when source images are present.
  • Unified Pipeline: FlowInOne uses a single image-in, image-out pipeline that maps unified visual prompts to target images without task-specific modules or auxiliary channels.Visual instruction tokens are mapped through a text-image VAE, while target images are encoded with an isomorphic frozen image VAE.
  • Inference: During inference, FlowInOne samples an initial visual-instruction latent, solves an ODE to obtain the terminal latent, and decodes it into the final image.This avoids additional noise scheduling and conditional branching while maintaining an isomorphic representation of the visual instructions.

5 Experiment Results

FlowInOne is evaluated through VP-Bench using automated and human assessments, quantitative metrics, qualitative comparisons, and ablations of compression, modulation, and training strategy. It performs strongly across unified generation tasks, spatial and physical controls, and open-source comparisons, while joint training and adaptive modulation improve results.

  • Evaluation setup: FlowInOne is evaluated on VP-Bench with VLM and human assessments, four-dimensional scores, quantitative metrics, qualitative comparisons, and ablations.The evaluation covers implementation, benchmark performance, fine-grained quality, visual comparisons, compression, modulation, and training strategy.
  • Benchmark results: 54.0%, 39.2%, 50.3%, and 44.9% are FlowInOne’s total success rates under Gemini3, GPT5.2, Qwen3.5, and human evaluation.These results are reported across the benchmark’s unified generation tasks.
  • Benchmark results: 3.42, 3.24, and 3.30 are FlowInOne’s best spatial precision scores under Gemini3, GPT5.2, and Qwen3.5.FlowInOne also shows strong instruction faithfulness and content consistency, while Nano Banana achieves higher visual realism.
  • Quantitative analysis: 48.7% is FlowInOne’s highest average DINOv3 Sim score, exceeding Nano Banana’s 47.3% and indicating strong fine-grained spatial and physical control.The reported gains include force and trajectory understanding and text-bbox editing, alongside comparable overall visual realism and semantic alignment with the commercial model.
  • Qualitative comparison: FlowInOne executes force, trajectory, text-bbox, marker, and doodle edits more precisely than baselines while preserving background consistency.Baselines struggle with physically plausible motion, spatial constraints, size specifications, and localized visual cues even after prompt enhancement.
  • Ablation studies: Dual MLP projection preserves semantic and spatial structure while maintaining pretrained priors better than the alternative compression strategies.The compression study compares MLP plus truncation, VAE expansion, and MLP plus MLP projection.
  • Ablation studies: 47.8% is achieved by joint training on all 5M samples, surpassing the two-stage approach’s 29.1% pass rate.The authors associate two-stage training with catastrophic forgetting and joint training with learning semantic, geometric, and physical tasks together.

6 Conclusion

The paper presents FlowInOne as a purely visual flow that unifies multimodal generation through a shared visual space and continuous transport between visual instructions and image states. With VisPrompt-5M, it reports state-of-the-art performance among open-source models and competitiveness with commercial systems across evaluated tasks.

  • Conclusion: FlowInOne reframes multimodal generation as a purely visual flow with continuous transport between visual instructions and image states.The framework uses a shared visual space and a single visual interface across diverse tasks.
  • Conclusion: VisPrompt-5M supports cross-task generalization under a single visual interface.The dataset is presented as supporting the unified visual-generation paradigm.
  • Conclusion: FlowInOne achieves state-of-the-art performance among open-source models and remains competitive with leading commercial systems in automated and human evaluations.The conclusion states this result across all evaluated tasks.

A.1 Error Analysis on human evaluation

Human evaluation identifies Instruction Fidelity as VP-Bench’s main error bottleneck, with error patterns varying substantially across task subsets.

  • Evaluation protocol: Human evaluators judged generated samples as qualified or failed, then assigned multiple failure dimensions to each failed sample.The four dimensions are Instruction Fidelity, Spatial Precision, Visual Realism, and Content Consistency.
  • Global error distribution: 56.2% of all tagged errors are Instruction Fidelity errors, making semantic adherence the primary bottleneck.Spatial Precision accounts for 20.5%, followed by Visual Realism at 13.5% and Content Consistency at 9.7%.
  • Subset-level error distribution: Instruction Fidelity errors dominate semantics-driven class-to-image and text bbox control tasks, reaching 85.7% and 80.0%, respectively.The subset-level breakdown shows that error distributions depend strongly on task type.

A.2 Robustness test

FlowInOne remains stable under moderate visual-instruction and resolution perturbations, while ablations show that textual semantics and visual spatial cues are jointly necessary for accurate editing.

  • Instruction perturbations: Style, length, and strong-blurring perturbations preserve stable generation, whereas random text corruption significantly degrades performance.The degradation under corrupted text indicates reliance on explicit semantic guidance rather than visual-context-only hallucination.
  • Resolution robustness: 256 × 256 and higher resolutions generally produce high-quality generations, while 128 × 128 causes noticeable declines in visual fidelity and instruction following.The tested resolutions are 128 × 128, 256 × 256, 384 × 384, and 512 × 512 pixels.
  • Visual instruction ablation: Only the complete Text & visual prompt configuration consistently generates correct, high-quality targets.Blank, Text only, and Visual prompt only inputs fail to provide the necessary combination of semantic and spatial control.
  • Visual instruction ablation: The ablation results show that in-image text supplies functional meaning, while graphical cues provide spatial and operational grounding.Text alone mainly erases overlaid text, whereas visual prompts alone produce chaotic results.
  • Evaluator agreement: Kendall’s W is 0.7422 and Fleiss’ κ is 0.6132, indicating aligned ordinal rankings and substantial binary pass/fail consensus across evaluators.Absolute success rates vary, but comparative conclusions remain stable across evaluators.

B.2 Data Analysis

VP-Bench and VisPrompt-5M operationalize vision-centric generation through diverse visual-instruction tasks, lightweight textual triggers, and curated image-based supervision spanning generation, editing, and physics understanding.

  • VP-Bench composition: VP-Bench contains 1,060 curated image pairs across eight task categories, led by Visual Marker Editing and Text-in-Image Editing.These two categories contribute 320 pairs and 290 pairs, respectively, together exceeding 57% of the benchmark.
  • VP-Bench composition: VP-Bench includes Force Understanding, Trajectory Understanding, Text BBox Control, Doodles Editing, Class-to-Image, and Text-to-Image beyond its two core editing categories.The benchmark therefore covers physics-aware reasoning, spatial layout, sketch-based control, and fundamental generation.
  • Visual instruction analysis: VP-Bench instructions use lightweight text dominated by visual pointers and operational verbs rather than dense descriptive adjectives.Examples include “arrow” and “pointed” alongside “change,” “turn,” and “swap.”
  • Visual instruction analysis: The benchmark uses text as a lightweight operational trigger while testing image-in, image-out visual instruction following.This design shifts cognitive load from lengthy textual descriptions toward explicit visual instructions.
  • VisPrompt-5M construction: VisPrompt-5M comprises approximately 5 million visual-instruction pairs with text, spatial layouts, and physical constraints rendered directly onto input canvases.Its eight fine-grained task categories span fundamental generation, unified image editing, and physics understanding.
  • VisPrompt-5M construction: The dataset integrates annotated trajectories and force arrows for physics-aware image-in, image-out generation.These visual elements encode precise object dynamics within the input canvas.
  • VisPrompt-5M construction: A multi-stage construction and filtering pipeline is intended to promote accurate spatial-visual logic rather than noisy dataset correlations.The pipeline includes MLLM synthesis, manual annotation, and filtering.
  • Unified image editing: Structural condition maps such as Canny edges are treated as source images, unifying condition-to-image generation with image editing.This framing integrates structured image pairs into the unified editing objective.

D.5 Dataset Composition and Detailed Statistics

VisPrompt-5M contains approximately 5M curated image pairs with multi-granularity coverage spanning semantic editing, spatial constraints, and physics-aware dynamics. Its composition combines broad user-intent diversity with structured and physically grounded supervision.

  • Dataset scale and curation: Approximately 5M high-quality image pairs remain after rendering and rigorous filtering across major task categories.Table 9 reports macro-level composition and retention volumes, while Table 10 reports filtering survival rates for selected complex tasks.
  • Semantic and stylistic coverage: Text-in-image editing data provides broad, long-tailed semantic and stylistic coverage, including frequent operations such as Color Alteration and Subject Addition.The distribution also includes rare edge-case operations, extending the range of user intents represented in the dataset.
  • Spatial reasoning and constraints: VisMarker supplies balanced, region-aware supervision across eight core categories, with Object Swap and Removal each ranging from 33K to 60K pairs.These structured editing subsets train adherence to local visual markers and spatial constraints rather than only global semantic changes.
  • Spatial reasoning and constraints: The Structured Editing subset emphasizes dense spatial translations, including Image-to-Sketch and Face Restoration, to enforce geometric and structural adherence.Figure 17 highlights this subset alongside specialized text-in-image editing operations.
  • Physics-aware dynamics: The Force & Trajectory subset includes curated classes such as balls_poke at ∼11K and wind at ∼9K, prioritizing high-fidelity kinematic dynamics over scale.It maps visual force vectors to consequences including motion blur, structural deformation, and trajectory extrapolation.
  • Design rationale: The dataset’s engineered distribution combines massive semantic coverage, structured marker supervision, and a curated physics subset for unified instruction-driven image generation.The intended result is a single model trained across semantic, spatial, and dynamic generation capabilities.

E Limitations and future work

The framework’s current scope is bounded by model capacity, training-data scale, output resolution, and single-turn interaction. Future work targets larger models and datasets, high-resolution generation, and multi-turn visual editing.

  • Scope boundaries: Generalization remains somewhat limited in highly complex, unconstrained scenarios, bounded primarily by the 1.2B-parameter model and training-data scale.The limitation is stated for the current benchmarked framework rather than as a universal failure claim.
  • Resolution: Generation is restricted to 256 × 256 pixels because of training-time computational constraints, limiting suitability for some high-fidelity creative workflows.The authors identify high-resolution generation as a future optimization target.
  • Interaction scope: The current approach is optimized for single-turn instruction execution, while continuous multi-turn interactive editing remains unexplored.Future work proposes extending the visual-centric paradigm to seamless multi-turn visual editing.
  • Dataset context: The dataset composition aggregates diverse public sources through multi-stage OCR and VLM filtering, with selected task categories characterized by filtering survival rates.These dataset statistics describe curation and composition rather than resolving the framework’s stated generalization and resolution limitations.

F.1 VLM evaluation

The VLM evaluation pipeline combines source images, generated outputs, and extracted text instructions, scoring instruction fidelity, content consistency, visual realism, and spatial precision. It supplements automated assessment with quantitative similarity metrics and expert human inspection.

  • VLM evaluation pipeline: The evaluation pipeline takes a source image, generated output image, and plain-text generation instruction as its primary inputs.Source images are categorized as text-only canvases or annotated real-world images.
  • Evaluation dimensions: VLM evaluators score Instruction Fidelity, Content Consistency, Visual Realism, and Spatial Precision on a 1-5 scale.The criteria cover semantic response, preservation of unedited regions, image quality, and compliance with marked spatial boundaries or directions.
  • Pass criteria: A generated image passes only if Instruction Fidelity is ≥3.0, the overall average is ≥3.0, and no dimension scores ≤2.0.The evaluator returns structured JSON containing analysis and a PASS or FAIL verdict.
  • Quantitative metrics: Directional CLIP Similarity measures whether the visual change aligns with the textual description of the edit.It compares directional changes derived from captions of the input and generated images using CLIP text and image encoders.
  • Quantitative metrics: DINOv3 Directional Similarity targets dense spatial and physical transformations while checking against unintended background artifacts.The metric uses edit displacement vectors and is intended to capture changes that CLIP may not represent as precisely.
  • Human evaluation: Human evaluation uses 10 independent experts to inspect a stratified random subset of 250 VP-Bench pairs and make binary Pass or Fail judgments.The subset includes a balanced representation of 25 samples per subset, with failed samples assigned specific dimensions where possible.
  • Model context: FlowInOne’s main trainable component is the 1108.40M-parameter Flow Backbone, while its Visual Encoder and Image VAE remain frozen.The frozen components are reported as reducing training cost while preserving visual representation capacity.

G.3 Hyperparameter ablation

The ablations show that instruction adherence depends on guidance, sampling steps, and loss-weight balance. The selected defaults are CFG = 7, 50 sampling steps, β1 = 0.01, and β2 = 1.

  • CFG scale: CFG scale produces an inverted U-shaped PASS Rate, peaking at 54.0% with CFG = 7.Lower scales yield 38.5% and 44.8% at CFG values 1.1 and 1.5, while excessive guidance at CFG = 9 reduces performance.
  • Sampling steps: A 50-step sampling schedule reaches the best reported PASS Rate of 54.0%, whereas 10 steps yield 25.7%.Performance improves with additional steps and begins to converge, motivating 50 steps for primary evaluations.
  • Loss weights: The loss setting β2 = 1 and β1 = 0.01 achieves a 54.0% PASS Rate, while reducing β2 to 0.1 lowers performance to approximately 45%–46%.The ablation attributes the sensitivity to the balance between semantic alignment and latent-space regularization.
  • Training configuration: Training uses balanced mini-batch sampling so eight dataset sub-categories are uniformly distributed within each batch.This strategy is intended to reduce overfitting to dominant task categories and stabilize gradient descent.
Loading 2604.06757v3…