PaperlayerView exact revision on arXiv

Source-linked AI summary

VINO: A Unified Visual Generator with Interleaved OmniModal Context

Junyi Chen, Tong He, Zhoujie Fu, Pengfei Wan, Kun Gai, Weicai Ye

arXiv:2601.02358v2cs.CV

TL;DR

VINO targets the fragmentation of image and video generation and editing by coupling a VLM with an MMDiT in one unified framework. Its interleaved multimodal conditioning and progressive training support diverse tasks with strong visual quality, instruction following, and reference preservation, while remaining limited by text-rendering weaknesses, editing-data quality, attention cost, and VLM-supported modalities.

  • Problem

    Visual generation remains fragmented across modality- and task-specific pipelines, while multimodal models still rely on external diffusion backbones for high-resolution generation.

  • Method

    VINO couples a VLM and MMDiT, adds learnable query tokens and paired VLM/VAE reference representations, and progressively trains a video model for multi-task generation and editing.

  • Results

    Across diverse image and video generation and editing tasks, VINO delivers strong visual quality, improved identity preservation, and faithful instruction following.

  • Takeaways & Limitations

    VINO provides a unified, scalable foundation for many-to-many visual generation while retaining the generative strengths of its base video backbone.

  • Takeaways & Limitations

    VINO lacks text-rendering capability, uses lower-quality editing data that may reduce fidelity or motion richness, incurs quadratic full-attention complexity, and supports modalities constrained by its VLM.

Abstract

from arXiv · show

We present VINO, a unified visual generator that performs image and video generation and editing within a single framework. Instead of relying on task-specific models or independent modules for each modality, VINO uses a shared diffusion backbone that conditions on text, images and videos, enabling a broad range of visual creation and editing tasks under one model. Specifically, VINO couples a vision-language model (VLM) with a Multimodal Diffusion Transformer (MMDiT), where multimodal inputs are encoded as interleaved conditioning tokens, and then used to guide the diffusion process. This design supports multi-reference grounding, long-form instruction following, and coherent identity preservation across static and dynamic content, while avoiding modality-specific architectural components. To train such a unified system, we introduce a multi-stage training pipeline that progressively expands a video generation base model into a unified, multi-task generator capable of both image and video input and output. Across diverse generation and editing benchmarks, VINO demonstrates strong visual quality, faithful instruction following, improved reference and attribute preservation, and more controllable multi-identity edits. Our results highlight a practical path toward scalable unified visual generation, and the promise of interleaved, in-context computation as a foundation for general-purpose visual creation.

1. Introduction

VINO addresses fragmented visual-generation pipelines by unifying image and video generation and editing under one diffusion framework. It combines multimodal conditioning, reference-preserving representations, and progressive training to support diverse tasks with strong instruction following and identity preservation.

  • Existing visual-generation pipelines separately develop text-to-image, text-to-video, and editing models, while multimodal assistants depend on external diffusion components.
  • VINO couples a VLM with an MMDiT so one diffusion backbone handles interleaved text, image, and video controls without task-specific modules.
  • Learnable query tokens connect high-level instructions to diffusion features, while paired VLM and VAE representations preserve reference-source identity and attributes.
  • Progressive training expands a pretrained video model into a multi-task generator by adapting it to long and short captions before multi-task training.
  • Across image and video generation and editing tasks, VINO shows strong visual quality, improved identity preservation, and faithful instruction following.

2. Methods

VINO combines a frozen VLM, projected multimodal conditioning, VAE visual latents, and an MMDiT diffusion generator to support heterogeneous visual inputs. A progressive training strategy transforms a text-to-video base model into a unified multi-task visual generator.

  • VINO accepts text, reference images or videos, and learnable tokens as heterogeneous controls for diffusion-based image and video generation or editing.
  • A frozen VLM encodes language and visual conditions, with inputs ordered by modality and projected through a two-layer MLP into MMDiT conditioning.The VLM uses causal masking and penultimate-layer hidden states for the encoded conditioning.
  • Because VLM embeddings compress fine-grained spatial and texture information, VINO supplements them with VAE-encoded latents from all visual modalities.This compensation targets precise structural control, including local editing.
  • Projected latent blocks receive explicit boundary markers that provide positional cues for distinguishing separate visual conditioning inputs.
  • VINO starts from a text-to-video diffusion model, aligns VLM outputs with its native text-encoder space, and progressively expands it into a multi-task visual generator.The initial alignment stage trains only a two-layer MLP connector.

3. Experiments

VINO is evaluated across foundational generation, reference-guided generation, and image/video editing tasks using diverse datasets and baselines. Results indicate that progressive training preserves the base model’s generation capabilities while adding strong multimodal customization and editing performance.

  • Experimental Setup: VINO is evaluated across text-to-image, text-to-video, reference-driven video generation, image editing, and video editing tasks.The experiments use Geneval, VBench, OpenS2V, ImgEdit, GEdit, and OpenVE-Bench, alongside qualitative comparisons.
  • Visual Generation: VINO retains text-to-image and text-to-video performance comparable to its HunyuanVideo base model despite limited Stage 3 allocation to standard generation samples.The results are presented in Geneval and VBench evaluations and are described as evidence against catastrophic forgetting.
  • Visual Generation: VINO demonstrates strong subject-driven video generation, including multi-reference generation across humans, objects, and face identity consistency.OpenS2V results show that the model internalizes reference-aligned generation abilities introduced during later training stages.
  • Visual Editing: After 1k Stage 3 training steps, VINO surpasses many open-source models on ImgEdit, showing rapid acquisition of instruction-following image editing.Full Stage 3 training further improves performance, although text-change evaluation is excluded because the base model lacks text-rendering ability.
  • Visual Editing: VINO significantly outperforms competing methods on OpenVE-Bench and shows stronger instruction comprehension and visual quality in qualitative video-editing comparisons.The quantitative comparison uses Gemini 2.5 Pro and Qwen3VL as evaluators; qualitative comparisons include Ditto and Lucy-Edit.
  • Ablation Study: Ablations indicate that the proposed components collectively support optimization stability, controllability, and modality disentanglement.The ablation conclusions characterize these components as foundational rather than redundant.

4. Related Works

Related work reflects a shift toward diffusion-based visual generation, unified multimodal generators, and VLM-guided control. These directions motivate systems that combine shared generation backbones with richer multimodal conditioning.

  • Diffusion Models: Diffusion models have become the dominant backbone for high-fidelity image and video synthesis and downstream editing.They provide versatile conditioning interfaces for generation and editing methods.
  • Multimodal Conditioning: Unified multimodal generation faces a trade-off in image-CFG scaling: stronger reference identity adherence can reduce motion diversity and temporal dynamics.The qualitative comparison attributes the loss of dynamics to excessively large image-CFG values.
  • Unified Visual Generators: Unified visual generators consolidate tasks such as text-to-image, reference-guided generation, and image editing within shared frameworks.Unified video systems additionally target combinations of video understanding, generation, and instruction-based editing.
  • Vision–Language Models: VLMs are increasingly used as controllers, teachers, evaluators, and sources of semantic localization for diffusion-based editing.These roles support instruction decomposition, supervision, and instruction-aware evaluation.
  • Multimodal Conditioning: Separating VAE latents with a special token prevents temporal structure from input videos becoming entangled with static image latents.Removing the token produces artifacts, especially distorted structures in the first generated frame.

5. Conclusion

VINO unifies image and video generation and editing through interleaved omnimodal conditioning and progressive training. The framework remains bounded by limitations in text rendering, editing-data quality, inference efficiency, and supported modalities.

  • Conclusion: VINO performs image and video generation and editing under a single framework that integrates heterogeneous inputs through interleaved omnimodal context.Its unified conditioning pipeline supports a broad spectrum of visual tasks.
  • Conclusion: Progressive training preserves the generative strengths of the base video backbone while adding robust multitask capabilities.The resulting system is presented as a coherent and unified visual generator.
  • Limitations and future works: VINO lacks text-rendering capability, limiting performance on benchmarks that evaluate text editing or text generation.
  • Limitations and future works: Lower-quality instruction-editing datasets may bias the target distribution and reduce visual fidelity or motion richness relative to the original base generator.The paper identifies higher-quality editing data and more balanced training as future needs.
  • Limitations and future works: VINO’s supported modalities are constrained by the capabilities of its vision-language model and currently focus on text, image, and video.The paper suggests exploring more powerful and comprehensive VLMs.

Appendix

VINO combines Qwen3-VL for visual-instruction understanding with HunyuanVideo as its MMDiT-based video-generation backbone. These base models provide cross-modal alignment, visual tokenization, motion dynamics, and controllable conditioning.

  • Base Models: Qwen3-VL provides VINO with cross-modal alignment, visual tokenization, and visual understanding of images, videos, and natural-language instructions.It integrates image and video encoders and uses cross-modal attention without handcrafted fusion modules.
  • Base Models: HunyuanVideo supplies a token-based MMDiT framework for coherent video generation and multimodal conditioning.Its full attention directly integrates visual and multimodal inputs within a single diffusion framework.

B. More Details

The training and inference details combine synchronized multi-task data loading, proxy objectives, modality-specific condition dropping, and classifier-free guidance. These procedures support visual alignment, reference-image correspondence, and multimodal generation.

  • Training details: Task selection is synchronized across GPUs, while video inputs to Qwen3-VL are capped at 8 frames at a default rate of 1 FPS.Conditions are independently dropped with probability 0.1 for each modality, with synchronized dropping when needed for load balance.
  • Proxy tasks: An image-reconstruction proxy task aligns VLM-extracted visual features with low-level image details for the MMDiT model.The task feeds an image into the VLM while bypassing the VAE and trains reconstruction of the input image.
  • Proxy tasks: An image-selection proxy task teaches correspondence between image IDs in instructions and their associated visual content.It samples multiple reference images and prompts reconstruction of a selected image.
  • Inference details: Classifier-free guidance uses w_text=7 for text-conditioned generation and w_text=5 with w_image=1.5 for multimodal condition-controlled generation.HunyuanVideo’s native single-inference CFG input was not adopted because it requires additional post-training.

C.1. Visual Understanding

The paper reports visual-understanding evaluation for VINO’s incorporated understanding components, using Qwen3-VL without directly training it. Table 9 compares understanding-only and unified understanding–generation models across standard benchmarks.

  • Visual Understanding: Table 9 reports quantitative visual-understanding results for VINO-related components on standard benchmarks.
  • Visual Understanding: The evaluation leverages Qwen3-VL to measure understanding-related metrics even though the VLM itself was not trained directly.The authors include these metrics because visual understanding is an integral part of VINO.
  • Visual Understanding: The table compares understanding-only models with unified understanding–generation models and reports the number of parameters used for understanding tasks.

C.2. Visual Generation and Editing

This section presents quantitative results for VINO on visual generation and editing benchmarks, including VBench, ImgEdit, and GEdit.

  • Quantitative results are reported on VBench.The supplied material identifies Table 10 as containing these results.
  • Quantitative results are reported on ImgEdit.The supplied material identifies Table 11 as containing these results.

D. Qualitative results

The qualitative results illustrate progressive improvement during Stage 3 and examine how learnable tokens contribute to visual detail. The examples span diverse generated scenes, while the supplied quantitative-results passage provides only a table label.

  • Stage 3 training progression: Stage 3 benefits from alignment achieved in the previous two stages, then progressively improves visual-instruction correspondence and detailed feature capture.
  • Learnable tokens: Longer captions direct more attention toward high-level semantic text features, while learnable tokens represent more low-level details.
  • Learnable tokens: The learnable token functions as a flexible VLM interface optimized jointly with MMDiT, with a more pronounced role for shorter captions.
  • Qualitative examples: Qualitative examples include a mechanically intricate whale-like spaceship flying above clouds, alongside scenes featuring a post-apocalyptic character, a transparent creature habitat, and an anthropomorphic plant.
  • Evaluation views: Figure 13 compares the effect of learnable tokens using the same initial Gaussian noise, while Table 12 reports quantitative results on GEdit.
Loading 2601.02358v2…