PaperlayerView exact revision on arXiv

Source-linked AI summary

Lance: Unified Multimodal Modeling by Multi-Task Synergy

Fengyi Fu, Mengqi Huang, Shaojin Wu, Yunsheng Jiang, Yufei Huo, Hao Li, Yinghang Song, Fei Ding, Jianzhu Guo, Qian He, Zheren Fu, Zhendong Mao, Yongdong Zhang

arXiv:2605.18678v2cs.CVcs.AI

TL;DR

Existing multimodal systems largely separate understanding and generation, while unified models struggle to balance their differing visual-representation needs. Lance addresses this with unified context modeling and decoupled capability pathways, achieving strong performance across image and video understanding, generation, and editing benchmarks with 3B activated parameters.

  • Problem

    Existing multimodal systems largely separate understanding and generation, while unified models struggle to balance their differing visual-representation requirements.

  • Method

    Lance combines shared interleaved multimodal context modeling with dual-stream mixture-of-experts pathways for semantic understanding and visual synthesis.

  • Results

    Lance achieves strong performance across image and video understanding, generation, and editing benchmarks, with only 3B activated parameters and a maximum 128-GPU training budget.

  • Takeaways & Limitations

    The results indicate that multi-task synergy can advance unified multimodal modeling by allowing diverse tasks to mutually enhance one another within a shared framework.

  • Takeaways & Limitations

    Accurate, layout-consistent text rendering remains limited because dedicated text-rendering data is absent from pre-training and continued training.

Abstract

from arXiv · show

We present Lance, a lightweight native unified model supporting multimodal understanding, generation, and editing for both images and videos. Rather than relying on model capacity scaling or text-image-dominant designs, Lance explores a practical paradigm for unified multimodal modeling via collaborative multi-task training. It is grounded in two core principles: unified context modeling and decoupled capability pathways. Specifically, Lance is trained from scratch and employs a dual-stream mixture-of-experts architecture on shared interleaved multimodal sequences, enabling joint context learning while decoupling the pathways for understanding and generation. We further introduce modality-aware rotary positional encoding to mitigate interference among heterogeneous visual tokens and boost cross-task alignment. During training, Lance adopts a staged multi-task training paradigm with capability-oriented objectives and adaptive data scheduling to strengthen both semantic comprehension and visual generation performance. Experimental results demonstrate that Lance substantially outperforms existing open-source unified models in image and video generation, while retaining strong multimodal understanding capabilities. The homepage is available at https://lance-project.github.io.

1 Introduction

Lance addresses the separation and limited coverage of existing unified multimodal models with a lightweight native model spanning image/video understanding, generation, and editing. It combines shared multimodal context modeling with decoupled capability pathways and achieves strong performance with only 3B activated parameters.

  • Motivation: Existing multimodal systems largely separate understanding from generation, despite increasing interest in native unified modeling.Understanding favors high-level language-aligned semantics, whereas generation requires continuous representations preserving texture, geometry, and temporal dynamics.
  • Limitations: Prior unified models remain limited in task coverage, often focusing on text-image domains or partial combinations rather than the full image-video space.Recent video-capable systems still cover only limited subsets of image/video understanding and generation.
  • Lance: Lance integrates X2T, X2I, and X2V tasks in one native model for image and video understanding, generation, and editing.The model frames multi-task synergy as transfer across modality-task boundaries rather than simple capability aggregation.
  • Performance: 3B activated parameters enable Lance to substantially outperform existing open-source unified models on image and video generation while retaining advanced multimodal understanding.The reported gains are achieved within a 128-GPU training budget.
  • Method: Lance uses a dual-stream mixture-of-experts architecture with shared interleaved multimodal sequences and dedicated visual representations for understanding and generation.A modality-aware positional encoding scheme and staged multi-task training further improve heterogeneous visual-token coordination and cross-task context modeling.

2 Related Work

Related work spans specialized multimodal understanding and visual generation systems, alongside unified multimodal models that seek to combine both capabilities. However, existing unified modeling remains largely image-centric, making video understanding, generation, and editing particularly challenging.

  • Multimodal understanding: MLLMs align pretrained visual encoders with language backbones for image and video understanding, progressing from Flamingo, IDEFICS, and InstructBLIP to LLaVA, Qwen-VL, and InternVL.Later open-source families improve instruction following, high-resolution perception, and long-context multimodal reasoning.
  • Visual generation: Diffusion- and flow-based frameworks remain mainstream paradigms for high-fidelity image and video synthesis.Representative image-generation systems include Stable Diffusion, FLUX, Qwen-Image, and HunyuanImage 3.0.
  • Unified multimodal modeling: Unified multimodal models combine understanding and visual generation within a single framework, with autoregressive systems casting both tasks as next-token prediction in a shared token space.Representative systems include Chameleon, Emu3/Emu3.5, TokenFlow, and HunyuanImage 3.0.
  • Unified multimodal modeling: Unified multimodal modeling remains largely image-centric, while video extension requires semantic understanding, temporal reasoning, motion modeling, long-context generation, and consistent editing.Early general any-to-any or modular systems such as NEXT-GPT and GPT4Video use external generative backends.

3 Methodology

Lance unifies multimodal context modeling across text, images, and videos while decoupling understanding and generation into specialized pathways. Its design combines heterogeneous token representations, generalized 3D causal attention, and modality-aware positional encoding to preserve task-specific structure and capabilities.

  • Overall Framework: Lance is grounded in unified context learning and decoupled capability pathways for multimodal understanding, generation, and editing.Interleaved multimodal sequence modeling enables shared context learning, while specialized pathways address task-specific requirements.
  • Unified Context Learning: Text, semantic ViT tokens, clean VAE latents, and noisy VAE latents are organized into one interleaved multimodal sequence.This representation supports understanding, generation, and mixed interleaved multimodal samples within a single context-modeling framework.
  • Unified Context Learning: Generalized 3D causal attention combines causal text processing with bidirectional visual-token attention within modality-specific segments.Each segment attends to preceding clean segments, preserving causal dependencies while capturing spatial and spatiotemporal structure.
  • Decoupled Capability Pathways: The understanding expert autoregressively predicts text from text and semantic visual tokens, whereas the generation expert predicts flow in continuous VAE latent space.The two experts use separate prediction heads and pathway-specific objectives for semantic understanding and visual synthesis.
  • Modality-Aware Rotary Positional Encoding: MaPE separates visual token groups through fixed temporal offsets while preserving their intrinsic spatial layouts.The offsets distinguish semantic ViT features, clean VAE conditions, and noisy VAE targets without changing spatial coordinates.

4 Training and Data

Lance uses staged multi-task training to balance multimodal understanding and generation, progressing from paired pre-training through continual training, supervised fine-tuning, and reinforcement learning. The stages expand task coverage, refine controllability and consistency, and directly optimize fine-grained image-generation behavior.

  • Training stages: Lance’s four-stage pipeline progresses from paired pre-training (PT) to continual training (CT), supervised fine-tuning (SFT), and reinforcement learning (RL).PT establishes basic image/video understanding and generation; CT expands the task space; SFT refines task-aligned behavior; RL further optimizes image generation.
  • Pre-training: Approximately 1B image-text pairs and 140M video-text pairs support PT across diverse visual domains and dynamic scenarios.PT organizes these datasets around paired captioning and conditional generation tasks while freezing the VAE and ViT encoders.
  • Continual training: CT introduces 2.73M interleaved understanding samples plus 2.8M image-editing, 2.6M video-editing, 3.6M subject-driven image-generation, and 1M subject-driven video-generation samples.A progressive data-mixture strategy increases the sampling ratio of challenging tasks such as editing and subject-driven generation.
  • Continual training: CT uses task-specific system prompts to provide explicit task priors and guide task-specific input-output formats while preserving unified sequence modeling.The prompts distinguish heterogeneous understanding and generation tasks within a unified multimodal context.
  • Supervised fine-tuning: SFT uses high-quality, task-aligned supervision to improve instruction fidelity, visual consistency, editing accuracy, identity preservation, controllability, and downstream performance.Its data includes 190K high-quality image-captioning, 5K video-captioning, 190K image-generation, and 84K image-editing samples, alongside 2.73M interleaved understanding samples.
  • Reinforcement learning: RL applies GRPO with PaddleOCR rewards to 20K image-generation prompts, targeting text-rendering accuracy, image-text correspondence, and fine-grained textual constraints.The reward evaluates consistency between generated images and prompt-specified textual constraints, providing feedback beyond supervised fine-tuning.

5 Experiments

Lance achieves strong unified performance across image and video generation, multimodal editing, and video understanding. Quantitative and qualitative results show competitive or leading benchmark scores alongside semantic fidelity, visual quality, temporal coherence, and fine-grained understanding.

  • Image Generation: Lance matches the best overall GenEval score (0.90) among unified models and shows strong counting, color, spatial-position, and relation modeling.On DPG-Bench, Lance performs particularly well on relation modeling, indicating preservation of fine-grained semantic consistency.
  • Image Generation: Qualitative comparisons show stronger visual aesthetics and image-text alignment than Bagel and InternVL-U, including accurate object counts and spatial relations.The comparisons include 7B Bagel, 1.7B InternVL-U, 20B Qwen-Image, and Nano Banana.
  • Video Generation: Lance achieves the best VBench Total Score (85.11) among unified models with only 3B activated parameters, spanning quality and semantic dimensions.The reported strengths include visual quality, object grounding, color consistency, spatial relationships, scene understanding, and temporal style.
  • Multimodal Editing: Lance achieves the best GEdit-Bench Avg/G_O score (7.30) among unified models and leads several editing categories under a compact parameter budget.Leading categories include background change, material modification, motion change, portrait beautification, subject removal, replacement, and tone transfer.

6 Ablation Study

The ablations show that increasing training tokens progressively improves generation quality, while task mixing and MaPE strengthen Lance’s unified multimodal capabilities. Understanding data aids visual synthesis, multi-task generation data improves both generation and video understanding, and MaPE benefits generation, editing, and understanding.

  • Quantitative Analysis: Image and video generation improve rapidly during early pretraining, then enter a slower-growth regime as training tokens increase.Later tokens mainly refine prompt alignment, visual fidelity, and temporal consistency.
  • Quantitative Analysis: The CT stage further improves native generation capability, validating multi-task synergy in unified multimodal modeling.
  • Qualitative Analysis: Increasing the training budget from 0.5T to 1.5T improves prompt alignment, visual fidelity, text rendering, and temporal coherence.The 1.5T model produces more faithful compositions and more coherent multi-object dynamics than earlier variants.
  • Effect of Understanding Data: The Gen.:Und. = 8 : 2 setting improves both image and video generation, indicating that understanding data provides semantic grounding for visual synthesis.
  • Effect of Multi-task Data: Both multi-task generation mixtures outperform the generation-only baseline, with Gen.:MT-Gen. = 6 : 4 achieving the best overall results and improving video understanding.This suggests multi-task synergy extends beyond simple accumulation of capabilities.
  • Modality-Aware Rotary Positional Encoding: 6.30 to 6.86: removing MaPE consistently degrades generation, editing, and understanding performance, especially for image editing.MaPE reduces positional ambiguity among heterogeneous visual token groups, improving cross-task contextual alignment and visual synthesis stability.

7 Conclusion, Limitations and Future Work

Lance demonstrates that multi-task synergy can advance unified multimodal modeling for image and video understanding, generation, and editing. Future work targets stronger post-training supervision, improved text rendering, expanded modalities, scaling, and real-time interaction.

  • Conclusion: Lance is a lightweight native unified multimodal model for image and video understanding, generation, and editing, built around multi-task synergy.It combines unified interleaved context modeling with decoupled capability pathways so semantic understanding and visual synthesis can interact.
  • Future Work: More comprehensive video-aware reward models and reward-based optimization could strengthen temporally coherent, visually appealing, and user-aligned generation.These methods are proposed as stronger supervision for post-training.
  • Future Work: Dedicated text-rendering data remains needed because current pre-training and continued-training stages omit it, limiting accurate and layout-consistent text rendering.The reinforcement-learning stage partially improves text-related generation, while incorporating dedicated data may further improve the capability.
  • Future Work: Scaling model capacity, expert capacity, and context length may improve Lance’s overall capability and cross-task generalization.This is identified as a model-scaling direction for future exploration.
  • Future Work: Adding audio, speech, 3D, depth, and embodied sensory signals could advance Lance toward general-purpose any-to-any multimodal intelligence.These broader modalities are described as a natural next step.
  • Future Work: Streaming perception and generation mechanisms could extend Lance toward real-time interaction and closed-loop multimodal agents.This direction would support streaming multimodal interaction.
  • Conclusion: Lance is intended as a practical foundation for future research on efficient, scalable, and task-general unified multimodal systems.The authors present this as the model’s broader research role.
Loading 2605.18678v2…